Upgrading plugins to Ruby 2.0 for SketchUp 2014

tt_su

I see you included some description in the BugSplat, but I'm afraid it's been mangled formatting. Can you attach sample RB file and sample CSV file? That'll ensure we are reproducing 100% correctly.

Dan Rathbun

FWIW. Now in Ruby 2.x, you can specify the arguments that will be used for IO.new via IO.open, as the last argument to File.foreach(), which is inherited from [%(#BF0000)[IO.foreach()]](http://www.ruby-doc.org/core-2.0.0/IO.html#method-c-foreach).

Example


File.foreach( csv_filepath, {;mode=>"r", ;encoding=>"ASCII;UTF-8"} ) do |file_data|
  # statements
end

tt_su

@dan rathbun said:

"ASCII:UTF-8"

Dan Rathbun

I also wrote:
@dan rathbun said:

..., you can specify the arguments that will be used for IO.new

so ...
http://www.ruby-doc.org/core-2.0.0/IO.html#method-c-new

It's an example.

At the console you can get names like this:
Encoding::ASCII.names %(#008040)[>> ["US-ASCII", "ASCII", "ANSI_X3.4-1968", "646"]]

Encoding::ASCII_8BIT.names %(#008040)[>> ["ASCII-8BIT", "BINARY"]]

TIG

I had a weird problem with a txt file using IO.read to get a string...
Notepad++ said was encoded as ANSI, but Ruby2.0 said was was encoded as UTF-8...
I could read and parse the string in Ruby1.8, but in Ruby2.0 it sometime 'threw a wobbler' about wrongly encoded characters...

If the string was all normal ASCII type characters then no issue...
BUT if the string contained an accented character it caused the 'wobbler'...

If I re-encoded it in Notepad++ to UTF8-without-BOM - then again no issues with the accents in Ruby2.0 or Ruby1.8.

A 'puts' for the strings read for the two file encodings showed differences with \E... etc where the accents were.

In Ruby2.0 I tried to force the encoding to UFT-8, but since it thought it already was in that it failed.
This was my workaround:

data = IO.read(@data_file) data = data.force_encoding('ISO-8859-1').encode("UTF-8") if defined?(Encoding) @uid = data.split('&')[-1].to_s.chomp --> Gábor
etc...

Now in v2014 it reads and parses OK and in earlier versions it works too...
Forcing the encoding into one that it is never going to be [as in my case anyway] and then back to UTF-8 works...
I don't know why it thought the encoding was NOT in ANSI but...

Incidentally - this also uncovered another oddity...
In Ruby1.8 if you use the File.new to make a txt file and write an ASCII string it is reported as a UTF8... encoded file by Notepad++
BUT if the string contains an accented character the file encoding is reported as ANSI.
Since Ruby1.8 can cope with either encoding when parsing strings etc it causes no issues, but if you have such a txt file, without the above double encoding workaround it causes issues with Ruby2.0...

marksup

TIG - THANK YOU!

I was working on the UTF-8 errors issue when your last post displayed. I can confirm that...

File.foreach(csv_filepath) do |file_data|
file_data.force_encoding('ISO-8859-1').encode("UTF-8") if defined?(Encoding)

Works in both (SketchUp) Ruby releases exactly as you say.

tt_su

I've reproduced the crash and it happens in the Importer class wrapper that is supposed to return the result code. That's why it works when you call the method directly and not via the Importer dialog.

tt_su

Right, so I got a fix for it on our side. Until then you can work around the crash by catching Ruby errors in Importer.load_file

<span class="syntaxdefault"><br /></span><span class="syntaxkeyword">class&nbsp;</span><span class="syntaxdefault">CSV_ImporterC&nbsp;</span><span class="syntaxkeyword"><&nbsp;</span><span class="syntaxdefault">Sketchup</span><span class="syntaxkeyword">;;</span><span class="syntaxdefault">Importer<br />&nbsp;&nbsp;def&nbsp;description<br />&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="syntaxkeyword">return&nbsp;</span><span class="syntaxstring">"CSV&nbsp;Importer&nbsp;(*.csv)"<br />&nbsp;&nbsp;</span><span class="syntaxdefault">end<br />&nbsp;&nbsp;def&nbsp;file_extension<br />&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="syntaxkeyword">return&nbsp;</span><span class="syntaxstring">"csv"<br />&nbsp;&nbsp;</span><span class="syntaxdefault">end<br />&nbsp;&nbsp;def&nbsp;id<br />&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="syntaxkeyword">return&nbsp;</span><span class="syntaxstring">"CSV-Importer"<br />&nbsp;&nbsp;</span><span class="syntaxdefault">end<br />&nbsp;&nbsp;def&nbsp;supports_options</span><span class="syntaxkeyword">?<br />&nbsp;&nbsp;&nbsp;&nbsp;return&nbsp;</span><span class="syntaxdefault">false<br />&nbsp;&nbsp;end<br />&nbsp;&nbsp;def&nbsp;load_file</span><span class="syntaxkeyword">(</span><span class="syntaxdefault">file_path</span><span class="syntaxkeyword">,&nbsp;</span><span class="syntaxdefault">status</span><span class="syntaxkeyword">)<br />&nbsp;&nbsp;&nbsp;&nbsp;return&nbsp;</span><span class="syntaxdefault">1&nbsp;</span><span class="syntaxkeyword">if&nbsp;(&nbsp;</span><span class="syntaxdefault">not&nbsp;status</span><span class="syntaxkeyword">)<br />&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="syntaxdefault">CSV_importerF</span><span class="syntaxkeyword">(</span><span class="syntaxdefault">file_path</span><span class="syntaxkeyword">)<br />&nbsp;&nbsp;&nbsp;&nbsp;return&nbsp;</span><span class="syntaxdefault">0<br />&nbsp;&nbsp;rescue&nbsp;StandardError&nbsp;</span><span class="syntaxkeyword">=>&nbsp;</span><span class="syntaxdefault">e<br />&nbsp;&nbsp;&nbsp;&nbsp;puts&nbsp;e</span><span class="syntaxkeyword">.</span><span class="syntaxdefault">message<br />&nbsp;&nbsp;&nbsp;&nbsp;puts&nbsp;e</span><span class="syntaxkeyword">.</span><span class="syntaxdefault">backtrace</span><span class="syntaxkeyword">.</span><span class="syntaxdefault">join</span><span class="syntaxkeyword">(</span><span class="syntaxstring">"\n"</span><span class="syntaxkeyword">)<br />&nbsp;&nbsp;&nbsp;&nbsp;return&nbsp;</span><span class="syntaxdefault">1<br />&nbsp;&nbsp;end<br />end<br /><br />def&nbsp;CSV_importerF</span><span class="syntaxkeyword">(</span><span class="syntaxdefault">csv_filepath</span><span class="syntaxkeyword">)<br />&nbsp;&nbsp;</span><span class="syntaxdefault">File</span><span class="syntaxkeyword">.foreach(</span><span class="syntaxdefault">csv_filepath</span><span class="syntaxkeyword">)&nbsp;do&nbsp;|</span><span class="syntaxdefault">file_data</span><span class="syntaxkeyword">|<br />&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="syntaxdefault">next&nbsp;</span><span class="syntaxkeyword">if&nbsp;(&nbsp;</span><span class="syntaxdefault">file_data</span><span class="syntaxkeyword">.</span><span class="syntaxdefault">match</span><span class="syntaxkeyword">(/\</span><span class="syntaxdefault">A</span><span class="syntaxcomment">#/)&nbsp;or&nbsp;not&nbsp;file_data.include?(",")&nbsp;)<br />&nbsp;&nbsp;&nbsp;&nbsp;</span><span class="syntaxdefault">puts&nbsp;</span><span class="syntaxstring">"Floor&nbsp;#{file_data.chomp.split("</span><span class="syntaxkeyword">,</span><span class="syntaxstring">")[1].to_i}"<br />&nbsp;&nbsp;</span><span class="syntaxdefault">end<br />end<br /><br />Sketchup</span><span class="syntaxkeyword">.</span><span class="syntaxdefault">register_importer</span><span class="syntaxkeyword">(</span><span class="syntaxdefault">CSV_ImporterC</span><span class="syntaxkeyword">.new)<br />&nbsp;</span><span class="syntaxdefault"></span>

As for reading the file correctly, I would first try to read the CSV file in UT-8, then if that fails, fall back to ISO-8859-1 (or maybe try to use whatever codepage is in use if Ruby will let you know this.)

Dan Rathbun

Didn't Bugra say the two Ruby loadpaths were encoded in the local code page ?

Jim

@tt_su said:

I've reproduced the crash and it happens in the Importer class wrapper that is supposed to return the result code. That's why it works when you call the method directly and not via the Importer dialog.

This isn't new for SketchUp 2014 - it's been around for awhile.

tt_su

@jim said:

This isn't new for SketchUp 2014 - it's been around for awhile.

Yup, I saw that when I tested. Bad one - but at least it can be caught. ...one just have to know about it...
But it's fixed in-house now.

marksup

Update - I have sent a user an .rbz, and he gets the error:

Error: #<ArgumentError: unknown encoding name - ISO-8859-1>

resulting from...
file_data.force_encoding('ISO-8859-1').encode("UTF-8") if defined?(Encoding)

He is using Windows 8, SketchUp 14, on a PC. How can this be??! - and what advice can I pass on?

marksup

Further (on my PC, which has no encoding issue)...
Encoding.default_external = UTF-8

But..
puts file_data.encoding
file_data.force_encoding('ISO-8859-1').encode("UTF-8") if defined?(Encoding)
puts file_data.encoding

Generates...
UTF-8
ISO-8859-1

I would have thought that the second encoding would always be UTF-8? I am trying to understand the insidious world of Encoding (thanks to Ruby 2)! Any advice or explanation would be appreciated.

Context: A major plugin which simply wishes to reliably read a CSV file is currently useless due to the Encoding ISO-8859-1 issue.

tt_su

@marksup said:

Update - I have sent a user an .rbz, and he gets the error:

Error: #<ArgumentError: unknown encoding name - ISO-8859-1>

resulting from...
file_data.force_encoding('ISO-8859-1').encode("UTF-8") if defined?(Encoding)

He is using Windows 8, SketchUp 14, on a PC. How can this be??! - and what advice can I pass on?

Could it be that he's using the first release for SU2014? There was a bug there where when opening SketchUp via an SKP file that is located on a drive other than where SketchUp is installed parts of the Ruby interpreter wasn't initialized - that include some of the encodings.
Can you please make sure the user has kept his installation up to date?

tt_su

@marksup said:

file_data.force_encoding('ISO-8859-1').encode("UTF-8") if defined?(Encoding)

By the way, one should really try to avoid using force_encoding. There are some scenarios where one has to because Ruby itself return strings with the incorrect encoding (Like the FILE magic variable.) but I often see this pattern when people open files. If that is the case then one should ideally spesify the encoding in the IO.open (File.open) arguments.

marksup

Thanks ThomThom,

I noticed (during a shared Skype screen session) that the user was launching SketchUp via the .skp file - so your explanation is very promising.

Regarding the use of force_encoding and File.open, I am using the following (on the advice of another SketchUcation expert!)...

File.foreach(file_path) do |file_data|
file_data.force_encoding('ISO-8859-1').encode("UTF-8") if defined?(Encoding)

Please can you provide a similar File.open example of what you would prefer/recommend.

Would it be possible for SketchUp to provide a function to reenable foolproof reading of text files? (i.e. to reinstate the automatic encoding recognition functionality that Ruby 2 is missing compared to Ruby 1.8)

tt_su

@marksup said:

Would it be possible for SketchUp to provide a function to reenable foolproof reading of text files? (i.e. to reinstate the automatic encoding recognition functionality that Ruby 2 is missing compared to Ruby 1.8)

There never was an automatic detection - as Ruby 1.8 treated all strings as 8bit byte sequences.

To provide a proper answer to you I need to know a little bit more about what type of files you are opening.

If they are binary files:


filemode = 'rb'
if RUBY_VERSION.to_f > 1.8
  filemode << ';ASCII-8BIT'
end
File.open(file_name, filemode) {|file|
  # read file
}

If you know the file is UTF-8:


filemode = 'rb'
if RUBY_VERSION.to_f > 1.8
  filemode << ';UTF-8'
end
File.open(file_name, filemode) {|file|
  # read file
}

If you know the file is ISO-8859-1 but you want it as UTF-8:


filemode = 'rb'
if RUBY_VERSION.to_f > 1.8
  filemode << ';ISO-8859-1;UTF-8'
end
File.open(file_name, filemode) {|file|
  # read file
}

I recommend reading up on the IO class and Encoding class:
http://www.ruby-doc.org/core-2.1.2/IO.html#method-c-new

Forcing an encoding is prone to errors - it's like brute forcing and crossing your fingers hoping it will work. By being clear by what encoding you expect you will catch incorrectly coded strings early and at the correct points.

marksup

Hi, thanks, but you may be missing the point...

I would wish to read a simple text (not binary) file, (typically in CSV format).

I do not know what encoding a user may use to create the text file - and most likely neither will the user!

Thus the file may be UTF-8 already, or it may be something else, I just wish to reliably and simply process it whatever the encoding, as was apparently possible with earlier releases.

I have no attachment to force_encoding - merely that it was suggested and it does appear to work, and will happily comply with any preferred alternative.

So, what extra logic is required for anyone and everyone to reliably process any text file, (which might include £ or m[sup:1cfz4sft]2[/sup:1cfz4sft], for example) regardless of how it might be encoded?

tt_su

Your force encoding example will only work if the input text file (CSV) file is ISO-8859-1. ISO-8859-1 is all 8bit per character. However, if you open a file that is saved with UTF-8 encoding and it contains characters over the ASCII range (127) then there will be multi-byte characters - when you then force ISO-8859-1 encoding and convert that to UTF-8 you will mangle the characters.

If you have no idea or control over the input data then I would try to use a try/catch approach of first reading the file in what is more likely for the file to be encoded in. If you get encoding errors thrown which you can catch and try the next likely encoding. You can then fall back to just reading it as binary:


if RUBY_VERSION.to_f > 1.8
  filemode << ';ASCII-8BIT'
end
File.open(file_name, filemode) {|file|
  file.seek(80, IO;;SEEK_SET)
  face_count = file.read(4).unpack('i')[0]
}

Look at these errors and see if they might be thrown:

Encoding;;CompatibilityError
Encoding;;ConverterNotFoundError
Encoding;;InvalidByteSequenceError
Encoding;;UndefinedConversionError
EncodingError

If you are not familiar with Unicode and how it's represented in byte data I would recommend reading up on that as well. The reason it has worked for you so far has probably been that you have had ASCII compatible data. UTF-8 is byte compatible with ASCII where it uses only one byte per character - but the moment you go outside the ASCII (US-ASCII to be precise) it get multibyte characters.

For your testing purposes I would strongly recommend you test with non-english characters. For good measure make sure you go outside of European languages as well, try Japanese or Chinese for instance which might be four byte per characters.