Ruby 2.0 __FILE__ contains incorrect encoding.

Dan Rathbun

@aerilius said:

I consider it a failure that you have to force the encoding. Because that means Ruby is messing up stuff. If the strings where marked with proper encoding it should have been able to correctly transpose the string.

Well set the default bleepin' encodin' then...

fredo6

Thomthom,

I use FILE in my plugins (actually mostly in the top rb files. I also use $LOAD_PATH in a few places.

Although many users seem to have no problem (but how many do use a non-ascii username?), it is good to understand if the problem is critical and require an urgent fix.

What do you think?

Fredo

Aerilius

When it fails, it would show something different (?). This is what I get:


C;/Users/Administrator/AppData/Roaming/SketchUp/SketchUp 2014/SketchUp/Plugins/encoding_test/hello.rb
> Hello World
require succeeded!

> Hello World
load succeeded!

Forcing UTF-8 encoding...

require succeeded!

„Native Windows? Or Linux and Wine?“
If you had a Linux version of SU (nice!), if SU2014 ran in Wine (nice!), otherwise for all my bug reports I test on a native Windows in a virtual machine. Linux is the luxury and comfort of a Mac with the freedom to work on any hardware. But I have to accept what I can get from SketchUp. I wished SketchUp would become kindof usable on higher-dpi screens. All my other applications scale perfectly. I need patience, at least I'm still with SketchUp after years because I'm addicted.

Dan Rathbun

Ruby does not support Windows-1258 encoding. It is a known bug.

@unknownuser said:

SketchUcation Tools 2.5](http://sketchucation.com/forums/viewtopic.php?f)":3vtmbnty]
@kienhp said:

Not run in skechUp 2014, why?

Because Vietnamese encoding is not supported by Ruby yet.

Bug # 7742 : System encoding (Windows-1258) is not recognized by Ruby to convert back to UTF-8

One user changed his Regional and Langauge settings to US English to overcome this:
StackOverflow : Error installing Rubygems on ruby command prompt in Win7

tt_su

@aerilius said:

When it fails, it would show something different (?). This is what I get:

You are logged in as "Administrator" - the issue occur when the username contain non-English characters.

tt_su

@fredo6 said:

I use FILE in my plugins (actually mostly in the top rb files. I also use $LOAD_PATH in a few places.

Although many users seem to have no problem (but how many do use a non-ascii username?), it is good to understand if the problem is critical and require an urgent fix.

What do you think?

We're still digging into this. The Ruby Unicode issues under Windows is a deep rabbit hole.
On one side there is the wrong encoding begin returned for strings that contain UTF-8 data.
On the other side there is Ruby C Extensions that doesn't load with non-English characters.

Now, there might be cases where paths with non-english characters will load, if the system code page fits. Though we've not got around to test this yet. But on my machine, which is an English system with Windows-1252 code page - ASCII calls to file functions fail to load non-English characters.

Now, other users, say Japanese users, might have a Japenese code page configured for their machine and they might experience that having a Japanese username works for them - even though it fails for me.

Encoding under Windows is a jumble. Under OSX it's all fine because it all UTF-8 by default - even the file functions.

As I mentioned, we're still digging into this and we'll come back with more info. I wouldn't necessarily start updating scripts right away. We don't know what's the best recommendation yet. But just so you are aware there is a known issue that's being investigated - and it can crop up differently from machine to machine.

tt_su

@driven said:

file = __FILE__
> file_encoding = file.encoding.name
> puts file_encoding.to_s
> if not file_encoding.valid_encoding?
> puts "bad encoding, fixing"
> data = File.open(file, (file_encoding + ';utf-8')).read
> else
> puts "nothing to fix???"
> data = File.open(file).read
> end

I don't quite understand what your code snippet is doing here. It seems like you're mixing up the encoding of the filename string with the encoding of the File object.

And this: file_encoding + ':utf-8' - what is that doing? Appending UTF-8 to another encoding declaration?

@driven said:

so whats going on???

I don't know. Can you explain what you are testing with that snippet?

driven

I have a test file encoded in Vietnamesse(Windows) with CRLF
using unix I can see ...

 
> %x(file "path../dev/file•enc•test.rb")
path../dev/file•enc•test.rb; ASCII text, with CRLF line terminators

In SU...

load("path../dev/file•enc•test.rb")
UTF-8
nothing to fix???
true

EDIT.... added missing notation [ 'r:' ]

file = __FILE__
file_encoding = file.encoding.name
puts file_encoding.to_s
if not file_encoding.valid_encoding?
puts "bad encoding, fixing"
# specify both external and internal encodings
data = File.open(file, ('r;' + file_encoding + ';utf-8')).read
else
puts "nothing to fix???"
data = File.open(file).read
end

so whats going on???

john

driven

@tt_su said:

I don't know. :? Can you explain what you are testing with that snippet?

It was missing a critical piece of notation... [r:] I modified the snippet, will retest later...
I was trying to set both external and internal encoding before the read...
based on this
http://stackoverflow.com/questions/8610100/changing-character-encoding

john

Dan Rathbun

@tt_su said:

And this: file_encoding + ':utf-8' - what is that doing? Appending UTF-8 to another encoding declaration?

NO. John is building a Ruby v2+ IO Mode String.

Many of the IO and subclasses (like File,) use it in methods, which transparently call IO::new within themselves. (Examples, are IO.read, File.read, File.open, etc.)

So if you read the doc for IO.new, you'll see that wherever you previously used just the filemode ("r" "w+" "rb" etc.,) that argument can now have up to 3 sections:

" *filemode* : *external_encoding* : *internal_encoding* "

The online doc shows this example (among others.):

open("transcoded.txt", "r;ISO-8859-1;UTF-8") do |io|
  puts "transcoded text;"
  p io.read
end

So the new mode string above is: "r:ISO-8859-1:UTF-8"

Also for better readability, many of these methods that take that 1|2|3 part string (separated by colons,) can instead, take hash arguments, like this:

open(
  "transcoded.txt", 
  ;mode => "r",
  ;external_encoding => "ISO-8859-1",
  ;internal_encoding => "UTF-8"
) do |io|
  puts "transcoded text;"
  p io.read
end

In some cases you can use an or'ed encoding string, like: "BOM|UTF-8"

But really ya'll should read the extensive info now listed under IO::new() AND the extensive information at the top of the Encoding class page.

fredo6

@tt_su said:

As I mentioned, we're still digging into this and we'll come back with more info. I wouldn't necessarily start updating scripts right away. We don't know what's the best recommendation yet. But just so you are aware there is a known issue that's being investigated - and it can crop up differently from machine to machine.

OK. I'll wait.

But can you confirm that your suggested fix is safe anyway.

Fredo

tt_su

@fredo6 said:

But can you confirm that your suggested fix is safe anyway.

That's why I ask that people to hold. We need to perform some testing.

driven

I started a new encoding bug topic incase it's unrelated...
http://sketchucation.com/forums/viewtopic.php?f=180%26amp;t=57074#p518534
can you all have a look?
john