[Code] UnicodeEx - (0.2.0a) Sketchup + Character Encoding
-
Very valuable research, Thom. I've got one user who is currently suffering with these problems, and I now feel armed to release a patch for him. Thanks much. CB.
-
What kind of problems where there?
-
Thanks a lot Thomas for the library. SU2TK fails to export to folders and to files containing unicode characters.. I will try the lib and maybe will be able to help you improve it.
It is so irritating. I was furious when I couldn't figure out why Ruby didn't want to write to a C:\Chałupa\chałupa.xml when the folder existed. Now I know why.Will definitely come back with comments.
Tomasz
-
Yea. I've not had the time to make methods to read and write files. Basically any IO functions are missing from my lib as they seem to require quite a bit of work to get fully working. It's been put on my backburner.
-
@thomthom said:
Overview
Sketchup uses UTF-8 character encoding. All the Sketchup API methods return UTF-8 strings.Is that why the $KCODE (alias $-K) global is set by default to Unicode by Sketchup? (at Least in SU ver 7.1)
From the book Programming Ruby (for ver1.6.x) p137:
@unknownuser said:
Command-Line Options
-K kcode
Specifies the code set to be used. This option is useful mainly when Ruby is used for Japanese-language processing. kcode may be one of:
- e, E for EUC; * s, S for SJIS; * u, U for UTF-8 * a, A, n, N for ASCII
I take the n, N to stand for 'None', meaning ASCII is the default.
same book Programming Ruby, (chap 'The Ruby Language')
@unknownuser said:Source Layout
Ruby programs are written in 7-bit ASCII.
[Ruby also has extensive support for Kanji, using the EUC, SJIS, or UTF-8 coding system. If a code set other than 7-bit ASCII is used, the KCODE option must be set appropriately, as shown on page 137.][later, in same chapter...]
Execution Environment Variables
$-K String [...alias $KCODE ]
Sets the multibyte coding system for strings and regular expressions. Equivalent to the -K command-line option. See page 137.[holds one of: EUC, SJIS, UTF-8, or ASCII ]
I looked at my rbconfig.rb file in the full install (ver 1.9.1) and by default on Win32 machines it's supposed to be blank (or 'None') by default. from:
"#{ENV['RUBYLIB']}/#{RUBY_PLATFORM}/rbconfig.rb", line 75:@unknownuser said:
CONFIG["DEFAULT_KCODE"] = ""
'Purty' sure it was the same in Ruby ver 1.8.6SO... Ruby does have encoding support, but it's located in the 'enc' folder, under the platform folder.
If the 'bubble Ruby' with Sketchup doesn't have access to these files, how can changing $KCODE (or any value it might hold,) have any effect?
I have come to the conclusion, that SU Ruby [at least on Win32 PC,] really needs to have a 'Library' folder (under the program folder, not the Plugins folder,) that has many of the full Ruby library files and subfolders.
Thom, if you still think you need to do some Unicode translation, there MAY be a ready made method within the JSON package, chekout %RUBYLIB%/json/pure/generator.rb
He has built-in a UTF-8 to UTF-16 big endian converter. You'd need to modify it, as it outputs JSON strings. -
Isn't $KCODE related to Ruby 1.9?
(And are you sure it's set by SU - and not some ruby plugin?)AFIK the only Unicode support that 1.8 has is in the RegEx functions. Or have I missed something?
@dan rathbun said:
Thom, if you still think you need to do some Unicode translation, there MAY be a ready made method within the JSON package, chekout %RUBYLIB%/json/pure/generator.rb
He has built-in a UTF-8 to UTF-16 big endian converter. You'd need to modify it, as it outputs JSON strings.Dealing with strings is one thing - I have not found that so critical as long as you are careful.
The main problem is IO functions under Windows. It's a no-go. Some help here to be able to read, write files would be great. Either to create simple read, write methods. Or IO streams.
-
@thomthom said:
Isn't $KCODE related to Ruby 1.9?
No, it cannot be, because the book was written for Ruby ver 1.6.x@thomthom said:
(And are you sure it's set by SU - and not some ruby plugin?)
I turned all my plugins OFF, while doing development. And it must be SU that sets it.
I was having problems making Win32API calls, seemed the ANSI version was not being used, so I called it specifically, but didn't work at first. Then it did.. still don't know why. Anyhow I thought the default $KCODE setting was causing Windows to use the Unicode versions by default.
I tried changing it to ASCII and it didn't seem to make any difference as far as the Win32API call. Every time it failed, and I made a change, I totally rebooted SU. And then it suddenly worked, not because of anything I did. Anyhow.. weird glich maybe.. memory gremlins!What do you suggest? always use the Unicode Win32API calls?
@thomthom said:
AFIK the only Unicode support that 1.8 has is in the RegEx functions. Or have I missed something?
Well... the book says "$-K Sets the multibyte coding system for stringsand regular expressions." -
@dan rathbun said:
What do you suggest? always use the Unicode Win32API calls?
When I call Win APIs I had to call directly to the W version of the APIs.
Example, for the
Kernel32functionFindFirstFileI must callFindFirstFileWdirectly, because trying to callFindFirstFilewill useFindFirstFileA. At least in SU7.0. I have not tried this after 7.1. -
@dan rathbun said:
Well... the book says "$-K Sets the multibyte coding system for strings and regular expressions."
hmm... when does Ruby 1.8 ever treat strings as multibyte? From all my testing I found it to always treat strings as sets of single bytes. Though, please enlighten me if I'm incorrect - as that would be very interesting.
For treating strings I've been using
pack('U*)andunpack('U*)- then using the source code for the original String methods to recreate them in Unicode. -
@thomthom said:
Example, for the
Kernel32functionFindFirstFileI must callFindFirstFileWdirectly, because trying to callFindFirstFilewill useFindFirstFileA. At least in SU7.0. I have not tried this after 7.1.This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)
Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the
FindFirstFilecall to either the ANSI version of the function (FindFirstFileA) or to the Wide version (FindFirstFileW) based on if the UNICODE flag is set at compile time.The MSDN website mentions 'extra' files are needed for Unicode support on Windows.
I thought (maybe I'm wrong,) that most foreign sold Windows versions were specially compiled as Unicode versions.
But like I said, I was having similar problems, seemed like it was the Wide versions that were being called for me, instead of the ANSI versions. This is strange...
-
@thomthom said:
hmm... when does Ruby 1.8 ever treat strings as multibyte? From all my testing I found it to always treat strings as sets of single bytes. Though, please enlighten me if I'm incorrect -
Looks like your right, in that respect (referencing your testing.)
Maybe there's a hidden single/multi-byte flag or switch [for strings] setting we don't know about...
-
@dan rathbun said:
This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)
Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the FindFirstFile call to either the ANSI version of the function (FindFirstFileA) or to the Wide version (FindFirstFileW) based on if the UNICODE flag is set at compile time.
No. That is set per application. If I had an ANSI version of Windows I'd have some big problems with my other applications.
-
@dan rathbun said:
Maybe there's a hidden single/multi-byte flag or switch [for strings] setting we don't know about...
From all I read on this - multibyte support in String wasn't added until 1.9.
-
Some further reading from MS:
http://msdn.microsoft.com/en-us/library/dd374089%28VS.85%29.aspx - General Overviewhttp://msdn.microsoft.com/en-us/library/dd317766%28VS.85%29.aspx - Described the A vs W
http://msdn.microsoft.com/en-us/library/dd317748%28VS.85%29.aspx - Character sets in file names
-
@thomthom said:
@dan rathbun said:
This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)
Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the FindFirstFile call to either the ANSI version of the function (FindFirstFileA) or to the Wide version (FindFirstFileW) based on if the UNICODE flag is set at compile time.
No. That is set per application. If I had an ANSI version of Windows I'd have some big problems with my other applications.
To elaborate:
http://en.wikibooks.org/wiki/Windows_Programming/Unicode#Windows_ImplementationApplications need to define UNICODE
#define UNICODEbefore including the windows headers - where the compiler then decides to map the API call to the A or W version.
-
@thomthom said:
@thomthom said:
@dan rathbun said:
This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)
Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the FindFirstFile call to either the ANSI version of the function (FindFirstFileA) or to the Wide version (FindFirstFileW) based on if the UNICODE flag is set at compile time.
No. That is set per application. If I had an ANSI version of Windows I'd have some big problems with my other applications.
To elaborate:
http://en.wikibooks.org/wiki/Windows_Programming/Unicode#Windows_ImplementationApplications need to define UNICODE
#define UNICODEbefore including the windows headers - where the compiler then decides to map the API call to the A or W version.
Thinking of it. It might not be Sketchup that doesn't define the UNICODE. That would be odd considering SU deals with UTF-8 internally and can open SU models with Unicode characters.
But I'm guessing it's the Ruby binaries that isn't compiled with that flag. -
What I am trying to achieve is make Ruby create this file:
file = File.new('C:\Półka\Test.xml',"w")
instead of stopping a script execution with an error:
Error: #<Errno::ENOENT: No such file or directory - C:\Półka\Test.xml>
For the time being IO operations on a file are not a problem for me. Is there way to convert 'C:\Półka\Test.xml' string into something that will be recognized by Windows?Thanks
Tomasz -
But that is an IO error. You're trying to create a new file with Unicode characters in the path.
You won't get around it by converting the string with the Unicode path to a different encoding - because the file is located under the folder named "Półka" and that's where you need to tell windows to look. Which means you need to give a Unicode string - which the ruby IO methods doesn't handle.
What you need is to call the Unicode APIs that creates a file. -
@thomthom said:
... Which means you need to give a Unicode string - which the ruby IO methods doesn't handle.
What you need is to call the Unicode APIs that creates a file.OK I agree with that.
It's the Fileand Dirclasses that STILLseem to have problems on Windows, even for Ruby ver 1.9.1
see this bug report
(I'd think the easiest solution would be to add a new parameter to many of the File and Dir class methods, ie "ANSI|UNICODE" for the mswin32 edition, that would give ruby coders a 'high-level' ruby way of forcing which API call to use, [ie: Ansi or Wide] without having to resort to direct API calls.)By the way several people have created unicode libraries (extensions) for string and character.. also iconvis mentioned.
An old (2005) unicode library, this may be obsolete
A list of extensions or gems at rubyforge for unicode and unidecode -
@thomthom said:
But that is an IO error. You're trying to create a new file with Unicode characters in the path.
Can a file be created through WIN32ole.so and returned as a Ruby variable and could all writing to the file go through that extension?
Advertisement