[Code] UnicodeEx - (0.2.0a) Sketchup + Character Encoding

[thomthom]

Overview
Sketchup uses UTF-8 character encoding. All the Sketchup API methods return UTF-8 strings. An UTF-8 character can consist of 1–4 bytes.
The first 128 characters in UTF-8 is identical with ASCII. Characters that lie outside the ASCII range will consist of 2-4 bytes. (æøåüûùú all is two bytes in UTF+8)

Ruby 1.8 which is shipped with Sketchup is not aware of character encoding. A String is a series of 8bits - always! A "character" in the eyes of ruby is a byte.

For English users there isn't a problem. All letters and numbers fall within the ASCII range and the UTF-8 strings will only contain 1byte characters.

For Non-English users it's a bit more tricky.

What works and what doesn't
When you deal with UTF-8 characters from Sketchup you must remember the following:

"Test æøå".length => 11 String.length returns the number of bytes, not characters.

Methods that modify string will most likely mangle the UTF-8 strings that contain multi-byte characters.
Consider this string, "Test æøå" - if we investigate the bytes that makes up this UTF-8 string we will see this:
"Test æøå".unpack('C*')
[84, 101, 115, 116, 32, 195, 166, 195, 184, 195, 165]

As you see, "æ" consists of the two bytes 195 and 166.
Now observer what happens when we reverse the string using Ruby's built in method:

` "Test æøå".reverse

¥Ã¸Ã¦Ã tseTThe string is completely mangled. If we look at the bytes we see the reason for this: "Test æøå".reverse.unpack('C*')`
[165, 195, 184, 195, 166, 195, 32, 116, 115, 101, 84]
Ruby blindly processed each byte individually instead of each character.

Other methods:
` "Test æøå".upcase

TEST æøå`
Ruby only seem to process the bytes within the ASCII range - ignoring everything else.

` "Test æøå".chop

Test æøÃ`
One byte removed from the multibyte character "å" mangles the whole string.

As you can see, any manipulation of UTF-8 strings is a risky business as you could very easily split a multi-byte character.

How to deal with it
Fortunately, when writing Ruby scripts you don't have to manipulate strings that often. (In a web development environment this would be a true nightmare as you deal with strings all the time.)

Ruby 1.8 does have some Unicode awareness in the forms of .pack and .unpack.
"Test æøå £$€".unpack('U*') [84, 101, 115, 116, 32, 230, 248, 229, 32, 163, 36, 8364]
Here we got an array with integer values that corresponds to the UTF+8 characters. These integer values are the Unicode Code Point for the characters. Notice the Euro symbol which is a recent symbol has a high Code Point.

With the array of Code Points we can more reliably process the UTF-8 strings. To get the correct length we can use "Test æøå £$€".unpack('U*').length => 12

And we can extract characters from the string without risking splitting multi-byte characters.
[ruby:3crgdpnx]"Test æøå £$€".unpack('U*')[5..7].pack('U*')

æøå[/ruby:3crgdpnx]
This would have split the "ø" had you used the regular [ruby:3crgdpnx]"Test æøå £$€"[5..7[/ruby:3crgdpnx]] to extract the substring.

What more problematic is swapping upper and lower case letters. When you deal with Unicode characters there's allot more rules. Same thing goes for sorting alphabetically.

I have begun to create an UTF-8 sub-class of the String object that's UTF-8 aware.

# (SAMPLE CODE! - DO NOT USE OR DISTRIBUTE!)
# UTF-8 subclass of +String+
class UTF8 < String

	# Rewire the String.length to be the UTF-8 string in bytes
	alias ;bytes ;size # ;doc;
	
	# Return the number of UTF-8 characters.
	def length
		return self.unpack('U*').length
	end
	
	# (!) Substitute the .each .replace and sub-string functions.

end

This will be populated with more methods that can deal with UTF-8 strings. I have started an Google Code Project page: http://code.google.com/p/sketchup-unicode/ so that other people can contribute. Unicode support is a mammoth task and I have not set as goal to do a complete rewrite of the String class - but slowly add methods as I need them. I hope that if other developers out there write useful methods that will feed it back to this project for the benefit of the plugin community.
Note that contributions to the project should not be dirty hacks - they should produce proper reliable output.

What fails horribly - File methods
There's one major problem with Ruby under Windows. All the File methods fail when the file name or path include a non-ASCII characters. Ruby calls the ASCII version of the Windows API and therefore fails miserably. It will return errors saying the file could not be found.

On OSX Ruby calls the correct system API and there's no problems at all.

I have found a way to do file operations under Windows with UTF-8 file names - by calling the Unicode aware Windows API directly. The problem is that the File classes in Ruby is many and extensive. As with the UTF-8 sub-class of the String object I'm writing sub-class [ruby:3crgdpnx]FileW[/ruby:3crgdpnx] and [ruby:3crgdpnx]FileTestW[/ruby:3crgdpnx] of the default [ruby:3crgdpnx]File[/ruby:3crgdpnx] and [ruby:3crgdpnx]FileTest[/ruby:3crgdpnx]. On OSX they are pure sub-classes of [ruby:3crgdpnx]File[/ruby:3crgdpnx] and [ruby:3crgdpnx]FileTest[/ruby:3crgdpnx]. But under Windows I override methods to call the Unicode aware Windows API. Methods not overridden will behave as their parent class.

So far I have some a few proof of concept methods such as [ruby:3crgdpnx].exists?[/ruby:3crgdpnx], [ruby:3crgdpnx].zero?[/ruby:3crgdpnx], [ruby:3crgdpnx].size?[/ruby:3crgdpnx] and similar which provides info about the files. But I'm still not sure how to tackle read and write to files as I'm not entirely sure the relationship with the [ruby:3crgdpnx]IO[/ruby:3crgdpnx] class. I'm not sure if I can open and UTF-8 aware file stream and pass on the the [ruby:3crgdpnx]IO[/ruby:3crgdpnx] class or if I have to sub-class the [ruby:3crgdpnx]IO[/ruby:3crgdpnx] class as well.

The UnicodeEx project
I have started the UnicodeEx project as an effort to assist in the dealings of UTF-8 string, on obth platforms, and File methods on the Windows platform.

It's still early on - many thing can and will change. So please do not rely on this 100% yet. I make it available now in the hope that more people can assist to this project that's quickly growing in scope.

Repository: http://code.google.com/p/sketchup-unicode/source/checkout (latest source - most up to date)
Download .zip archive: http://code.google.com/p/sketchup-unicode/downloads/list (latest build)
Documentation: http://workshop.thomthom.net/su-unicode/

[TIG]

Thanks for taking this over and on to the next phase... my brain was melting !

[thomthom]

Unicode is a beast of a topic. The deeper I get into this the more I feel I'm in over my head. I'd rather focus on CityGen or DoubleCut, but for CityGen I need Unicode support.

@tig said:

my brain was melting !

Same here! But I found that an ice cold pint does wonders. *

*** To the head. Not the code...**

[daiku]

Very valuable research, Thom. I've got one user who is currently suffering with these problems, and I now feel armed to release a patch for him. Thanks much. CB.

[thomthom]

What kind of problems where there?

[tomasz]

Thanks a lot Thomas for the library. SU2TK fails to export to folders and to files containing unicode characters.. I will try the lib and maybe will be able to help you improve it.
It is so irritating. I was furious when I couldn't figure out why Ruby didn't want to write to a C:\Chałupa\chałupa.xml when the folder existed. Now I know why.

Will definitely come back with comments.

Tomasz

[thomthom]

Yea. I've not had the time to make methods to read and write files. Basically any IO functions are missing from my lib as they seem to require quite a bit of work to get fully working. It's been put on my backburner.

[Dan Rathbun]

@thomthom said:

Overview
Sketchup uses UTF-8 character encoding. All the Sketchup API methods return UTF-8 strings.

Is that why the $KCODE (alias $-K) global is set by default to Unicode by Sketchup? (at Least in SU ver 7.1)

From the book Programming Ruby (for ver1.6.x) p137:

@unknownuser said:

Command-Line Options

-K kcode

Specifies the code set to be used. This option is useful mainly when Ruby is used for Japanese-language processing. kcode may be one of:

• e, E for EUC; * s, S for SJIS; * u, U for UTF-8 * a, A, n, N for ASCII

I take the n, N to stand for 'None', meaning ASCII is the default.

same book Programming Ruby, (chap 'The Ruby Language')
@unknownuser said:

Source Layout

Ruby programs are written in 7-bit ASCII.
[Ruby also has extensive support for Kanji, using the EUC, SJIS, or UTF-8 coding system. If a code set other than 7-bit ASCII is used, the KCODE option must be set appropriately, as shown on page 137.]

[later, in same chapter...]

Execution Environment Variables

$-K String [...alias $KCODE ]

Sets the multibyte coding system for strings and regular expressions. Equivalent to the -K command-line option. See page 137.[holds one of: EUC, SJIS, UTF-8, or ASCII ]

I looked at my rbconfig.rb file in the full install (ver 1.9.1) and by default on Win32 machines it's supposed to be blank (or 'None') by default. from:
"#{ENV['RUBYLIB']}/#{RUBY_PLATFORM}/rbconfig.rb", line 75:

@unknownuser said:

CONFIG["DEFAULT_KCODE"] = ""
'Purty' sure it was the same in Ruby ver 1.8.6

SO... Ruby does have encoding support, but it's located in the 'enc' folder, under the platform folder.

If the 'bubble Ruby' with Sketchup doesn't have access to these files, how can changing $KCODE (or any value it might hold,) have any effect?

I have come to the conclusion, that SU Ruby [at least on Win32 PC,] really needs to have a 'Library' folder (under the program folder, not the Plugins folder,) that has many of the full Ruby library files and subfolders.

Thom, if you still think you need to do some Unicode translation, there MAY be a ready made method within the JSON package, chekout %RUBYLIB%/json/pure/generator.rb
He has built-in a UTF-8 to UTF-16 big endian converter. You'd need to modify it, as it outputs JSON strings.

[thomthom]

Isn't $KCODE related to Ruby 1.9?
(And are you sure it's set by SU - and not some ruby plugin?)

AFIK the only Unicode support that 1.8 has is in the RegEx functions. Or have I missed something?

@dan rathbun said:

Thom, if you still think you need to do some Unicode translation, there MAY be a ready made method within the JSON package, chekout %RUBYLIB%/json/pure/generator.rb
He has built-in a UTF-8 to UTF-16 big endian converter. You'd need to modify it, as it outputs JSON strings.

Dealing with strings is one thing - I have not found that so critical as long as you are careful.

The main problem is IO functions under Windows. It's a no-go. Some help here to be able to read, write files would be great. Either to create simple read, write methods. Or IO streams.

[Dan Rathbun]

@thomthom said:

Isn't $KCODE related to Ruby 1.9?
No, it cannot be, because the book was written for Ruby ver 1.6.x

@thomthom said:

(And are you sure it's set by SU - and not some ruby plugin?)

I turned all my plugins OFF, while doing development. And it must be SU that sets it.
I was having problems making Win32API calls, seemed the ANSI version was not being used, so I called it specifically, but didn't work at first. Then it did.. still don't know why. Anyhow I thought the default $KCODE setting was causing Windows to use the Unicode versions by default.
I tried changing it to ASCII and it didn't seem to make any difference as far as the Win32API call. Every time it failed, and I made a change, I totally rebooted SU. And then it suddenly worked, not because of anything I did. Anyhow.. weird glich maybe.. memory gremlins!

What do you suggest? always use the Unicode Win32API calls?

@thomthom said:

AFIK the only Unicode support that 1.8 has is in the RegEx functions. Or have I missed something?
Well... the book says "$-K Sets the multibyte coding system for stringsand regular expressions."

[thomthom]

@dan rathbun said:

What do you suggest? always use the Unicode Win32API calls?

When I call Win APIs I had to call directly to the W version of the APIs.

Example, for the Kernel32function FindFirstFile I must call FindFirstFileW directly, because trying to call FindFirstFile will use FindFirstFileA. At least in SU7.0. I have not tried this after 7.1.

[thomthom]

@dan rathbun said:

Well... the book says "$-K Sets the multibyte coding system for strings and regular expressions."

hmm... when does Ruby 1.8 ever treat strings as multibyte? From all my testing I found it to always treat strings as sets of single bytes. Though, please enlighten me if I'm incorrect - as that would be very interesting.

For treating strings I've been using pack('U*) and unpack('U*) - then using the source code for the original String methods to recreate them in Unicode.

[Dan Rathbun]

@thomthom said:

Example, for the Kernel32function FindFirstFile I must call FindFirstFileW directly, because trying to call FindFirstFile will use FindFirstFileA. At least in SU7.0. I have not tried this after 7.1.

This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)

Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the FindFirstFile call to either the ANSI version of the function ( FindFirstFileA) or to the Wide version ( FindFirstFileW) based on if the UNICODE flag is set at compile time.

The MSDN website mentions 'extra' files are needed for Unicode support on Windows.

I thought (maybe I'm wrong,) that most foreign sold Windows versions were specially compiled as Unicode versions.

But like I said, I was having similar problems, seemed like it was the Wide versions that were being called for me, instead of the ANSI versions. This is strange...

[Dan Rathbun]

@thomthom said:

hmm... when does Ruby 1.8 ever treat strings as multibyte? From all my testing I found it to always treat strings as sets of single bytes. Though, please enlighten me if I'm incorrect -

Looks like your right, in that respect (referencing your testing.)

Maybe there's a hidden single/multi-byte flag or switch [for strings] setting we don't know about...

[thomthom]

@dan rathbun said:

This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)

Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the FindFirstFile call to either the ANSI version of the function (FindFirstFileA) or to the Wide version (FindFirstFileW) based on if the UNICODE flag is set at compile time.

No. That is set per application. If I had an ANSI version of Windows I'd have some big problems with my other applications.

[thomthom]

@dan rathbun said:

Maybe there's a hidden single/multi-byte flag or switch [for strings] setting we don't know about...

From all I read on this - multibyte support in String wasn't added until 1.9.

[thomthom]

Some further reading from MS:
http://msdn.microsoft.com/en-us/library/dd374089%28VS.85%29.aspx - General Overview

http://msdn.microsoft.com/en-us/library/dd317766%28VS.85%29.aspx - Described the A vs W

http://msdn.microsoft.com/en-us/library/dd317748%28VS.85%29.aspx - Character sets in file names

[thomthom]

@thomthom said:

@dan rathbun said:

This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)

Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the FindFirstFile call to either the ANSI version of the function (FindFirstFileA) or to the Wide version (FindFirstFileW) based on if the UNICODE flag is set at compile time.

No. That is set per application. If I had an ANSI version of Windows I'd have some big problems with my other applications.

To elaborate:
http://en.wikibooks.org/wiki/Windows_Programming/Unicode#Windows_Implementation

Applications need to define UNICODE

#define UNICODE

before including the windows headers - where the compiler then decides to map the API call to the A or W version.

[thomthom]

@thomthom said:

@thomthom said:

@dan rathbun said:

This is not something that is caused by SU or Ruby... this is a Windows 'thang'. (Unless Ruby is somehow screwin' it up...)

Sounds like you have an ANSI Windows version. Windows is 'supposed' to map the FindFirstFile call to either the ANSI version of the function (FindFirstFileA) or to the Wide version (FindFirstFileW) based on if the UNICODE flag is set at compile time.

No. That is set per application. If I had an ANSI version of Windows I'd have some big problems with my other applications.

To elaborate:
http://en.wikibooks.org/wiki/Windows_Programming/Unicode#Windows_Implementation

Applications need to define UNICODE

#define UNICODE

before including the windows headers - where the compiler then decides to map the API call to the A or W version.

Thinking of it. It might not be Sketchup that doesn't define the UNICODE. That would be odd considering SU deals with UTF-8 internally and can open SU models with Unicode characters.
But I'm guessing it's the Ruby binaries that isn't compiled with that flag.

[tomasz]

What I am trying to achieve is make Ruby create this file:
file = File.new('C:\Półka\Test.xml',"w")
instead of stopping a script execution with an error:
Error: #<Errno::ENOENT: No such file or directory - C:\Półka\Test.xml>
For the time being IO operations on a file are not a problem for me. Is there way to convert 'C:\Półka\Test.xml' string into something that will be recognized by Windows?

Thanks
Tomasz