Unicode, UTF8 and Ruby

Didier Bur · 2 Mar 2008, 18:59

Bonjour Fredo,
Le problème ne vient pas d'un fichier. J'ai des noms de matériaux à récupérer dans un modèle SketchUp, pour les re-exporter vers une feuille Excel. Quand tu récupère le nom d'un matériau pour une face f, f.material renvoie par exemple une chaîne s "béton". Quand tu écris cette chaîne dans le fichier Excel, par exemple fichier.puts(s) tu n'obtiens pas "béton", mais "bÃton", parce que les caractères accentués sont codés sur 2 octets au lieu d'un. Et Ruby n'a pas de méthode pour convertir de l'UTF8 en Unicode.
Je suis obligé de faire une fonction comme celle-là:

def ocr_change_name(str)
  # replace non-digit non-letter with empty string
  str = str.gsub(/([ -#;'"$£=()|{}&+<>,;@-])/, '')
  #replace french characters
  str=str.gsub(/(Ã )/, 'a')
  str=str.gsub(/(Ã¢)/, 'a')
  str=str.gsub(/(Ã©)/, 'e')
  str=str.gsub(/(Ã¨)/, 'e')
  str=str.gsub(/(Ãª)/, 'e')
  str=str.gsub(/(Ã«)/, 'e')
  str=str.gsub(/(Ã®)/, 'i')
  str=str.gsub(/(Ã¯)/, 'i')
  str=str.gsub(/(Ã´)/, 'o')
  str=str.gsub(/(Ã¹)/, 'u')
  str=str.gsub(/(Ã§)/, 'c')
  end

Mais c'est valable juste pour le français, pas pour les autres langues. Galère...

todd burch · 2 Mar 2008, 21:14

UTF8 doesn't work with the SU Ruby API. I figured out this sad bit of news when I wrote the 3DTextTool.

UTF8 works in Ruby just fine.

Google knows. They've known for the past several maintenance updates.

Todd

TIG · 2 Mar 2008, 22:53

I came upon this somewhere... Don't know if it has any ideas that help ?

US-ASCII.rb

fredo6 · 3 Mar 2008, 20:26

@didier bur said:

Bonjour Fredo,
Le problème ne vient pas d'un fichier. J'ai des noms de matériaux à récupérer dans un modèle SketchUp, pour les re-exporter vers une feuille Excel. Quand tu récupère le nom d'un matériau pour une face f, f.material renvoie par exemple une chaîne s "béton". Quand tu écris cette chaîne dans le fichier Excel, par exemple fichier.puts(s) tu n'obtiens pas "béton", mais "bÃton", parce que les caractères accentués sont codés sur 2 octets au lieu d'un. Et Ruby n'a pas de méthode pour convertir de l'UTF8 en Unicode.

Then, with the explanation from Todd, I understand why I had problem with the dialog boxes, as Windows does support UTF8.

Didier Bur · 3 Mar 2008, 21:21

@unknownuser said:

Don't know if it has any ideas that help

TIG, it seems the "register" method is missing. Apparently not a standard method...

TIG · 3 Mar 2008, 22:09

But couldn't we (you!) use the pack / unpack tricks to convert between the two encoding ?

thomthom · 28 Jun 2009, 22:22

@unknownuser said:

UTF8 doesn't work with the SU Ruby API. I figured out this sad bit of news when I wrote the 3DTextTool.

UTF8 works in Ruby just fine.

Google knows. They've known for the past several maintenance updates.

Todd

That was my first problem when I first tried to write ruby plugins; writing in UTF-8. From doing websites I've grown into the custom of using UTF-8 to account for most languages. I figured that I was doing something wrong and meant to go back and have another look at some point. So, essentially UTF-8 is no-go? And this is due to the SU API - not Ruby?

Didier Bur · 29 Jun 2009, 10:29

Good advice, thanks TIG

TIG · 29 Jun 2009, 11:09

Yes. You can't use FileTest.exist?(Sketchup.active_model.path) if the file has unicode. The 'path' SUp reports looks OK with say ascii_chr=233 for 'é', however the FileTest sees the 'é' as a unicode and so returns false - although they both 'look' the same, the character encoding is different.
My clunky fix only works on the top-most file (or folder) containing the unicode parts, as the Dir.entities(dir) falls over if there are accents earlier in the path...
It can't be beyond the wit of man to take 'Sketchup.active_model.path' and encode it as unicode in a way that would match the Ruby built-ins like FileTest.exist?(path) or Dir.entities(dir)... however it is beyond the wit of me...

thomthom · 29 Jun 2009, 11:25

I think that I couldn't even get UTF-8 scripts to run... I'll have a look at Ruby + SU + UTF. Wonder if Ruby has some nice encoding methods.
Seeing how there's many scripts that uses localisation it's be very nice to have UTF-8.
.SKP has a weird combination of UTF+8 and regular ACSII. Seems that it wasn't originally UTF-8 and it was later added. Maybe we're running into problems due to this.

TIG · 29 Jun 2009, 11:54

233.chr ### a plain ascii é
é
233.chr+233.chr ### 2 number plain ascii é make éé
éé
195.chr ### a plain ascii capital A with an umlaut
Ã
169.chr ### a plain ascii the (c)opyright symbol

195.chr+169.chr ### BUT these 2 number ascii codes added together = one unicode é that looks like an ascii é !!!
é

??? go figure ???

thomthom · 29 Jun 2009, 12:19

UTF only uses two bytes for some of the characters. For most of the latin characters it uses 1byte equal to normal ASCII.

TIG · 1 Jul 2009, 13:58

Didieret al...

After more than a year and a bit...

typical usage: file_found?(Sketchup.active_model.path)

returns trueif the file found,

even with accented unicode characters in name/path,

e.g. qualisé.skp

EDIT: see here for latest file... http://forums.sketchucation.com/viewtopic.php?p=169225#p169225

TIG · 1 Jul 2009, 14:01

file_found?(path) that fixes ascii in SUp Ruby path and unicode in returned filepath returning false negatives with 'FileTest.exist?(path)' - even with accented characters - is updated and moved here... http://forums.sketchucation.com/viewtopic.php?p=169225#p169225