[Code] PCFileTools

Dan Rathbun

@tig said:

... and I can't see currently how to do the equivalent of ' binmode' etc. Any suggestions/additions gratefully received..

Ya know.. it's weird that the Core guys made this switch, without a way to test later IF the stream was IN binmode or not.

I checked the docs, and the Ruby 1.8.7 branch is still the same.

BUT... in the 1.9.x trunk, they have added a binmode?() boolean query method. (See the online docs for IO class. They also added methods binread and binwrite, special open methods.)
Of course, they have added a bunch of options, to read and write in several encodings.

Dan Rathbun

What effect do you imagine WIN32OLE.codepage= WIN32OLE::CP_UTF8 would have over CP_ACP (ANSI?/ASCII?), which seems to be the default.

Dan Rathbun

Ok TIG.. a general question:

most all of the methods are doing this:

arg = arg.unpack("U*").map{|c|c.chr}.join

which seems to convert a UTF-8 string (if it is one,) to an ANSI string, before passing it to Windows FSO methods that take and return Unicode strings ...

1) correct ??

2) and why ??

TIG

I found that if I didn't do that change to the string then any tests of UTF-8 strings, like PCFile.exist?(path) do not work properly and return ' false' when it should be ' true'... just like the File.exist?(path) version; BUT making that change to the string before testing it seems to return correct results - consistently ' true' when it should be ' true' and ' false' when it should be ' false'. For a simple ANSI character string it works fine either way [the unpack/join has no affect], but if you test with a UTF-8 string with accented characters [perhaps obtained from a UI.openpanel()], that is unpack/joined etc then you can see the difference between what the File.. and PCFile.. versions return...
There's probably a more elegant way to do this... BUT it seems to work the way I've bodged it together, so now perhaps we can think of better ways of achieving the same difference...

Dan Rathbun

That seems to indicate that the FSO methods are doing ANSI comparisons (perhaps by default.)

thomthom

So, does PCFile.exist?() return true for a file with, for example, Japanese characters?

Dan Rathbun

Would the Japanese Kanji chars be in the UTF-16 set?

Ya know we are all back to the ol' String encoding problem, really.

I thot about using Dan Berger's String subclass(es) WideString or whatever he called them, but it seem like it would be combersome. Unless they converted themselves automatically similar to how Numerics use coerce().
Currently the interpreter always makes ANSI strings from **" "** and **' '** literals. (and their **%**dilimeter equivs.)

I wonder if possible to create a %u function that creates UTF8 strings. And maybe a %U that creates UTF16 ?
(Are these defined in Kernel, or are they C-side interpreter functions?

(Just throwing issues in the air, "musing out load.")

thomthom

The underlying problem in Ruby 1.8 under windows is that it calls the A version of the file functions instead of the W versions. If a function is called FileFunction is used in C/C++ - when compiled it will translate to FileFunctionA or FileFunctionW depending on whether UNICODE is defined.
I was thinking that a C Extension that would forcefully call the FileFunctionW variants would be sure to work as it would be the system doing all the work. Meddling with the string in Ruby is quite likely to cause data to be lost or corrupted.

TIG

I only works for UTF-8 [i.e. 'European' accented-characters etc] - the more complex Chinese/Japanese return false when it should be true
However, if we have a way of resolving one hopefully the other will follow...

TIG

http://www.danielstutzman.com/2011/04/how-to-write-unicode-filenames-in-ruby-1-8-6 uses Win32API and iconv ...

Dan Rathbun

BTW.. if interested:

This is the Extended lib module FileUtils from Ruby v1.8.6-p287
module FileUtils (Ruby v1.8.6-p287)

Dan Rathbun

from the old Pick-axe Book:

@unknownuser said:

Strings are stored as sequences of 8-bit bytes,[For use in Japan, the jcode library supports a set of operations of strings written with EUC, SJIS, or UTF-8 encoding. The underlying string, however, is still accessed as a series of bytes.] and each byte may contain any of the 256 8-bit values, including null and newline. The substitution mechanisms in Table 18.2* on page 203 allow nonprinting characters to be inserted conveniently and portably.

refers to the table of ** codes

Also Standard Types: Strings

So it seems that (in my mind,) since Sketchup sets $KCODE to UTF8 when it loads the interpreter, we may not actually have as much of a problem on the Rubyside as I thought.

So we have a choice...

1) A pure-Ruby patch, that either accesses the system calls (for File functions,) via WIN32OLE or WIN32API (the so libraries.)

2) A compiled C patch, ie: "Cut out" the c code files that define classes IO, Dir and File (perhaps also FileTest,) and recompile with either UNICODE #defined, or change the C function calls explicitly to the wide versions. These would be ".so" files, and they would redefine the old methods. (What happens on the C-side when you re-define a C function that has already been defined? Do the C functions that the new Ruby wrappers call, need to be renamed as well?)

thomthom

I was thinking that a C extension that did most of the functions used, like File.exist?, read, write, delete and list files in folders would go a long way. It wouldn't be as extensive as a complete rewrite - therefor more quicker to develop. Then off course not replacing the existing methods - as it'd just open up a vast pool of possible problems which would require even more testing and development.

Trying to rewrite the entire File, Dir and IO classes to call the W variant file function just seems like such a massive undertaking that it'd probably never be completed.

TIG

Does Sketchup set $KCODE to "UTF8" in all locales - I know it does in UK/US and probably European language locales BUT what about Chinese ? I'll ask someone...
If we get a consistent code for their locale can we modify the pack/join code to say use 'u' not 'U' and get an appropriate conversion in different $KCODE cases ?

Dan Rathbun

@thomthom said:

Then of course not replacing the existing methods - as it'd just open up a vast pool of possible problems which would require even more testing and development.

Well what I said before still goes... The new class(es) are in a Library namespace, need to be require(d), and then referenced within an author's namespace via an alias, (as I showed in the examples above.)

I mispoke when I said redefine, they would have the same identifiers, but be within, say SKX::WIN module namespace.

@thomthom said:

Trying to rewrite the entire File, Dir and IO classes to call the W variant file function just seems like such a massive undertaking that it'd probably never be completed.

I was hoping (without yet digging into the C source,) that it might be easy to use Notepad++ search and replace to stick "W" where they need to be.

thomthom

@dan rathbun said:

@thomthom said:

Trying to rewrite the entire File, Dir and IO classes to call the W variant file function just seems like such a massive undertaking that it'd probably never be completed.

I was hoping (without yet digging into the C source,) that it might be easy to use Notepad++ search and replace to stick "W" where they need to be.

Don't think it's that easy. I think there's a few type definitions that also needs to be adjusted. And then you need to ensure that there's no hard coded struct or data type sizes used...

Dan Rathbun

Poking around, finding links to help Adam on his Hebrew webdialog problem, I ran across a link on the Wikipedia Windows Code Pages doc.

Down at the bottom:

@unknownuser said:

See also

AppLocale — a utility to run non-Unicode (code page-based) applications in a locale of the user's choice.

Dan Rathbun

BUMP