Safe place to store user-defined parameters

thomthom · 24 Jun 2013, 09:09

The "correct" place for persistent data for an application that is not part of the installation is the appdata location. These directories are separate for each user and guaranteed to be readable/writable for the associated user. This is normally were things like plugins/extensions would exist, also.

But as I mentioned - Ruby 1.8 cannot handle non-ASCII characters. Usernames might include Unicode characters and then all file operations on the user directory will simply fail.

fredo6 · 24 Jun 2013, 09:17

I will release the new versions of LibFredo6

with ENV["LOCALAPPDATA"] on Windows
with TIG's method on Mac, that is the parent directory of the ENV["TMPDIR"] folder. I think using ENV["HOME"] with some hidden subfolder would have been considered too intrusive.

These will be root folders into which I will create the LibFredo6 subfolders for persisting data.

I think in both cases these folder won't contain non-ascii characters, as they are defined and created by the system itself.

Fredo

TIG · 24 Jun 2013, 09:25

On windows the Local path (ENV["LOCALAPPDATA"]) could return as say
f=ENV["LOCALAPPDATA"]) 'C:\Users\TIGé\AppData\Local'
which will return ' false' in a File.exist?(f) test when it actually exists, just because the user-name contains an accented character!

So use my 'safe' way [a few post ago] to make the folder, even if it exists already!

danielbowring · 24 Jun 2013, 09:44

@thomthom said:

But as I mentioned - Ruby 1.8 cannot handle non-ASCII characters. Usernames might include Unicode characters and then all file operations on the user directory will simply fail.

All the more reason to update the ruby installation. The provided order would still "work" if you checked each of the directories existed (because the check would "simply fail"), but it's certainly no supplement for the actual feature.

It seems to be reoccurring that the SketchUp ruby API will gives you "clipped wings". I'm excited to see where it can go, but also impatient with its shortcomings (even though this one isn't really SU fault).

Side-note: No unicode support seems strange for something coming out of japan

danielbowring · 24 Jun 2013, 09:47

@fredo6 said:

I will release the new versions of LibFredo6

with ENV["LOCALAPPDATA"] on Windows

with TIG's method on Mac, that is the parent directory of the ENV["TMPDIR"] folder. I think using ENV["HOME"] with some hidden subfolder would have been considered too intrusive.

These will be root folders into which I will create the LibFredo6 subfolders for persisting data.

I think in both cases these folder won't contain non-ascii characters, as they are defined and created by the system itself.

Fredo

ENV["LOCALAPPDATA"] may contain non-ascii characters, as it contains the current users username. Also, you should use ENV["LOCALAPPDATA"] || ENV["APPDATA"] so you can support XP at no extra effort

Bonus:

thomthom · 24 Jun 2013, 10:28

Just to test things I created a user account "tæst" under Windows7. And ran some tests in the Ruby Console:


ENV['HOMEPATH']
\Users\t‘st

ENV["LOCALAPPDATA"]
C;\Users\t‘st\AppData\Local

ENV["APPDATA"]
C;\Users\t‘st\AppData\Roaming

file = File.join( ENV['LOCALAPPDATA'], 'FooBar.txt' )
C;\Users\t‘st\AppData\Local/FooBar.txt

File.open( tempfile, 'wb' ) { |file| file.puts 'Hello world' }
Error; #<Errno;;ENOENT; (eval);0;in `initialize'; No such file or directory - C;\Users\t‘st\AppData\Local/FooBar.txt>
(eval)
(eval);0;in `open'
(eval);0


ENV['TEMP']
C;\Users\TST~1\AppData\Local\Temp

tempfile = File.join( ENV['TEMP'], 'FooBar.txt' )
C;\Users\TST~1\AppData\Local\Temp/FooBar.txt

File.open( tempfile, 'wb' ) { |file| file.puts 'Hello world' }
nil

Interesting observations:

Most ENV has some strange character replacements. Is this a Ruby thing?

ENV['TEMP'] returned a path in the format of the old DOS 8.3 format. And this could be written to.


ENV.each { |k,v| puts "#{k}\t#{v}" }
ALLUSERSPROFILE	C;\ProgramData
APPDATA	C;\Users\t‘st\AppData\Roaming
CommonProgramFiles	C;\Program Files (x86)\Common Files
CommonProgramFiles(x86)	C;\Program Files (x86)\Common Files
CommonProgramW6432	C;\Program Files\Common Files
COMPUTERNAME	ARC-CU-10-09
ComSpec	C;\Windows\system32\cmd.exe
FP_NO_HOST_CHECK	NO
HOMEDRIVE	C;
HOMEPATH	\Users\t‘st
LOCALAPPDATA	C;\Users\t‘st\AppData\Local
LOGONSERVER	\\ARC-CU-10-09
NUMBER_OF_PROCESSORS	8
OS	Windows_NT
PATHEXT	.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE	x86
PROCESSOR_ARCHITEW6432	AMD64
PROCESSOR_IDENTIFIER	Intel64 Family 6 Model 26 Stepping 5, GenuineIntel
PROCESSOR_LEVEL	6
PROCESSOR_REVISION	1a05
ProgramData	C;\ProgramData
ProgramFiles	C;\Program Files (x86)
ProgramFiles(x86)	C;\Program Files (x86)
ProgramW6432	C;\Program Files
PSModulePath	C;\Windows\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC	C;\Users\Public
SESSIONNAME	Console
SystemDrive	C;
SystemRoot	C;\Windows
TEMP	C;\Users\TST~1\AppData\Local\Temp
TMP	C;\Users\TST~1\AppData\Local\Temp
ULTRAMON_LANGDIR	C;\Program Files\UltraMon\Resources\en
USERDOMAIN	arc-cu-10-09
USERNAME	t‘st
USERPROFILE	C;\Users\t‘st
windir	C;\Windows

The TEMP path seem to be the only one using this format.
So, question is, is it possible to monkey-patch this so one can extract the user directory in DOS8.3 format and use that to replace the mangled paths of the other variables?

And what does it return under other Windows versions? (Win8, Vista, XP)

What about chineese/japanese types or characters - does the same thing apply?

thomthom · 24 Jun 2013, 10:29

@unknownuser said:

Also, you should use ENV["LOCALAPPDATA"] || ENV["APPDATA"] so you can support XP at no extra effort

What folders does this yield on XP? Under Win7 it's both in the user directory.

fredo6 · 24 Jun 2013, 11:25

I will use the parent of ENV["TMPDIR"] only on Mac. Apparently, this directory has a long cryptic name, which I assume will only contain valid characters.

On Windows, this is more of a problem is ENV["LOCALAPPDATA"] cannot be manipulated from Ruby dues to non Ascii characters. It's not only for creating subfolders there, but also to manage filepaths of files to be created and updated.

The best I can do is to check if the username contains non-ascii characters and then simplify the name to Ascii, and create a folder with that simplified name in C:\users. This really becomes convoluted!!

There must be a trick however, because when Sketchup installs, it sets the default file locations pointing to the "MyDocuments" folder, whose name also contains the username (something like C:\Userfiles\<username>\Documents\). But maybe these locations are not used from Ruby.

Fredo

thomthom · 24 Jun 2013, 11:35

@fredo6 said:

The best I can do is to check if the username contains non-ascii characters and then simplify the name to Ascii, and create a folder with that simplified name in C:\users. This really becomes convoluted!!

Creating random folder directly in the location where user folders are? I don't think people would too happy about this. I'd like to explore if we can get the 8.3 filename for the appdata folder.

@fredo6 said:

There must be a trick however, because when Sketchup installs, it sets the default file locations pointing to the "MyDocuments" folder, whose name also contains the username (something like C:\Userfiles\<username>\Documents\). But maybe these locations are not used from Ruby.

Correct - SketchUp itself has no problems with Unicode paths. It's the Ruby 1.8 core.

fredo6 · 24 Jun 2013, 11:42

@thomthom said:

@unknownuser said:

Also, you should use ENV["LOCALAPPDATA"] || ENV["APPDATA"] so you can support XP at no extra effort

What folders does this yield on XP? Under Win7 it's both in the user directory.

Actually as I use both, LOCALAPPDATA first and then APPDATA.

For info, on WindowsXP, ENV["APPDATA"] is defined and uses the old Windows user directory in the form of C:\Documents and Settings\<username>\Application Data.

Fredo

thomthom · 24 Jun 2013, 11:50

@fredo6 said:

For info, on WindowsXP, ENV["APPDATA"] is defined and uses the old Windows user directory in the form of C:\Documents and Settings\<username>\Application Data.

Got a full list of ENV variables under XP?

Btw- if I recall correctly "Documents and Settings" is localized to the user's language, isn't it? (without the shim that Vista,Win7, win8 uses)

fredo6 · 24 Jun 2013, 13:12

We could also substitute <username> in the paths by

Default on Win7, Vista, Win8
All users on XP

At least this will avoid the problem of unicode in the username.

Fredo

thomthom · 24 Jun 2013, 13:16

@fredo6 said:

We could also substitute <username> in the paths by

Default on Win7, Vista, Win8

All users on XP

At least this will avoid the problem of unicode in the username.

And the setting would not be per-use - but per computer. ...which might not be a big issue since the plugin folder is shared by all users.

But again I raise my concern that XP paths might be translated. "All users" I think is "Alle brukere" in Norwegian XP. (I can dig out my old XP machine and test.)

driven · 24 Jun 2013, 13:54

Just found a BIG PROBLEM with the 'TIG' and new 'FREDO' path trick on my mac.

My User 'private' settings for 'BOTH' are now available to any account on my computer or connected to my network.

By stepping up stream, external to ENV["TMPDIR"] you are bypassing the LOCK to other user accounts and network traffic.

So my SFC Password Cookie and my Fredo settings are usable from any account, I do not expect a Plugin to share this type of information with the world.

Please move these back into User Space asap.

This could be a huge problem on a work or university network...

john

slbaumgartner · 24 Jun 2013, 15:24

@driven said:

Just found a BIG PROBLEM with the 'TIG' and new 'FREDO' path trick on my mac.

My User 'private' settings for 'BOTH' are now available to any account on my computer or connected to my network.

By stepping up stream, external to ENV["TMPDIR"] you are bypassing the LOCK to other user accounts and network traffic.

So my SFC Password Cookie and my Fredo settings are usable from any account, I do not expect a Plugin to share this type of information with the world.

Please move these back into User Space asap.

This could be a huge problem on a work or university network...

john

Confirm: the parent of ENV[TMPDIR] has permissions drwxr-xr-x on my Mac - that means it is readable by anyone, though only the owner (me) can write new files there. ENV[TMPDIR] itself is drwx------, which means it is inaccessible to anyone but me. This is not a big deal if you are the only user on your computer, but a serious issue if anyone else has access.

Steve

fredo6 · 24 Jun 2013, 15:45

I think I will default to the old method of DEFPARAM_Dir in the SU Plugins directory

thomthom · 24 Jun 2013, 17:03

Using Win32 API calls I've been able to get short (DOS 8.3 style) path name to appdata. I need to run some tests to make sure it works under XP etc.

fredo6 · 24 Jun 2013, 17:06

Trying to recap

1) On Windows
For the non-ascii username, we have a much more general problem, because Temp directory and very likely any files (skp, materials), are going to be stamped with the username in the file paths. So nothing will work. For instance, I prepare the HTML files of webdialog boxes in the Temp directory. This is also where I put some traces.
--> I will ignore this until Sketchup moves to Ruby 1.9.2 which solves the problem (or thomthom to find a solution!). Too bad for users having a non-ascii name!

For parameters, the usage of LOCALAPPDATA and APPDATA is OK. It is based on users and should correspond to writable areas
--> I will use this method

2) on Mac
For the non-ascii username, apparently there is no issue with non-ascii usernames (to be confirmed), but anyway, the Temp dir is independent, writable and only visible to users. In addition, the ~/ notation allows avoiding having the user name in the path
--> NO issue there

For parameters, there are visibility issues with the parent folder of ENV["TMPDIR"]. So the method cannot be used.
--> I will use driven's method based on a root directory ~/Library/Application Support/ with a hidden folder "LIBFREDO6_DATA_Dir" under which I will create the files and subfolders.

It's interesting to see that we have had a long debate on a basic question which we should expect to be covered by Sketchup itself. I think we should formulate it clearly for the Trimble SU team to address it in SU 2014.

Fredo

thomthom · 24 Jun 2013, 17:26

As for OSX, what about this:

plugin_path = Sketchup.find_support_file('Plugins') path = File.join( plugin_path, '..', '..' ) File.expand_path( path )

Making some assumption here, but tested under SU8 and SU2013 on OSX 10.8.3:

/Library/Application Support/Google SketchUp 8 /Users/thomas/Library/Application Support/SketchUp 2013

thomthom · 24 Jun 2013, 17:34

@fredo6 said:

1) On Windows
For the non-ascii username, we have a much more general problem, because Temp directory and very likely any files (skp, materials), are going to be stamped with the username in the file paths. So nothing will work. For instance, I prepare the HTML files of webdialog boxes in the Temp directory. This is also where I put some traces.
--> I will ignore this until Sketchup moves to Ruby 1.9.2 which solves the problem (or thomthom to find a solution!). Too bad for users having a non-ascii name!

On my Win7 systems I tried, the TEMP/TMP ENV variables returned a short (DOS 8.3 style) path name. These short paths was writable with Ruby 1.8 as the system resolved their real paths. (I assume because the short paths where given to the ANSI version of the system file functions which in turn handled it.)

@fredo6 said:

For parameters, the usage of LOCALAPPDATA and APPDATA is OK. It is based on users and should correspond to writable areas
--> I will use this method

There also include username in the path and may be non-writable. In fact - you are more likely to get issues here because the ENV variable doesn't return a short path - but a path string with replaced characters.

@fredo6 said:

2) on Mac
For the non-ascii username, apparently there is no issue with non-ascii usernames (to be confirmed), but anyway, the Temp dir is independent, writable and only visible to users. In addition, the ~/ notation allows avoiding having the user name in the path
--> NO issue there

Unicode filenames works under OSX with Ruby 1.8 because the file system functions calls handles UTF-8 strings.

@fredo6 said:

For parameters, there are visibility issues with the parent folder of ENV["TMPDIR"]. So the method cannot be used.
--> I will use driven's method based on a root directory ~/Library/Application Support/ with a hidden folder "LIBFREDO6_DATA_Dir" under which I will create the files and subfolders.

What about under '~\Library\Application Support\SketchUp 2013' ? That way the data is isolated from version to version.