Download RBZ file

honkinberry

I'm so close to a really incredible idea here, could really use some help, especially if what I'm doing is futile.

Problem: No way for my plugin to automatically install updates.

Solution: Distribute RBZ file converted to an image file.

It would go something like this:

Rename .RBZ file to .RAW file
Convert .RAW file to .PNG (it will look like static at this point)
Import .PNG as Image
Save file as .SKP
Client downloads .SKP using definitions.load_from_url
Client saves out .PNG texture using TextureWriter
Saved .PNG converted back to .RAW
.RAW file renamed to .RBZ
Updated plugin installed with Sketchup.install_from_archive

Right now I am at step 7. (!!!)
The image file output by the TextureWriter is indeed the exact same file originally put in. (!!!!!)
Only now, I can't find a way to convert the PNG back to a RAW file.
I've been fiddling with different formats, BMP, TIFF, but can't find a good single resource for true lossless conversion in both directions. To proceed with this any further, it seems time to integrate something like libpng. TIFF also seems very promising, but again, I can't find a solid way to convert a TIFF back to a RAW.
But interestingly!!!! -- The TIFF is clearly very close to to the RAW format, you can actually see the PK header and folder names right there in the file.

So now it's looking like the next step is to port libpng to Ruby (or find a working one somewhere), and develop a Packager and Extractor.
But good time to pause and see what you all think!

Cheers,

--J

Dan Rathbun

That is just plain crazy.

Example downloading and installing Aerilius' Texture Resizer:

(1) UI.openURL('http://sketchucation.com/forums/download/file.php?id=99653')

(2) Wait for file to be downloaded. (A loop is needed here too see when the filesize stops getting larger.)

(3) Windows XP:
Sketchup.install_from_archive(File.join(ENV['USERPROFILE'],'My Documents/Downloads/ae_TextureResizer_1_4_6.rbz'))

Jim

Seems like Dana's downloader is a better solution.

https://github.com/danawoodman/google-sketchup-file-downloader

I see thomthom has contributed - is it ready for general use?

TIG

Dana's project seems to have completely stalled - if not died altogether...
There are some other ideas in embryo...
So we'll just have to wait and see where it all ends up...

Dan Rathbun

TBD also made a change, and requested a pull, but Dana never did do the merge.

honkinberry

just plain crazy? wow, there's some strong words.

Dan's solution, like usual, doesn't work on a Mac.
It looks like ENV['HOME']/"Downloads" would be a starting point there, but I just don't like the sketchy looping until "filesize stops getting larger." (let alone the giant web browser window that pops up)

And Dana's wildly complicated and non-working solution? What's there to say?

Meanwhile, it's looking like the BMP file will work best for my purposes, as it should have minimal modification in creating/removing the header.

So anyway, I guess I'll just keep this one to myself for now.
Cheers,

--J

Dan Rathbun

@honkinberry said:

just plain crazy? wow, there's some strong words.

It was not meant as an insult. Just that your image solution seems rather convoluted, to me.

@honkinberry said:

Dan's solution, like usual, doesn't work on a Mac.
It looks like ENV['HOME']/"Downloads" would be a starting point there,...

One way I use to create a PC HOME environment variable is:
ENV['HOME']= ENV['USERPROFILE'] unless RUBY_PLATFORM =~ /(darwin)/i || ENV['HOME']
That way path expansion using " ~" and Ruby's File.expand_path() can work on the PC.

One sticking point on the PC, under MS Windows: The name of the "Documents" or "My Documents" directory is language localized.

This raises one of the peeves in the SketchUp API. Standard and User configurable path variables which I'd like as module constants within the UI module.
Example:
UI::DOWNLOAD_DIR
.. or perhaps a UI::PATH sub-module with methods to get and set certain paths.

@honkinberry said:

... but I just don't like the sketchy looping until "filesize stops getting larger."

Granted... perhaps we need a better way to determine when the file is done downloading.

@honkinberry said:

... (let alone the giant web browser window that pops up)

I wonder if we could write a SketchUp WebDialog that could serve as a downloader?

@honkinberry said:

And Dana's wildly complicated and non-working solution? What's there to say?

Well on the Mac... is the cURL executable provided "out-of-the-box" ?
(Dana's utility was firstly meant as a PC solution where the cURL DLL is not provided.)

But, IMHO, what's the difference in having the user just go download and install the actual cURL lib, and a utility that uses a subset of it's code?

I'd rather install [and I have,] the actual library. And then find or modify a Ruby binding for the actual cURL library.

My preference at this time is for the CURB gem (which relies upon the URI Standard Ruby Library.)

honkinberry

@unknownuser said:

My preference at this time is for the CURB gem (which relies upon the URI Standard Ruby Library.)

Understood, and I would agree. But as that is not complete, I'm offering my solution as something that can get up and running quite quickly.

I did find this:
https://github.com/tario/imageruby

From which I am in the process of extracting just the BMP writing and reading.
Which reduces down the complexity of my solution to just this:
use built-in File.Binary read and write capability, convert each byte to an integer, store integer as RGB bytes in a BMP file. And then of course reverse that after extraction.
Which is looking to be no more than a couple hundred lines of code.

So yes, not an ideal solution, but with very minimal coding I'll have RBZtoBMP and BMPtoRBZ.

--J

TIG

I still can't to see why you need to jump through all of these extra hoops
Could you just download the RBZ [it's only a renamed ZIP] and then extract/install its contents with the built-in tools anyway...

You really need to have something for the user to confirm - the whole of the raison d'être for javascript/java etc is to stop secretive invasive data transfer in/out www<>computer... In any case it's good manners... the user should feel in control, your tool advises about update availability, but they choose to update ?
You could use a tiny webdialog located 'off-screen', so there is no need for a massive web-page... the trick is by-passing the main browser's checker dialog...
This auto-opens a download prompt from an 'off-screen' web-dialog...

<html><head></head><body><script type="text/javascript">
window.onload = function(){document.location = 'http://sketchucation.com/forums/download/file.php?id=99653';}
</script></body></html>

I haven't tried this... But you could also think about definitions.load_from_url(url) with a 'LoadHandler', this will fail because the file type isn't a skp... BUT it might leave the downloaded RBZ file intact in the user's TEMP folder [easily found]... EDIT: I've now tried it it only works with components skp files, other file types stop it before it starts downloading ! Plan B...

driven

@Tig,
I have tested that in the past, and on a mac it works if you have a second/alternate download manager, it won't use safari's unless you use open_url, but that's not in the background (not a bad thing).
When you do have a downloader available, the javascript function, never closes and SU runs at 96%, doing nothing until you quit the process. Not ideal.

@ honkinberry I personally think you will encounter lots of issues with macs built in security,
The extension name is primarily for applying the right file decoration, unix checks if the contents is valid, [especially since the hacker managed to bury a shell script in a PNG, posted on a forum, with "hey guys, has anyone seen this new iPhone secret pic, I won't download it because it looks suspicious".
Loads of people went ahead and downloaded it, and it sent itself to every contact book address and they opened it and... etc...]

If your an admin on your mac, you can do all sorts of masking, hiding, moving, on your mac. But that won't mean it will work on anyone else's, some might some won't ever... unless there is a user request to allows it.

I think Fredo's method is the best for user plugins at the moment, or just pop an options box asking if the user wants the download.

I use 'Little Snitch' and I get notifications for every attempted download from the system, I remove software that hasn't asked me if it can do that.

I don't know what's possible on Windows but you can have SU automatically install any .rbz that's download to a mac.
i.e. download a .rbz (programatically or manually), SU opens on Prefs Dialog, then opens on the download, then installs the download. You just need to set up an Automator Service to do that, and call it from your ruby. System may pop a notice or not depending on the receiving Users security prefs...

john

honkinberry

I think there's still some confusion on what I am proposing, so let me try this again.

Again, Problem -- no ability for a WebDialog (on a Mac) to say, "Hey, trusted user, there is an updated version of this plugin available, would you like to install it now?" (okay, yes, I can open up another giant browser window to my download page, and then the user can click the downloaded file, and might then have to choose to open it with Sketchup, or wade through Preferences.... omg! I just want an easy updater, right???)

So, Solution -- I encode the RBZ as a BMP file -- this is simple process, that doesn't need to be done with Sketchup Ruby, but might as well. The Binary File functions are used to read in the file, and then store each byte as an integer, into a valid BMP file.
(This is technically a form of Steganography. It will defeat any security system, as it is a legitimate BMP -- it will just look like static to the eye. One could technically go a level further and embed the binary file within an actual image, being a level of Steganography used by spy agencies and terrorists and such, but I'm not proposing going that far.)
Now a BMP file, it is easy to import into a blank SKP file.
And as an SKP file, I can download it in the background with load_from_url (heck, even with a progress bar and full error control!).
Once that file is downloaded, I can write out the image using TextureWriter. Again, there is no security issues, as this is a legitimate BMP file.
Lastly, a little Ruby routine again uses the binary File reader to read in the binary data, and then write it back, this time as a RBZ file.
Alas, now a local RBZ file, it can be passed off to install_from_archive.

And most importantly, I'm certainly not saying this is ideal!
The last thing I want is to write a couple thousand lines of code, only to have SU9 come out with http support in the install_from_archive function.
I'm merely proposing that in only writing a simple binary BMP reader/writer, I can be up and running with a workable background plugin updater, cross-platform.

But hey, it's cool if you don't like the idea. You certainly don't have to use it to distribute updates to your own plugins. But you have confirmed for me that there is not a workable alternative, so at least I'm not completely chasing windmills.

Cheers,

--J

TIG

This approach requires you to make special version of every RBZ archive as an 'image' and then use special 'unpacking' code at the Ruby end...

There are alternatives that can use just SketchUp's basic Ruby and OS tools on an RBZ file itself... At the download Ruby end you have your code write a short temporary 'script' to download the RBZ in the background and install it:-

On a PC - write a temporary VBS script that you then execute with UI.operURL() to download the RBZ, this is straightforward provided the html path ends with the 'filename.rbz' and it isn't redirected by ..?id=123456 etc. The downloaded file can go into the user's Temp folder [to check it's completed before installing it you can add a temp 'done' file maker after the main VBS code and Ruby can then wait until that exists in the Temp folder, then it can delete it and continue with the installation - trap for failure etc so it doesn't lock up], then you can complete the installation of it from using the RBZ archive installing code in the v8 API. Here's an example:

Call DownloadFile("http://..../abc.rbz")
' The Ruby side adjusts this first line to suit the actual url.
' The Ruby only runs if url it has only ends in say .rbz !
' the function will then download that file to the user's usual temp folder.
Function DownloadFile(DownloadUrl)
	'Get name of file from url (whatever follows the final forwardslash "/")
	Dim arURL, outputArray, x, FileName, FileSaveLocation
	arURL = Split(DownloadUrl,"/",-1,1)
	If arURL(UBound(arURL)) = "" Then 'if there is a trailing forwardslash
		FileName = arURL(UBound(arURL) -1)
	Else
                ' in case there's a redirect, but with a file-name "file=abc.rbz" etc
		outputArray = Split(arURL(UBound(arURL)), "=")
		for each x in outputArray
		  FileName = x
		next
	End If
	'Get temp folder location
	Dim oFS, TempDir
	Set oFS = CreateObject("Scripting.FileSystemObject")
	Set TempDir = oFS.getSpecialFolder(2)
	FileSaveLocation = TempDir & "\" & FileName
	' Wscript.Echo FileSaveLocation ' unrem this line to test to see temp file path
	' Fetch the file
	Dim oXMLHTTP, oADOStream
	Set oXMLHTTP = CreateObject("MSXML2.XMLHTTP")
	oXMLHTTP.open "GET", DownloadUrl, false
	oXMLHTTP.send()
	If oXMLHTTP.Status = 200 Then
		Set oADOStream = CreateObject("ADODB.Stream")
		oADOStream.Open
		oADOStream.Type = 1 'adTypeBinary
		oADOStream.Write oXMLHTTP.ResponseBody
		oADOStream.Position = 0 'Set the stream position to the start
	If oFS.Fileexists(FileSaveLocation) Then oFS.DeleteFile FileSaveLocation
		Set oFS = Nothing
		oADOStream.SaveToFile FileSaveLocation
		oADOStream.Close
		Set oADOStream = Nothing
	End if
	Set oXMLHTTP = Nothing
End Function

On a MAC - you do much the same as the PC, but now you'll need to execute a temporary Applescript to do the silent download. Ruby first compiles the 'reference' details [URL/TempFile] and makes a line of code... MAC users can advise best on this [driven?] - something like:
tell application "URL Access Scripting" to download "#{theURL}" to "#{theTempFile}" which I think can be run on a MAC Ruby as something like system...`` ?

honkinberry

I'm well aware of how to download a file use ADO Data Streams.
Would you like to run our Tech Support for a day, and deal with the myriad of people who have Security issues with it? It's the oADOStream.SaveToFile that most commonly gets snagged, and just doesn't have the ability to save to where I'd like it to. Then you end up using VBS File objects to move it around, or CIM.... oh, and wait, then you have the File object locked down, or the Domains where VBS is just not responding at all. In the mood to do a remote session with a machine in China or Dubai, where their internet is really locked down, and you have to use a Start menu in a foreign language? Trust me, it's not fun, and ADO is a security nightmare waiting to happen.

I've just spent the last couple weeks re-engineering our Component delivery system to not use ADO anymore, and oh my gosh is it a better system. Loving load_from_url, such a more elegant solution.

Anyway, I'll have some code for y'all within about a week or so, and you can judge then whether it's more or less bulletproof than any of these other solutions.

--J

Dan Rathbun

@honkinberry said:

Loving load_from_url, such a more elegant solution.

We'll if I was tasked with using a component via load_from_url(), to d/l and install a plugin... I would not convert Ruby to BMP or any other image.

Ruby scripts are text.. even scrambled rbs files.

I would TRY saving the plugin information (folder & file names etc.,) and the script text blocks themselves, into an attribute dictionary, and write out a "plugin" component definition, that has only this attribute dictionary.

Then download it via load_from_url using a custom LoadHandler class.
The onSuccess() callback would just use standard Ruby Dir and File classes to create the directories, and write out the script text files, using the fields of the attributes attached to the component definition.

If the definition contained images (within it's entities, or textured materials,) ... I would assume them to be toolbar button images etc., and they would be written out as images (using a custom TextureWriter class.)

When done, the definition is deleted, and the user is asked if they wish to load the plugin at that time.

ADD: I suppose an entire RBZ file could be written as one attribute value... it's worth a try.

TIG

The definitions.load_from_url works just fine - for components - I've used it a lot... BUT you can't get it to do much else...
The download will fail if the .skp isn't 'valid' and the downloaded file is auto-deleted !

Having an effectively 'empty' skp, with an attribute-dictionary containing the data might work, but isn't there a data size limit ?
BUT assuming that the RBZ file's data is of an acceptable size [or it is spread across several attributes which get recombined later]... then the 'Dan' trick might just work...

honkinberry

So rather than encode a single RBZ file as a BMP, and allow SU's built in install_from_archive,
Dan is proposing I maintain a file schema of the... (at last count 100+) files involved in my plugin, write them all out to the correct location.... agh, that sounds frustrating. How are you proposing I maintain the file manifest? I suppose another unusual trick, like storing it as Text objects in the Model?

I'll get you guys to come around to my idea... eventually....

--J

Dan Rathbun

@honkinberry said:

I suppose another unusual trick, like storing it as Text objects in the Model?

It is quite normal to store text data in an AttributeDictionary

The approach I am thinking about, does not actually add entities into the model's entities collection.

AND... this is this first we are hearing that you have 100+ files.

We are just brain-storming here.

Do what you wish.. have fun.

TIG

Out of interest I tried writing some methods that to my surprise worked effectively.
One that encodes the RBZ file as hex strings in a number of attribute keys in a dictionary attached to a SKP - it's an all but empty file otherwise [a guide-point is need so it's not seem as empty by the loader].
The others are as some connected methods used in any SKP later on - one that uses load_from_url to add a temporary-definition of the remote SKP into the current model.definitions without need for further user interaction [after a callback action from a webdialog button], the definitions attributes are got and recombined into the needed data, and the definition is deleted from the model.
The data is written out into a RBZ in the user's Temp folder and then auto-installed from that archive...
So to recap, you make a SKP containing the RBZ's data as a attribute key values, you temporarily load that SKP from a url and extract the RBZ's data, recombining it into a local file in a Temp folder, from where it can be auto-installed into Plugins...
The only downside I can see is that the SKP is considerably bigger than the RBZ as the data is 'hexed' ~3.7x, but a 2Mb RBZ would be quite unusual ?

honkinberry

Tig, that's brilliant, I wouldn't have thought of that.
That's very much the same approach, just a bit cleaner.
Yes of course there's a bit of loss in file file size, but it's all about the capability gain here.
Having an integrated update solution gives a much more professional and complete experience to the user.
Very nice. Would you care to share some of your code?

--J

Dan Rathbun

@honkinberry said:

Tig, that's brilliant, I wouldn't have thought of that.

I did.

to TIG for proving the concept.

@TIG: Do you need to hex encode it?

Would it help you, if you had Zip / Unzip methods in Ruby ? (I can PM them to you.)