Download RBZ file

honkinberry

My preference at this time is for the CURB gem (which relies upon the URI Standard Ruby Library.)

Understood, and I would agree. But as that is not complete, I'm offering my solution as something that can get up and running quite quickly.

I did find this:
https://github.com/tario/imageruby

From which I am in the process of extracting just the BMP writing and reading.
Which reduces down the complexity of my solution to just this:
use built-in File.Binary read and write capability, convert each byte to an integer, store integer as RGB bytes in a BMP file. And then of course reverse that after extraction.
Which is looking to be no more than a couple hundred lines of code.

So yes, not an ideal solution, but with very minimal coding I'll have RBZtoBMP and BMPtoRBZ.

--J

TIG

I still can't to see why you need to jump through all of these extra hoops
Could you just download the RBZ [it's only a renamed ZIP] and then extract/install its contents with the built-in tools anyway...

You really need to have something for the user to confirm - the whole of the raison d'être for javascript/java etc is to stop secretive invasive data transfer in/out www<>computer... In any case it's good manners... the user should feel in control, your tool advises about update availability, but they choose to update ?
You could use a tiny webdialog located 'off-screen', so there is no need for a massive web-page... the trick is by-passing the main browser's checker dialog...
This auto-opens a download prompt from an 'off-screen' web-dialog...

<html><head></head><body><script type="text/javascript">
window.onload = function(){document.location = 'http://sketchucation.com/forums/download/file.php?id=99653';}
</script></body></html>

I haven't tried this... But you could also think about definitions.load_from_url(url) with a 'LoadHandler', this will fail because the file type isn't a skp... BUT it might leave the downloaded RBZ file intact in the user's TEMP folder [easily found]... EDIT: I've now tried it it only works with components skp files, other file types stop it before it starts downloading ! Plan B...

driven

@Tig,
I have tested that in the past, and on a mac it works if you have a second/alternate download manager, it won't use safari's unless you use open_url, but that's not in the background (not a bad thing).
When you do have a downloader available, the javascript function, never closes and SU runs at 96%, doing nothing until you quit the process. Not ideal.

@ honkinberry I personally think you will encounter lots of issues with macs built in security,
The extension name is primarily for applying the right file decoration, unix checks if the contents is valid, [especially since the hacker managed to bury a shell script in a PNG, posted on a forum, with "hey guys, has anyone seen this new iPhone secret pic, I won't download it because it looks suspicious".
Loads of people went ahead and downloaded it, and it sent itself to every contact book address and they opened it and... etc...]

If your an admin on your mac, you can do all sorts of masking, hiding, moving, on your mac. But that won't mean it will work on anyone else's, some might some won't ever... unless there is a user request to allows it.

I think Fredo's method is the best for user plugins at the moment, or just pop an options box asking if the user wants the download.

I use 'Little Snitch' and I get notifications for every attempted download from the system, I remove software that hasn't asked me if it can do that.

I don't know what's possible on Windows but you can have SU automatically install any .rbz that's download to a mac.
i.e. download a .rbz (programatically or manually), SU opens on Prefs Dialog, then opens on the download, then installs the download. You just need to set up an Automator Service to do that, and call it from your ruby. System may pop a notice or not depending on the receiving Users security prefs...

john

honkinberry

I think there's still some confusion on what I am proposing, so let me try this again.

Again, Problem -- no ability for a WebDialog (on a Mac) to say, "Hey, trusted user, there is an updated version of this plugin available, would you like to install it now?" (okay, yes, I can open up another giant browser window to my download page, and then the user can click the downloaded file, and might then have to choose to open it with Sketchup, or wade through Preferences.... omg! I just want an easy updater, right???)

So, Solution -- I encode the RBZ as a BMP file -- this is simple process, that doesn't need to be done with Sketchup Ruby, but might as well. The Binary File functions are used to read in the file, and then store each byte as an integer, into a valid BMP file.
(This is technically a form of Steganography. It will defeat any security system, as it is a legitimate BMP -- it will just look like static to the eye. One could technically go a level further and embed the binary file within an actual image, being a level of Steganography used by spy agencies and terrorists and such, but I'm not proposing going that far.)
Now a BMP file, it is easy to import into a blank SKP file.
And as an SKP file, I can download it in the background with load_from_url (heck, even with a progress bar and full error control!).
Once that file is downloaded, I can write out the image using TextureWriter. Again, there is no security issues, as this is a legitimate BMP file.
Lastly, a little Ruby routine again uses the binary File reader to read in the binary data, and then write it back, this time as a RBZ file.
Alas, now a local RBZ file, it can be passed off to install_from_archive.

And most importantly, I'm certainly not saying this is ideal!
The last thing I want is to write a couple thousand lines of code, only to have SU9 come out with http support in the install_from_archive function.
I'm merely proposing that in only writing a simple binary BMP reader/writer, I can be up and running with a workable background plugin updater, cross-platform.

But hey, it's cool if you don't like the idea. You certainly don't have to use it to distribute updates to your own plugins. But you have confirmed for me that there is not a workable alternative, so at least I'm not completely chasing windmills.

Cheers,

--J

TIG

This approach requires you to make special version of every RBZ archive as an 'image' and then use special 'unpacking' code at the Ruby end...

There are alternatives that can use just SketchUp's basic Ruby and OS tools on an RBZ file itself... At the download Ruby end you have your code write a short temporary 'script' to download the RBZ in the background and install it:-

On a PC - write a temporary VBS script that you then execute with UI.operURL() to download the RBZ, this is straightforward provided the html path ends with the 'filename.rbz' and it isn't redirected by ..?id=123456 etc. The downloaded file can go into the user's Temp folder [to check it's completed before installing it you can add a temp 'done' file maker after the main VBS code and Ruby can then wait until that exists in the Temp folder, then it can delete it and continue with the installation - trap for failure etc so it doesn't lock up], then you can complete the installation of it from using the RBZ archive installing code in the v8 API. Here's an example:

Call DownloadFile("http://..../abc.rbz")
' The Ruby side adjusts this first line to suit the actual url.
' The Ruby only runs if url it has only ends in say .rbz !
' the function will then download that file to the user's usual temp folder.
Function DownloadFile(DownloadUrl)
	'Get name of file from url (whatever follows the final forwardslash "/")
	Dim arURL, outputArray, x, FileName, FileSaveLocation
	arURL = Split(DownloadUrl,"/",-1,1)
	If arURL(UBound(arURL)) = "" Then 'if there is a trailing forwardslash
		FileName = arURL(UBound(arURL) -1)
	Else
                ' in case there's a redirect, but with a file-name "file=abc.rbz" etc
		outputArray = Split(arURL(UBound(arURL)), "=")
		for each x in outputArray
		  FileName = x
		next
	End If
	'Get temp folder location
	Dim oFS, TempDir
	Set oFS = CreateObject("Scripting.FileSystemObject")
	Set TempDir = oFS.getSpecialFolder(2)
	FileSaveLocation = TempDir & "\" & FileName
	' Wscript.Echo FileSaveLocation ' unrem this line to test to see temp file path
	' Fetch the file
	Dim oXMLHTTP, oADOStream
	Set oXMLHTTP = CreateObject("MSXML2.XMLHTTP")
	oXMLHTTP.open "GET", DownloadUrl, false
	oXMLHTTP.send()
	If oXMLHTTP.Status = 200 Then
		Set oADOStream = CreateObject("ADODB.Stream")
		oADOStream.Open
		oADOStream.Type = 1 'adTypeBinary
		oADOStream.Write oXMLHTTP.ResponseBody
		oADOStream.Position = 0 'Set the stream position to the start
	If oFS.Fileexists(FileSaveLocation) Then oFS.DeleteFile FileSaveLocation
		Set oFS = Nothing
		oADOStream.SaveToFile FileSaveLocation
		oADOStream.Close
		Set oADOStream = Nothing
	End if
	Set oXMLHTTP = Nothing
End Function

On a MAC - you do much the same as the PC, but now you'll need to execute a temporary Applescript to do the silent download. Ruby first compiles the 'reference' details [URL/TempFile] and makes a line of code... MAC users can advise best on this [driven?] - something like:
tell application "URL Access Scripting" to download "#{theURL}" to "#{theTempFile}" which I think can be run on a MAC Ruby as something like system...`` ?

honkinberry

I'm well aware of how to download a file use ADO Data Streams.
Would you like to run our Tech Support for a day, and deal with the myriad of people who have Security issues with it? It's the oADOStream.SaveToFile that most commonly gets snagged, and just doesn't have the ability to save to where I'd like it to. Then you end up using VBS File objects to move it around, or CIM.... oh, and wait, then you have the File object locked down, or the Domains where VBS is just not responding at all. In the mood to do a remote session with a machine in China or Dubai, where their internet is really locked down, and you have to use a Start menu in a foreign language? Trust me, it's not fun, and ADO is a security nightmare waiting to happen.

I've just spent the last couple weeks re-engineering our Component delivery system to not use ADO anymore, and oh my gosh is it a better system. Loving load_from_url, such a more elegant solution.

Anyway, I'll have some code for y'all within about a week or so, and you can judge then whether it's more or less bulletproof than any of these other solutions.

--J

Dan Rathbun

@honkinberry said:

Loving load_from_url, such a more elegant solution.

We'll if I was tasked with using a component via load_from_url(), to d/l and install a plugin... I would not convert Ruby to BMP or any other image.

Ruby scripts are text.. even scrambled rbs files.

I would TRY saving the plugin information (folder & file names etc.,) and the script text blocks themselves, into an attribute dictionary, and write out a "plugin" component definition, that has only this attribute dictionary.

Then download it via load_from_url using a custom LoadHandler class.
The onSuccess() callback would just use standard Ruby Dir and File classes to create the directories, and write out the script text files, using the fields of the attributes attached to the component definition.

If the definition contained images (within it's entities, or textured materials,) ... I would assume them to be toolbar button images etc., and they would be written out as images (using a custom TextureWriter class.)

When done, the definition is deleted, and the user is asked if they wish to load the plugin at that time.

ADD: I suppose an entire RBZ file could be written as one attribute value... it's worth a try.

TIG

The definitions.load_from_url works just fine - for components - I've used it a lot... BUT you can't get it to do much else...
The download will fail if the .skp isn't 'valid' and the downloaded file is auto-deleted !

Having an effectively 'empty' skp, with an attribute-dictionary containing the data might work, but isn't there a data size limit ?
BUT assuming that the RBZ file's data is of an acceptable size [or it is spread across several attributes which get recombined later]... then the 'Dan' trick might just work...

honkinberry

So rather than encode a single RBZ file as a BMP, and allow SU's built in install_from_archive,
Dan is proposing I maintain a file schema of the... (at last count 100+) files involved in my plugin, write them all out to the correct location.... agh, that sounds frustrating. How are you proposing I maintain the file manifest? I suppose another unusual trick, like storing it as Text objects in the Model?

I'll get you guys to come around to my idea... eventually....

--J

Dan Rathbun

@honkinberry said:

I suppose another unusual trick, like storing it as Text objects in the Model?

It is quite normal to store text data in an AttributeDictionary

The approach I am thinking about, does not actually add entities into the model's entities collection.

AND... this is this first we are hearing that you have 100+ files.

We are just brain-storming here.

Do what you wish.. have fun.

TIG

Out of interest I tried writing some methods that to my surprise worked effectively.
One that encodes the RBZ file as hex strings in a number of attribute keys in a dictionary attached to a SKP - it's an all but empty file otherwise [a guide-point is need so it's not seem as empty by the loader].
The others are as some connected methods used in any SKP later on - one that uses load_from_url to add a temporary-definition of the remote SKP into the current model.definitions without need for further user interaction [after a callback action from a webdialog button], the definitions attributes are got and recombined into the needed data, and the definition is deleted from the model.
The data is written out into a RBZ in the user's Temp folder and then auto-installed from that archive...
So to recap, you make a SKP containing the RBZ's data as a attribute key values, you temporarily load that SKP from a url and extract the RBZ's data, recombining it into a local file in a Temp folder, from where it can be auto-installed into Plugins...
The only downside I can see is that the SKP is considerably bigger than the RBZ as the data is 'hexed' ~3.7x, but a 2Mb RBZ would be quite unusual ?

honkinberry

Tig, that's brilliant, I wouldn't have thought of that.
That's very much the same approach, just a bit cleaner.
Yes of course there's a bit of loss in file file size, but it's all about the capability gain here.
Having an integrated update solution gives a much more professional and complete experience to the user.
Very nice. Would you care to share some of your code?

--J

Dan Rathbun

@honkinberry said:

Tig, that's brilliant, I wouldn't have thought of that.

I did.

to TIG for proving the concept.

@TIG: Do you need to hex encode it?

Would it help you, if you had Zip / Unzip methods in Ruby ? (I can PM them to you.)

danielbowring

@dan rathbun said:

Do you need to hex encode it?

Unless you're sure there is no null character, you'll have to hex encode it or Sketchup will only store up to the first null character. Binary files will probably have plenty of nulls in them.

Here's a sample of how SketchUp deals with the situation:
` nullstr = 'My Test ' + 0.chr + ' String'
puts nullstr.inspect

=> "My Test \000 String"

puts nullstr.length

=> 16

Sketchup.active_model.set_attribute('test', 'nullstr', nullstr)
nullstr2 = Sketchup.active_model.get_attribute('test', 'nullstr')
puts nullstr2.inspect

=> "My Test "

puts nullstr2.length

=> 8

puts 'My Test '.length

=> 8`

TIG

I already tested it, and I used:
data = open(path,"rb"){|io|io.read}.unpack('H*').to_s
The 'path' is the file path to the RBZ file.
Although an attribute value can be an array it is made into a string because it could need splitting into 'chunks', because there's an attribute value size limit [I use ~20,000 chars] - you will get Bugsplats if it's much bigger [test it to see] - these 'chunks' are written into sequentially named attribute keys/values attached to the current model...

Later after the 'attributed' model's SKP has been added to a new model's definition using the API's load_from_url() methods accessed via awebdialog etc. It's then easy to extract the sequentially named 'chunks' from that new definition's attribute keys/values, recombining those into a single string, here named 'data', and then to remake the binary code 'string' using:
datab = [data].pack('H*')
This 'string' is then written into a new temp RBZ file...
Finally the temp definition can be deleted, and the RBZ file auto-installed using the API methods added in v8M2, before it too is deleted from the user's Temp folder...

The only downside to this is that the 'empty' SKP that has been 'attributed' is ~3.7x bigger than the original RBZ file, simply because of the unpack of its data into hex... BUT on the upside you can use it to encrypt, download and install an RBZ [masquerading as a SKP] from a URL without any user dialogs intervening etc; simply from a single user button-click in a webdialog running a callback to do the load_from_url(), or less secure when the page opens and no user involvement at all...

Dan Rathbun

I guess my question should be more specific.

Can the file's data be escaped rather than hex encoded ?
Could the printable characters could remain as they are... but the non-printables (including null,) could be escaped ?

honkinberry

Thanks TIG for the clear direction!

It absolutely does work, with my quick and dirty (no error handling yet) version like so:


def encoderbz ()
	chunksize = 2**14 # attrib data doesn't like more than ~20kb, so we limit each chunk to 16k

	if ( file = UI.openpanel "Select RBZ file to encode" )
		data = open(file) {|io|io.read}.unpack('H*').to_s # hex encode, converted to string
		model = Sketchup.active_model
		i = 0
		while data and data != ""
			i = i + 1 # counter
			d = "d" + i.to_s # name of attribute
			chunk = data[0,chunksize] # chunk of data
			data = data[chunksize..-1] # trim master data
			model.set_attribute 'rbz',d,chunk
		end # while
		# and set the next one to nil in case user is updating with a smaller rbz
		i = i + 1
		d = "d" + i.to_s
		model.set_attribute 'rbz',d,nil

		UI.messagebox "The selected RBZ file has been successfully encoded into the current model."
	end # if
end # encoderbz

def decoderbz ()
	model = Sketchup.active_model
	i = 1 # counter
	d = "d" + i.to_s # name of attribute
	data = "" # to assemble the chunks
	while dx = model.get_attribute('rbz',d)
		data = data + dx

		i = i + 1
		d = "d" + i.to_s # name of attribute
	end # while

	if ( data != "" )
		datab = [data].pack('H*')
		if ( filename = UI.savepanel "Save RBZ As...","","*.rbz" )
			file = File.open(filename, "w")
			file.write(datab)
			file.close
			
			UI.messagebox "The enclosed RBZ has been successfully saved out as an RBZ file."
		end # if
	end # if
end # decoderbz

Yes, there is a size overhead of about 4x, but the hex encoding does feel a very bulletproof way of doing this.
Certainly a bit easier than my initial thought of encoding as an image, but same basic idea.

Thanks all for the help and patience! This definitely raises the quality of the plug-in to a more professional level.

--J

TIG

Since my last ideas I think the better way to ensure an 'empty' SKP.
It'll work in any SKP too...
In code...
Inside a model_start/commit block...
Make a new definition.
Set its name to RBZ.basename + '.skp'
Add a cpoint to definition.entities.
Add the attributes to the definition.
Do a 'save_as' on definition into the same folder as the RBZ file.
The new SKP has the RBZ attached as attribute key[s]...
Do a definition.entities.clear! so it's removed from the Browser...

Done.

The import is much as you outlined...

honkinberry

Well, snap, not working on the PC side.
Mac side is flawless.

Given my 106k RBZ file, and this line of code:

data = open(file) {|io|io.read}.unpack('H*').to_s

On the Mac, data.length is 207954
While on the PC, data.length is 18216

And then there's issue on the other end too.
I'll have to assume it's here:

datab = [data].pack('H*')

It's adding a bunch of 0D's in there, good ol' Chr(13), carriage returns.
At least it seems it is there, I suppose it could be in the actual file.write(datab) ?
But it's adding about 352 of them in that 106k file.

Soooooooooo close!
Any brilliant ideas?

(And in case you're wondering how it is on the Mac side where it works? Um, it's amazing, in a word.)

--J

TIG

Hint:
File.open(filename, "w")
writes binary OR strings on MAC/Unix BUT only strings on PC [messing up any binary-data]
BUT
File.open(filename, "w**b**")
writes binary-data properly on all platforms ? ...