Scrambler issue

thomthom

In fact there are so many long lines of code in our script which weights nearly 500k

Five hundred thousand character long lines? Did I read that correctly? How do you end up with lines that long? Auto-generated code?
Is is embedded resources (binary data)?

If you cannot address the line lengths (I don't fully understand why this wouldn't be possible) then you'd have to find a way to scramble or encrypt your files yourself, have a proxy loader that reads your files and descrabmle/decrypt and then evaluate the code. You can do this as easily as base64 encoding it or as complicated as you want with true encryption etc. You can do the loading and decrypting in a Ruby C extension to hide the loading logic.

However, I really would recommend focusing on your source code line length. I struggle to see any reason for source code to be more than a couple of hundred characters long - none the less in the thousands!

Wait.... are you talking about number of lines? Because that'd make more sense. I was talking about number of characters per line.

WhyDi

You're right I'm talking about numbers of long lines, not a single line of five hundred thousand of chars. But why not? might be a way more to prevent code from being easily read.
Scramble or encrypt the script is not the issue and I fully understand your recommendation. I may end up with shortening lines length but would have liked to know more about the process you describe. My understanding is c++ reads and unscramble my data file then sends the result -which is a ruby script- straightforward to Sketchup. Am I right?
If so where can I find information about such process ? may you help me to go through?

thomthom

@whydi said:

If so where can I find information about such process ? may you help me to go through?

This would be a rare thing to do. I know of only one or two that do the base64 hashing, and one other developer that use their own. But none have fully described in detail the process.
I also have no specific encryption method to recommend because I've never had the need to do this.

I did debug this issue at some point, and I ran a script on the extension source that basically went through line by line and counted the size. In the two cases I had there where at max 5 lines that exceeded the limit - and they where all due to embedding binary or HTML data. I'd urge you to first do this simple assessment of your source. If it's only a handful of lines then that's so much easier to fix than rolling your own obfuscating system.
Another caveat with rolling your own obfuscating system is that you'd make debugging a nightmare for yourself.

WhyDi

Hi thomthom,
I agree with you but really meet hard time with that scrambler. have a look. The following code crashes on SU8 8.0.14346 but works well on 2016.
mainFile.rb


require 'sketchup.rb'
module Cie_qwerty
	module Ext_azerty
				
		#load "C;/Users/uName/SketchUp/Ruby/test.rb"
		Sketchup;;load "C;/Users/uName/SketchUp/Ruby/testSC"
		
		puts "------------------------------------------start"
		print_test()
		puts "------------------------------------------ready"

	end
end

test.rb


require 'sketchup'
module Cie_qwerty
	module Ext_azerty
	
		def self.print_test()
			puts "print test"
		end
		
	end
end

then wScrambler.exe test.rb
then rename test.rbs testSC.rbs
Sketchup::load or Sketchup::require doesn't matter
testSC.rbs or testSC doesn't matter

FIRST... load "C:/Users/NeW/SketchUp/Ruby/test.rb"
load SU8 : run correctly
load SU2016 : run correctly

SECOND... Sketchup::load "C:/Users/NeW/SketchUp/Ruby/testSC"
Unload then reload SU8: crash
Unload then reload SU2016 : works well

Ruby console results

SU8 one single line and crash. nothing more


------------------------------------------start

SU2016 all right


------------------------------------------start
print test
------------------------------------------ready

So what is wrong in my code?

have a good day

thomthom

Hm....

That is strange.

However, I do not see a crash in SU8. I get a Ruby error - but not crash.


require 'C;/Users/tthomas2/Desktop/scram/main.rb'
------------------------------------------start
Error; #<NoMethodError; undefined method `print_test' for Cie_qwerty;;Ext_azerty;Module>
(eval)
(eval);0;in `require'
(eval);0

However, I realize this is due to a bug in older SketchUp where Sketchup.require (Sketchup.load is an alias) will evaluate the content of the file directly into the current namespace.

So in your code you ended up with something like this:


module Cie_qwerty
 module Ext_azerty
  module Cie_qwerty
   module Ext_azerty
     def print_test

The workaround is to ensure that you do all require outside your namespace:


require 'sketchup.rb'
Sketchup;;require "C;/Users/uName/SketchUp/Ruby/testSC"
module Cie_qwerty
   module Ext_azerty
      
      puts "------------------------------------------start"
      print_test()
      puts "------------------------------------------ready"

   end
end

This made the file load correctly for me.

Btw, I never saw a crash - did you mean Ruby errors? Quite a significant different - a crash is when the application terminate (with SketchUp you get a BugSplat dialog).

WhyDi

No crash your way and no error shown. the script stops and nothing. The bug splat window appears when I run the full script of our extension.

I modified the code but get the same result. what sub version of SU8 do you run?

thomthom

@whydi said:

No crash your way and no error shown. the script stops and nothing.

When you load main.rb - do you use require or Sketchup.require. The former will yield an Ruby error. The latter will just return false.

@whydi said:

The bug splat window appears when I run the full script of our extension.

Have you submitted this crash? (If so entered any info?)

@whydi said:

I modified the code but get the same result. what sub version of SU8 do you run?

8.0.16846

Can you share the RBS you generated from that snippet above?

main.zip
Here's my test files. The path is hard coded to my machine so you'd have to update that to test.

thomthom

My version is the latest SU8 release - meaning Trimble branded.

Sketchup.require is the same as Sketchup::require. Just different Ruby syntax.
Please submit the bugsplat with some info that can be used to look it up. Like mention it is related to Sketchup.require.

WhyDi

Your version is near mine. Does it come from Trimble or Google? Google I think.
I use Sketchup::require, not Sketchup.require
I haven't submitted the crash yet and don't think I do. It looks so strange!

Edit...
from your code i modified this way the main file


require 'sketchup.rb'
Sketchup;;require "C;/Users/uName/SketchUp/Ruby/testSC"
module Cie_qwerty
	module Ext_azerty
		puts "------------------------------------------start"
		print_test()
		puts "------------------------------------------ready"

	end
end

This time it works!
Sketchup::require or Sketchup.require doesn't matter
keepgoing tests...

Here are the files not modified

test.zip

thomthom

@whydi said:

Your version is near mine. Does it come from Trimble or Google? Google I think.
I use Sketchup::require, not Sketchup.require
I haven't submitted the crash yet and don't think I do. It looks so strange!

Here are the files

I did not see a crash when loading your files.
But I did get a load error, you where still having the Sketchup::require statement within your module:

Move it to the global namespace, like so:

Then it loads fine.

WhyDi

I edited my previous post with result after reading your code. It works!

Sketchup::require needs to appear before any module.

I go ahead testing line length...

thomthom

@whydi said:

Sketchup::require needs to appear before any module.

At least in SketchUp versions earlier than SU2014. From then on it was fixed.

WhyDi

I found information about line length here
https://groups.google.com/forum/#!topic/sketchupruby/EV5KDgh6C0M
where ChuckVali in the last post wrote :

@unknownuser said:

...I noticed something else about the scrambler. If there are more than around 1000 characters in any line the resulting .rbs file can't be read by SU. Not a problem though: I just use shorter lines and concat them...

the longest line in our script is over 1500 chars and I won't be able to cut these lines under 503 chars due to if statement with and and and and and so forth.
Are 512 chars OK?

in the same thread I read something that I don't clearly understand. Jim wrote :

@unknownuser said:

...This may be related to a bug when scrambling Ruby files. If you are
loading the .rbs from a SketchupExtension, the loaded files may be put
under the SketchupExtension namespace. Maybe there are conflicting
names if these other plugins are also loading scrambled files using
SketchupExtension...

Is it what we have just done ? I believe but who knows

driven

@whydi said:

...I won't be able to cut these lines under 503 chars due to if statement with and and and and and so forth....

the ruby 'style guide' recommends 70 chars per line, so 503 is a'little' longer than that...

you can break **and, &&, or, ||** in statements by escaping a newline...

I leave a space so before the escape \ as I find it easier to read...

if (Sketchup.active_model.tools.active_tool_name == 'ScaleTool' || \
   Sketchup.active_model.tools.active_tool_name == 'CameraOrbitTool' || \
   Sketchup.active_model.tools.active_tool_name == 'MoveTool' || \
   Sketchup.active_model.tools.active_tool_name == 'RotateTool' )
   p "waiting for #{Sketchup.active_model.tools.active_tool_name}"
else
   # whatever
end

john

WhyDi

Hi driven
70 chars per line are very small but I'm making my best for reducing more an more. Fortunately we are speaking about Sketchup Scrambler not ruby. Using your way I think a might be able to decrease the numbers of characters per line down to 250 even 125 but no more.
Result in some hours...
Have a good day

WhyDi

Hi all
I also learnt here http://sketchucation.com/forums/viewtopic.php?t=39094 that FILE and LINE do not work with 2014 previous versions but none of these keywords in the script.

Sketchup 8.0.16846 installed on another PC and test on both.
Max line length is 1276 and reduced to 1000 gives BugSplat at start time. 750 then 500 the same and finally 250 with no change. BugSplat as ever. I give up decreasing! It doesn't make any sense.

So I've reached the dead end!

After hours and hours of labour to preserve backward compatibility down to SU8 and just before releasing our extension the baby blows out! we are now unable to reach our purpose. I will send the bug plast but the answer will come to late if any.

From one hand the scrambler does not allow for 8 and 2013 version.
On the other hand using C doesn't look enough secure. Although I need to know more on C extension I think it's very probably all about dlls which means call from the main script and some bad eval() has to appear at some point in the ruby script. This also means that anyone can 'puts' the return of the dll call into the sketchup console easily. No more sense than above!

It's clear that without some ongoing miracle : Really the dead end.

I almost forgot to thank you for your time
Best Regards

thomthom

Have you identified the file that crashes?

I would load them manually, or put a tracing puts statement in them. To find which one crashes. Then strip away half the file until you find the section that crashes.

Just to be sure that this is a long-line crash - have you tried removing lines that are longer than a couple of hounded characters?

I'm still puzzled by your line length - what isn't that cannot be broken down? I don't think I ever have seen a line of code that had to be 1000+ chars long.

Can you post one of these lines - just as an example so we can fully understand what is going on?

But the key point here is the need to identify what file/line crashes.

WhyDi

Unfortunately they all and each crash.
under development we use 8 files encompassing roughly 50 classes, 500 functions, 7500 lines and 300k ruby only code out of of comment.
Each of these scrambled file loaded separately - meaning the others loaded as .rb - ends up with the BugSplat
After gathering them I got the same outcome
No doubt there is something wrong somewhere but what and where ?
I don't have time to read what a 'puts' can yield! running the scrambled script pauses about one second then close SU8 and finally shows the bug splat window. Nothing else than that. I'm never at hand to do anything!
What may I do? nothing!
the file with shortest (longest)line is 120. no html string into .rbs they are in the main module.rb

here is one of these numerous lines we have. delivered as is!


if (@keep_prj_skp_file_last != Mdl;;prj_skp_file_last) or
  (@keep_olt_user_filter != str_flt_cmp) or
  (@keep_olt_user_filter_layer != Mdl;;olt_user_filter_layer) or
  (@keep_fps_make != Mdl;;fps_make) or
  (@keep_anim_wait != Mdl;;anim_wait) or
  (@keep_dialog_collapse_at_runtime != @dialog_collapse_at_runtime) or
  (@keep_do_mode != Mdl;;do_mode) or
  (@keep_anim_restore_shot != Mdl;;anim_restore_shot) or
  (@keep_anim_save_shot != Mdl;;anim_save_shot) or
  (@keep_anim_save_model != Mdl;;anim_save_model) or
  (@keep_anim_restore_cam != Mdl;;anim_restore_cam) or
  (@keep_anim_save_cam != Mdl;;anim_save_cam) or
  (@keep_anim_restore_lay != Mdl;;anim_restore_lay)
    return true
end

What's wrong in such code?
Remember that all works perfectly without scrambling, even the gathered file with LineFeed or Semicolon at each end of line. All that works.
Is it worth to decrease line length under 256 chars? I have my doubts in such context

WhyDi

You know what? today we have made a decision founded on

Either our way of writing ruby script doesn't please SketchUp Scrambler
Or the scrambler is not suitable for us.

Binary choice as any developer knows in its life and I'm inclined to think that the first one is likely to prove. My team doesn't like that but it's now the only way to go out this dead end. This may turn out as most of us have a strong MS background with c# or VB.net as I do. For who knows a bit about VB I like that 'redim preserve' and always works with OptionExplict and OptionStrict and OptionInfer. The compiler doesn't tolerate anything but nearly no error at runtime and intelliSense makes the remaining. This could lead to bad habits with ruby script - I never feel easy with its syntax and its way to auto-type variables as instance.

But let's get back to the point. So some times ago a script developer asked us to integrate our group. I've just called him. He comes tomorrow with a clear purpose. His mission is to rewrite one of our file its own way. If his way succeeds - meaning the file can go through the scrambler and works - he integrates.
Outcomes tomorrow

thomthom

Is the crash also happening on SU2013 as well?
Is your extension already available in some form? (I was wondering if I could run it on my SU8 version which is the latest. You mentioned you where using an older version - maybe that has a particular bug in it...)
(Or if you could provide even just one of these RBS files for testing - you can PM it to me if you want.)

The way the Ruby is written shouldn't really matter - it's only that old bug with long lines that I know of. (Though we're talking about SU8 here - might be some old bug that I never ran into. I no longer write for these old versions.)

Oh, and just to be sure it's not an issue with an old version of the scrambler - have you tried using this page to scramble? https://extensions.sketchup.com/en/developer_center/extension_signature