WebDialog encoding bug found!

tt_su · 1 Apr 2014, 11:41

except, get_element_value fails pre v2013 and get_element_value works.

? It does? Are you sure?

tt_su · 1 Apr 2014, 11:43

@dan rathbun said:

To me it looks like it IS UTF-8, but Ruby thinks it is some other encoding, and doubly transcodes* it into UTF-8 AGAIN.

That's a similar problem we have with FILE $LOAD_PATH, $LOADED_FEATURES and ENV. UTF-8 byte sequences isn't labeled with the correct encoding.

slbaumgartner · 1 Apr 2014, 12:45

@dan rathbun said:

@slbaumgartner said:

But in 2014, the original 5-byte UTF-8 string is somehow transmuted into the 8-byte string \xE2\x88\x9A\xC2\xA9lan. ... It's as if the callback processing code has an incorrect implementation of the transcoding.

YES, agree.

To me it looks like it IS UTF-8, but Ruby thinks it is some other encoding, and doubly transcodes* it into UTF-8 AGAIN.

P.S. - isn't transmute a math term ?

I was thinking more of alchemy

driven · 1 Apr 2014, 13:13

@tt_su said:

? It does? Are you sure?

It certainly looks like it...

I cleaned up my initial code an per the Nanny and the Professors prodding...

and these are the new images...
and the new script

# encoding; UTF-8
def show_problem

  @lang_hash = {'lid1'=>"élan"}  # I'm using a hash because it's what I use in my plugin...

  @dlg2 = UI;;WebDialog.new("Problem_Main", false,"main_prob", 700, 500, 600, 0, false)


  html = %Q(
  <!DOCTYPE html>
  <html>
  <head>
  <title>Problem_Main</title>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body>
    <p>Tested using Sketchup.version #{Sketchup.version}</p>

    <p>Tested on #{RUBY_PLATFORM =~ /(darwin)/ ? ((%x(sw_vers).sub(/ProductName;/,'').sub(/ProductVersion;/,'').sub(/BuildVersion;/,'_'))) ; 'windows'}</p>

    <p>click into the box and hit return or change to use another non ASCII-8BIT string...</p>

    <input id="#{@lang_hash.keys[0]}" value="#{@lang_hash.values[0]}" type="text" onkeydown="if (event.keyCode == 13) trans_L8();" title="type text and then 'Enter'">

    <p>below is the return using 'get_element_value'</p>

    <h4 id="translation1"><!-- return appears here --></h4>

    <p>below is the return using 'add_action_callback' </p>

    <h4 id="translation2"><!-- return appears here --></h4>

    <p>below is the return of the query  'add_action_callback'  == 'get_element_value'</p>

    <h4 id="translation3"><!-- return appears here --></h4>

    <script type="text/javascript" charset="UTF-8">

      function trans_L8() {

        window.location = 'skp;trans_L8@'+#{@lang_hash.keys[0]}.id+','+(#{@lang_hash.keys[0]}.value);
      }

    </script>
  </body>
</html>
)

  @dlg2.set_html html

  RUBY_PLATFORM =~ /(darwin)/ ? @dlg2.show_modal() ; @dlg2.show()

  @dlg2.add_action_callback("trans_L8") {|dialog, params|

    param_id = params.split(',')[0].to_s
    param_val = params.split(',')[1].to_s

    callback = 'return using add_action_callback ' # + param_id.to_s + ' = ' + param_val.to_s # commented out to avoid the error
    puts callback
    p(params)

    elm_val = (@dlg2.get_element_value(param_id)).to_s
    element_value = 'return using get_element_value => ' + elm_val
    puts element_value

    @dlg2.execute_script("translation1.textContent='#{(elm_val)}'")

    @dlg2.execute_script("translation2.textContent='#{param_val}'")

    @dlg2.execute_script("translation3.textContent='#{elm_val == param_val}'")
}
end
show_problem

# load("/Users/johns_iMac/Library/Application Support/SketchUp 2014/SketchUp/Plugins/jcb_ViewPortResize/dev/show_encoding_issue.rb")
# load("[add your path]/show_encoding_issue.rb")

and a question? Is there code to get the Windows operating system details?

john

driven · 1 Apr 2014, 15:22

a CLUE perhaps...

Using js codepoints for my input, 'get_element_value' has switched all the separators...

return using add_action_callback "lid1,/uD83D/uDC7D/u20AC/u00A3/u0061/u0009"
return using get_element_value => \uD83D\uDC7D\u20AC\u00A3\u0061\u0009

so I modified my code

  @dlg2.add_action_callback("trans_L8") {|dialog, params|

    param_id = params.split(',')[0].to_s
    param_val = params.split(',')[1].to_s
    param_val_sub = params.split(',')[1].gsub('/','\\').to_s
p param_val_sub
    callback = 'return using add_action_callback ' # + param_id.to_s + ' = ' + param_val.to_s # commented out to avoid the error
    puts callback
    p(params)

    elm_val = (@dlg2.get_element_value(param_id)).to_s
    element_value = 'return using get_element_value => ' + elm_val
    puts element_value

    @dlg2.execute_script("translation1.textContent='#{elm_val}'")

    @dlg2.execute_script("translation2.textContent='#{param_val}'")

    @dlg2.execute_script("translation3.textContent='#{param_val_sub}'")
}

and this is the result

john

driven · 1 Apr 2014, 18:30

So it seems there are in-fact two issues...

1: return having separators reversed

2: encoding being mis-read

I added the .gsub('/','\\') to both types of return, so it works on v8 or v13/v14

I added js to convert the string to UNICODE on keydown, before it retrieved by SU...

and the full workaround...

# encoding; UTF-8
def show_problem

  @lang_hash = {'lid1'=> %Q(élan  ümlet)}  # I'm using a hash because it's what I use in my plugin...

  @dlg2 = UI;;WebDialog.new("Problem_Main", false,"main_prob", 700, 500, 600, 0, false)


  html = %Q(
  <!DOCTYPE html>
  <html>
  <head>
  <title>Problem_Main</title>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body>
    <p>Tested using Sketchup.version #{Sketchup.version}</p>

    <p>Tested on #{RUBY_PLATFORM =~ /(darwin)/ ? ((%x(sw_vers).sub(/ProductName;/,'').sub(/ProductVersion;/,'').sub(/BuildVersion;/,'_'))) ; 'windows'}</p>

    <p>click into the box and hit return or change to use another non ASCII-8BIT string...</p>

    <input id="#{@lang_hash.keys[0]}" value="#{@lang_hash.values[0]}" type="text" onkeydown=" if (event.keyCode == 13) this.value=unicodeLiteral(this.value);trans_L8();"  title="type text and then 'Enter'">

    <p>below is the return using 'get_element_value'</p>

    <h4 id="translation1"><!-- return appears here --></h4>

    <p>below is the return using 'add_action_callback' </p>

    <h4 id="translation2"><!-- return appears here --></h4>


    <h4 id="translation3"><!-- return appears here --></h4>

    <script type="text/javascript" charset="UTF-8">

      function trans_L8() {

        window.location = 'skp;trans_L8@'+#{@lang_hash.keys[0]}.id+','+(#{@lang_hash.keys[0]}.value);
      }

      /* Creates a uppercase hex number with at least length digits from a given number */
function fixedHex(number, length){
    var str = number.toString(16).toUpperCase();
    while(str.length < length)
        str = "0" + str;
    return str;
}

      /* Creates a unicode literal based on the string */
function unicodeLiteral(str){
    var i;
    var result = "";
    for( i = 0; i < str.length; ++i){
        /* You should probably replace this by an isASCII test */
        if(str.charCodeAt(i) > 126 || str.charCodeAt(i) < 32)
            result += "\\\\" + "u" + fixedHex(str.charCodeAt(i),4);
        else
            result += str[i];
    }

    return result;
}

    </script>
  </body>
</html>
)

  @dlg2.set_html html

  RUBY_PLATFORM =~ /(darwin)/ ? @dlg2.show_modal() ; @dlg2.show()

  @dlg2.add_action_callback("trans_L8") {|dialog, params|

    param_id = params.split(',')[0].to_s
    param_val = params.split(',')[1].gsub('/',"\\").to_s
    callback = 'return using add_action_callback ' # + param_id.to_s + ' = ' + param_val.to_s # commented out to avoid the error
    puts callback
    p(params)

    elm_val = (@dlg2.get_element_value(param_id)).gsub('/',"\\").to_s
    element_value = 'return using get_element_value => ' + elm_val
    puts element_value

    @dlg2.execute_script("translation1.textContent='#{elm_val}'")

    @dlg2.execute_script("translation2.textContent='#{param_val}'")

}
end
show_problem

# load("/Users/johns_iMac/Library/Application Support/SketchUp 2014/SketchUp/Plugins/jcb_ViewPortResize/dev/show_encoding_issue.rb")
# load("[add your path]/show_encoding_issue.rb")

BIG question is will this work on other PC's, or would I need a platform conditional?

In review,
I think the http header has the incorrect encoding and as SU is the server it must come from there...
The internal 'separator' variations is still a mystery...

john

Dan Rathbun · 1 Apr 2014, 20:22

@driven said:

... and a question? Is there code to get the Windows operating system details?

Get the default system Encoding:
Encoding::find("filesystem")
or
Encoding::find("locale")

(On my machine it returns the #<Encoding:Windows-1252> object reference.)

If you want the Windows version:
%x[ver]

On my machine it returns: "Microsoft Windows [Version 6.1.7601]"
XP is 5.1
Vista is 6.0
Win7 is 6.1

What else do you want to know ?

driven · 1 Apr 2014, 20:52

@dan rathbun said:

What else do you want to know ?

does my last script run on your PC?
if swap these in, can you show me a screen shot?
a more complex input...

  @lang_hash = {'lid1'=> %Q(élan 勢い Schwung импульс)}  # I'm using a hash because it's what I use in my plugin...

and the widows versioning...

    <p>Tested on #{RUBY_PLATFORM =~ /(darwin)/ ? ((%x(sw_vers).sub(/ProductName;/,'').sub(/ProductVersion;/,'').sub(/BuildVersion;/,'_'))) ; %x[ver]}</p>

also, did you try it without the fix? did you get both returns on v2014?

john

Dan Rathbun · 2 Apr 2014, 00:01

I had not actually run the code at all, before.
Here goes:

@driven said:

..., did you try it without the fix? did you get both returns on v2014?

On SU2014, without the fix:

load "test/WebDialog_param_encoding_bug.rb" %(#004000)[true return using add_action_callback lid1 = élan return using get_element_value => élan]

After hitting return the elements are NOT replaced in the WebDialog because %(#8000BF)[element.textContent=] does not work on MSIE.
You need a platform code branch and use %(#8000BF)[element.innerText=] on PC.

On SU2014, WITH the fix, AFTER hitting return:

load "test/WebDialog_param_encoding_fix.rb" %(#004000)[true return using add_action_callback "lid1,\\u00E9lan \\uF8FF \\u00FCmlet" return using get_element_value => \u00E9lan \uF8FF \u00FCmlet]

First setting:
Encoding::default_internal="UTF-8" %(#404000)[UTF-8]
.. has not effect (no difference.)

Dan Rathbun · 2 Apr 2014, 00:15

With the two changes on SU2014.

Before hitting ENTER:

.. but when I click in the text control, and hit the END key, the ruby console shows:
%(#004000)[return using add_action_callback "lid1,\xC3\xA9lan \xE5\x8B\xA2\xE3\x81\x84 Schwung \xD0\xB8\xD0\xBC\xD0\xBF\xD1\x83\xD0\xBB\xD1\x8C\xD1\x81" return using get_element_value => élan 勢い Schwung импульс]

After hitting ENTER:

.. and then the Ruby console shows:
%(#004000)[return using add_action_callback "lid1,\\u00E9lan \\u52E2\\u3044 Schwung \\u0438\\u043C\\u043F\\u0443\\u043B\\u044C\\u0441" return using get_element_value => \u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441]

driven · 2 Apr 2014, 00:36

thanks dan

@unknownuser said:

You need a platform code branch and use element.innerText= on PC.

innerText works on mac, so I'll just change that...

can you run with the tweak...
and see if they show in the dialog...

the extra // in your console return confuses me. I had to add more to escape ruby escaping javascript...

john

Dan Rathbun · 2 Apr 2014, 00:47

OK, that latest tweek, after hitting enter:

And the console, after hitting enter:
%(#004000)[load "test/show_encoding_issue.rb" true return using add_action_callback "lid1,\\u00E9lan \\u52E2\\u3044 Schwung \\u0438\\u043C\\u043F\\u0443\\u043B\\u044C\\u0441" return using get_element_value => \u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441]

BTW.. if the WebDialog has focus after hitting ENTER, and I need to use the ALT key (for ALT+PrintScreen to copy the active window image to the clipboard, so I can save it from Paint.NET ,) there is a whole bunch of repeated console output, like 15 times while I held the ALT key down.

driven · 2 Apr 2014, 01:15

yeh it works...

@dan rathbun said:

BTW.. if the WebDialog has focus after hitting ENTER, and I need to use the ALT key (for ALT+PrintScreen to copy the active window image to the clipboard, so I can save it from Paint.NET ,) there is a whole bunch of repeated console output, like 15 times while I held the ALT key down.

with this should I just add a screen grab and dump it to desktop?

    @dlg2.write_image(File.join((File.expand_path("~")), "Desktop", "dlg_test.png"))

what's the PC equiv...
EDIT: for completeness I'll add the unmodified pair back into the mix after sleep...
thanks again dan
john

Dan Rathbun · 2 Apr 2014, 03:05

@driven said:

... should I just add a screen grab and dump it to desktop?

NO.

Replace the %(#8000BF)[onkeydown=] with:
%(#8000BF)[onkeyup=" if (event.keyCode == 13) { this.value=unicodeLiteral(this.value); trans_L8(); }"]

Notice that I enclose BOTH the following statements within a "curly block", so that the test only fires when ENTER key is released (after being pressed.)
The condition will not even be tested while any key is down, including any modifier key.

Dan Rathbun · 2 Apr 2014, 03:07

FYI: I am one of those who hates programs putting stuff on MY Desktop.

driven · 2 Apr 2014, 14:35

@dan rathbun said:

FYI: I am one of those who hates programs putting stuff on MY Desktop.

so do I so I added it as a button...
NEW VERSION...
I split the input method to make it function the same in all versions of SU...
I can't get the raw code too display in the dialog, so here it is for v2014, the others are similarly wrong...
EDIT: made a small adjustment to get 'fixed' to display correctly [puts v p]

true
return using add_action_callback [id,value] 
"\xE2\x88\x9A\xC2\xA9lan \xC3\x82\xC3\xA3\xC2\xA2\xE2\x80\x9E\xC3\x85\xC3\x91 Schwung \xE2\x80\x93\xE2\x88\x8F\xE2\x80\x93\xC2\xBA\xE2\x80\x93\xC3\xB8\xE2\x80\x94\xC3\x89\xE2\x80\x93\xC2\xAA\xE2\x80\x94\xC3\xA5\xE2\x80\x94\xC3\x85"
return using get_element_value => élan 勢い Schwung импульс
return using add_action_callback [id,value_fixed] 
\u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441
return using get_element_value fixed => \u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441

Please test the new script...with puts fix

john

TIG · 2 Apr 2014, 17:59

Here are my PC results.
Win7 IE11.
Note how v2014 misses platform ?
Only the v8 Russian text seems to have a problem on get_element_value ??

Garry K · 2 Apr 2014, 18:02

Here you go.
I've got SU 7 Free, SU 8 Free, SU 2013 Trial, SU 2014 Trial

Garry K · 2 Apr 2014, 18:05

Sorry - missed clicking the button for SU 7

driven · 2 Apr 2014, 19:31

Thanks Tig and Garry,

from your images it appears that v8 had an issue with the standard call but shows correctly if SU is handed a Unicode charset string with it's separators gsub'd.

The ongoing v2014 (PC) handles both standard or 'fixed', where my mac's fail.

I'll write up a bug report for this, and at least I can now carry on with my language translator...

The only conditions, in this script, are for the 'extra' system info, which I won't need...

@garry, if you click into the text field, then hit return and then click the button, you'll see the default results as well...

I had to split them onto Two buttons for v8 to use the same code as the others...

john