WebDialog encoding bug found!

driven

Nah, I should know enough to check your code :oops: . With a real test, equality fails in all cases (as it should!).

it's the tangents that count...

now you can write a proper test...

john

Dan Rathbun

The text that displays in your test, says UFT-8 in many places. It is UTF-8.
(The word format is at the end, of Unicode Transformation Format.)

(nag, nag)

Dan Rathbun

@slbaumgartner said:

But in 2014, the original 5-byte UTF-8 string is somehow transmuted into the 8-byte string \xE2\x88\x9A\xC2\xA9lan. ... It's as if the callback processing code has an incorrect implementation of the transcoding.

YES, agree.

To me it looks like it IS UTF-8, but Ruby thinks it is some other encoding, and doubly transcodes* it into UTF-8 AGAIN.

P.S. - isn't transmute a math term ?

tt_su

@driven said:

except, get_element_value fails pre v2013 and get_element_value works.

? It does? Are you sure?

tt_su

@dan rathbun said:

To me it looks like it IS UTF-8, but Ruby thinks it is some other encoding, and doubly transcodes* it into UTF-8 AGAIN.

That's a similar problem we have with FILE $LOAD_PATH, $LOADED_FEATURES and ENV. UTF-8 byte sequences isn't labeled with the correct encoding.

slbaumgartner

@dan rathbun said:

@slbaumgartner said:

But in 2014, the original 5-byte UTF-8 string is somehow transmuted into the 8-byte string \xE2\x88\x9A\xC2\xA9lan. ... It's as if the callback processing code has an incorrect implementation of the transcoding.

YES, agree.

To me it looks like it IS UTF-8, but Ruby thinks it is some other encoding, and doubly transcodes* it into UTF-8 AGAIN.

P.S. - isn't transmute a math term ?

I was thinking more of alchemy

driven

@tt_su said:

? It does? Are you sure?

It certainly looks like it...

I cleaned up my initial code an per the Nanny and the Professors prodding...

and these are the new images... v13 v14
and the new script

# encoding; UTF-8
def show_problem

  @lang_hash = {'lid1'=>"élan"}  # I'm using a hash because it's what I use in my plugin...

  @dlg2 = UI;;WebDialog.new("Problem_Main", false,"main_prob", 700, 500, 600, 0, false)


  html = %Q(
  <!DOCTYPE html>
  <html>
  <head>
  <title>Problem_Main</title>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body>
    <p>Tested using Sketchup.version #{Sketchup.version}</p>

    <p>Tested on #{RUBY_PLATFORM =~ /(darwin)/ ? ((%x(sw_vers).sub(/ProductName;/,'').sub(/ProductVersion;/,'').sub(/BuildVersion;/,'_'))) ; 'windows'}</p>

    <p>click into the box and hit return or change to use another non ASCII-8BIT string...</p>

    <input id="#{@lang_hash.keys[0]}" value="#{@lang_hash.values[0]}" type="text" onkeydown="if (event.keyCode == 13) trans_L8();" title="type text and then 'Enter'">

    <p>below is the return using 'get_element_value'</p>

    <h4 id="translation1"><!-- return appears here --></h4>

    <p>below is the return using 'add_action_callback' </p>

    <h4 id="translation2"><!-- return appears here --></h4>

    <p>below is the return of the query  'add_action_callback'  == 'get_element_value'</p>

    <h4 id="translation3"><!-- return appears here --></h4>

    <script type="text/javascript" charset="UTF-8">

      function trans_L8() {

        window.location = 'skp;trans_L8@'+#{@lang_hash.keys[0]}.id+','+(#{@lang_hash.keys[0]}.value);
      }

    </script>
  </body>
</html>
)

  @dlg2.set_html html

  RUBY_PLATFORM =~ /(darwin)/ ? @dlg2.show_modal() ; @dlg2.show()

  @dlg2.add_action_callback("trans_L8") {|dialog, params|

    param_id = params.split(',')[0].to_s
    param_val = params.split(',')[1].to_s

    callback = 'return using add_action_callback ' # + param_id.to_s + ' = ' + param_val.to_s # commented out to avoid the error
    puts callback
    p(params)

    elm_val = (@dlg2.get_element_value(param_id)).to_s
    element_value = 'return using get_element_value => ' + elm_val
    puts element_value

    @dlg2.execute_script("translation1.textContent='#{(elm_val)}'")

    @dlg2.execute_script("translation2.textContent='#{param_val}'")

    @dlg2.execute_script("translation3.textContent='#{elm_val == param_val}'")
}
end
show_problem

# load("/Users/johns_iMac/Library/Application Support/SketchUp 2014/SketchUp/Plugins/jcb_ViewPortResize/dev/show_encoding_issue.rb")
# load("[add your path]/show_encoding_issue.rb")

and a question? Is there code to get the Windows operating system details?

john

driven

a CLUE perhaps...

Using js codepoints for my input, 'get_element_value' has switched all the separators...

return using add_action_callback "lid1,/uD83D/uDC7D/u20AC/u00A3/u0061/u0009"
return using get_element_value => \uD83D\uDC7D\u20AC\u00A3\u0061\u0009

so I modified my code

  @dlg2.add_action_callback("trans_L8") {|dialog, params|

    param_id = params.split(',')[0].to_s
    param_val = params.split(',')[1].to_s
    param_val_sub = params.split(',')[1].gsub('/','\\').to_s
p param_val_sub
    callback = 'return using add_action_callback ' # + param_id.to_s + ' = ' + param_val.to_s # commented out to avoid the error
    puts callback
    p(params)

    elm_val = (@dlg2.get_element_value(param_id)).to_s
    element_value = 'return using get_element_value => ' + elm_val
    puts element_value

    @dlg2.execute_script("translation1.textContent='#{elm_val}'")

    @dlg2.execute_script("translation2.textContent='#{param_val}'")

    @dlg2.execute_script("translation3.textContent='#{param_val_sub}'")
}

and this is the result
using js codepoints
john

driven

So it seems there are in-fact two issues...

1: return having separators reversed

2: encoding being mis-read

I added the .gsub('/','\\') to both types of return, so it works on v8 or v13/v14

I added js to convert the string to UNICODE on keydown, before it retrieved by SU...

and the full workaround... v8 fixed get_element_value and encoding v14 fixed action_callback and encoding

# encoding; UTF-8
def show_problem

  @lang_hash = {'lid1'=> %Q(élan  ümlet)}  # I'm using a hash because it's what I use in my plugin...

  @dlg2 = UI;;WebDialog.new("Problem_Main", false,"main_prob", 700, 500, 600, 0, false)


  html = %Q(
  <!DOCTYPE html>
  <html>
  <head>
  <title>Problem_Main</title>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body>
    <p>Tested using Sketchup.version #{Sketchup.version}</p>

    <p>Tested on #{RUBY_PLATFORM =~ /(darwin)/ ? ((%x(sw_vers).sub(/ProductName;/,'').sub(/ProductVersion;/,'').sub(/BuildVersion;/,'_'))) ; 'windows'}</p>

    <p>click into the box and hit return or change to use another non ASCII-8BIT string...</p>

    <input id="#{@lang_hash.keys[0]}" value="#{@lang_hash.values[0]}" type="text" onkeydown=" if (event.keyCode == 13) this.value=unicodeLiteral(this.value);trans_L8();"  title="type text and then 'Enter'">

    <p>below is the return using 'get_element_value'</p>

    <h4 id="translation1"><!-- return appears here --></h4>

    <p>below is the return using 'add_action_callback' </p>

    <h4 id="translation2"><!-- return appears here --></h4>


    <h4 id="translation3"><!-- return appears here --></h4>

    <script type="text/javascript" charset="UTF-8">

      function trans_L8() {

        window.location = 'skp;trans_L8@'+#{@lang_hash.keys[0]}.id+','+(#{@lang_hash.keys[0]}.value);
      }

      /* Creates a uppercase hex number with at least length digits from a given number */
function fixedHex(number, length){
    var str = number.toString(16).toUpperCase();
    while(str.length < length)
        str = "0" + str;
    return str;
}

      /* Creates a unicode literal based on the string */
function unicodeLiteral(str){
    var i;
    var result = "";
    for( i = 0; i < str.length; ++i){
        /* You should probably replace this by an isASCII test */
        if(str.charCodeAt(i) > 126 || str.charCodeAt(i) < 32)
            result += "\\\\" + "u" + fixedHex(str.charCodeAt(i),4);
        else
            result += str[i];
    }

    return result;
}

    </script>
  </body>
</html>
)

  @dlg2.set_html html

  RUBY_PLATFORM =~ /(darwin)/ ? @dlg2.show_modal() ; @dlg2.show()

  @dlg2.add_action_callback("trans_L8") {|dialog, params|

    param_id = params.split(',')[0].to_s
    param_val = params.split(',')[1].gsub('/',"\\").to_s
    callback = 'return using add_action_callback ' # + param_id.to_s + ' = ' + param_val.to_s # commented out to avoid the error
    puts callback
    p(params)

    elm_val = (@dlg2.get_element_value(param_id)).gsub('/',"\\").to_s
    element_value = 'return using get_element_value => ' + elm_val
    puts element_value

    @dlg2.execute_script("translation1.textContent='#{elm_val}'")

    @dlg2.execute_script("translation2.textContent='#{param_val}'")

}
end
show_problem

# load("/Users/johns_iMac/Library/Application Support/SketchUp 2014/SketchUp/Plugins/jcb_ViewPortResize/dev/show_encoding_issue.rb")
# load("[add your path]/show_encoding_issue.rb")

BIG question is will this work on other PC's, or would I need a platform conditional?

In review,
I think the http header has the incorrect encoding and as SU is the server it must come from there...
The internal 'separator' variations is still a mystery...

john

Dan Rathbun

@driven said:

... and a question? Is there code to get the Windows operating system details?

Get the default system Encoding:
Encoding::find("filesystem")
or
Encoding::find("locale")

(On my machine it returns the #<Encoding:Windows-1252> object reference.)

If you want the Windows version:
%x[ver]

On my machine it returns: "Microsoft Windows [Version 6.1.7601]"
XP is 5.1
Vista is 6.0
Win7 is 6.1

What else do you want to know ?

driven

@dan rathbun said:

What else do you want to know ?

does my last script run on your PC?
if swap these in, can you show me a screen shot?
a more complex input...

  @lang_hash = {'lid1'=> %Q(élan 勢い Schwung импульс)}  # I'm using a hash because it's what I use in my plugin...

and the widows versioning...

    <p>Tested on #{RUBY_PLATFORM =~ /(darwin)/ ? ((%x(sw_vers).sub(/ProductName;/,'').sub(/ProductVersion;/,'').sub(/BuildVersion;/,'_'))) ; %x[ver]}</p>

also, did you try it without the fix? did you get both returns on v2014?

john

Dan Rathbun

I had not actually run the code at all, before.
Here goes:

@driven said:

..., did you try it without the fix? did you get both returns on v2014?

On SU2014, without the fix:

load "test/WebDialog_param_encoding_bug.rb" %(#004000)[true return using add_action_callback lid1 = élan return using get_element_value => élan]

After hitting return the elements are NOT replaced in the WebDialog because %(#8000BF)[element.textContent=] does not work on MSIE.
You need a platform code branch and use %(#8000BF)[element.innerText=] on PC.

On SU2014, WITH the fix, AFTER hitting return:

load "test/WebDialog_param_encoding_fix.rb" %(#004000)[true return using add_action_callback "lid1,\\u00E9lan \\uF8FF \\u00FCmlet" return using get_element_value => \u00E9lan \uF8FF \u00FCmlet]

First setting:
Encoding::default_internal="UTF-8" %(#404000)[UTF-8]
.. has not effect (no difference.)

Dan Rathbun

With the two changes on SU2014.

Before hitting ENTER:

.. but when I click in the text control, and hit the END key, the ruby console shows:
%(#004000)[return using add_action_callback "lid1,\xC3\xA9lan \xE5\x8B\xA2\xE3\x81\x84 Schwung \xD0\xB8\xD0\xBC\xD0\xBF\xD1\x83\xD0\xBB\xD1\x8C\xD1\x81" return using get_element_value => élan 勢い Schwung импульс]

After hitting ENTER:

.. and then the Ruby console shows:
%(#004000)[return using add_action_callback "lid1,\\u00E9lan \\u52E2\\u3044 Schwung \\u0438\\u043C\\u043F\\u0443\\u043B\\u044C\\u0441" return using get_element_value => \u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441]

driven

thanks dan

@unknownuser said:

You need a platform code branch and use element.innerText= on PC.

innerText works on mac, so I'll just change that...

can you run with the tweak...
and see if they show in the dialog...

the extra // in your console return confuses me. I had to add more to escape ruby escaping javascript...

john

Dan Rathbun

OK, that latest tweek, after hitting enter:

And the console, after hitting enter:
%(#004000)[load "test/show_encoding_issue.rb" true return using add_action_callback "lid1,\\u00E9lan \\u52E2\\u3044 Schwung \\u0438\\u043C\\u043F\\u0443\\u043B\\u044C\\u0441" return using get_element_value => \u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441]

BTW.. if the WebDialog has focus after hitting ENTER, and I need to use the ALT key (for ALT+PrintScreen to copy the active window image to the clipboard, so I can save it from Paint.NET,) there is a whole bunch of repeated console output, like 15 times while I held the ALT key down.

driven

yeh it works...

@dan rathbun said:

BTW.. if the WebDialog has focus after hitting ENTER, and I need to use the ALT key (for ALT+PrintScreen to copy the active window image to the clipboard, so I can save it from Paint.NET,) there is a whole bunch of repeated console output, like 15 times while I held the ALT key down.

with this should I just add a screen grab and dump it to desktop?

    @dlg2.write_image(File.join((File.expand_path("~")), "Desktop", "dlg_test.png"))

auto acquired image...
what's the PC equiv...
EDIT: for completeness I'll add the unmodified pair back into the mix after sleep...
thanks again dan
john

Dan Rathbun

@driven said:

... should I just add a screen grab and dump it to desktop?

NO.

Replace the %(#8000BF)[onkeydown=] with:
%(#8000BF)[onkeyup=" if (event.keyCode == 13) { this.value=unicodeLiteral(this.value); trans_L8(); }"]

Notice that I enclose BOTH the following statements within a "curly block", so that the test only fires when ENTER key is released (after being pressed.)
The condition will not even be tested while any key is down, including any modifier key.

Dan Rathbun

FYI: I am one of those who hates programs putting stuff on MY Desktop.

driven

@dan rathbun said:

FYI: I am one of those who hates programs putting stuff on MY Desktop.

so do I so I added it as a button...
NEW VERSION...
I split the input method to make it function the same in all versions of SU...
I can't get the raw code too display in the dialog, so here it is for v2014, the others are similarly wrong...
EDIT: made a small adjustment to get 'fixed' to display correctly [puts v p]

true
return using add_action_callback [id,value] 
"\xE2\x88\x9A\xC2\xA9lan \xC3\x82\xC3\xA3\xC2\xA2\xE2\x80\x9E\xC3\x85\xC3\x91 Schwung \xE2\x80\x93\xE2\x88\x8F\xE2\x80\x93\xC2\xBA\xE2\x80\x93\xC3\xB8\xE2\x80\x94\xC3\x89\xE2\x80\x93\xC2\xAA\xE2\x80\x94\xC3\xA5\xE2\x80\x94\xC3\x85"
return using get_element_value => élan 勢い Schwung импульс
return using add_action_callback [id,value_fixed] 
\u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441
return using get_element_value fixed => \u00E9lan \u52E2\u3044 Schwung \u0438\u043C\u043F\u0443\u043B\u044C\u0441

encoding_issue_14.0.4899.png encoding_issue_13.0.4811.png encoding_issue_8.0.16845.png

Please test the new script...with puts fix

john

TIG

Here are my PC results.
Win7 IE11.
Note how v2014 misses platform ?
Only the v8 Russian text seems to have a problem on get_element_value ??