[MacRuby] #849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases.
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- In the range that I understood, I collected the test-cases from Ruby 1.9.2 RC2 that MacRuby throws the exception of "Encoding::CompatibilityError".[[BR]] I atach the test-cases. {{{ $ macruby test_fileutils.rb Loaded suite test_fileutils Started EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE Finished in 2.835229 seconds. 1) Error: test_cd(TestFileUtils): Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT /Users/watson/tmp/test_encode/test_fileutils.rb:116:in `prepare_data_file' --- snip --- }}} {{{ $ macruby test_encode_error.rb Loaded suite test_encode_error Started EEEEEEEEEEEEEEEEEE Finished in 1.792149 seconds. 1) Error: test_cgi_unescape(CGIUtilTest): Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT /Users/watson/tmp/test_encode/test_encode_error.rb:20:in `test_cgi_unescape' --- snip --- }}} -- Ticket URL: <http://www.macruby.org/trac/ticket/849> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): When running test_encode_error.rb with trunk, the first exception that comes up seems to be from gsub. {{{ Breakpoint 1, rb_exc_raise (mesg=17184759744) at eval.c:326 326 rb_vm_raise(mesg); (gdb) po mesg incompatible character encodings: UTF-8 and ASCII-8BIT Current language: auto; currently c (gdb) bt #0 rb_exc_raise (mesg=17184759744) at eval.c:326 #1 0x000000010002d984 in rb_raise (exc=<value temporarily unavailable, due to optimizations>, fmt=<value temporarily unavailable, due to optimizations>) at error.c:1119 #2 0x00000001000bc7b4 in str_concat_string (self=0x40077c7a0, str=0x0) at string.c:192 #3 0x00000001000c31a1 in str_gsub (sel=<value temporarily unavailable, due to optimizations>, argc=<value temporarily unavailable, due to optimizations>, argv=<value temporarily unavailable, due to optimizations>, str=17188767616, bang=false) at string.c:3909 #4 0x00000001001302f9 in rb_vm_dispatch (_vm=0x100a16980, cache=0x100bc8640, top=17183089696, self=17188767616, klass=0x40030b1e0, sel=0x100aaf4f0, block=0x40040af00, opt=0 '\0', argc=1, argv=0x7fff5fbf7fb8) at dispatcher.cpp:435 }}} -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:1> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): It seems to come from cgi/util.rb: {{{ (gdb) p rb_symbolicate (0x0000000102c43acf) addr 0x102c43acf start 0x102c43960 selector unescape: location /Library/Frameworks/MacRuby.framework/Versions/0.7/usr/lib/ruby/1.9.2/cgi/util.rb:17 }}} {{{ # URL-decode a string with encoding(optional). # string = CGI::unescape("%27Stop%21%27+said+Fred") # # => "'Stop!' said Fred" def CGI::unescape(string,encoding=@@accept_charset) str=string.tr('+', ' ').gsub(/((?:%[0-9a-fA-F]{2})+)/) do [$1.delete('%')].pack('H*') end.force_encoding(encoding) str.valid_encoding? ? str : str.force_encoding(string.encoding) end }}} I suspect our pack function returns binary data by default which can't be concatenated to UTF-8. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:2> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): I have written "# -*- coding: undecided -*-" in test_encode_error.rb by mistake.[[BR]] Please delete it to occur the error when try test_encode_error.rb in Ruby 1.9.[[BR]] Sorry. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:3> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): It seems to be a issue of pack so that I collected the test cast of pack/unpack to be related. {{{ $ macruby test_pack.rb Loaded suite test_pack Started FFFFFFFFFFFFF Finished in 1.045697 seconds. 1) Failure: test_pack(TestArray) [/Library/Frameworks/MacRuby.framework/Versions/0.7/usr/lib/ruby/1.9.2/minitest/unit.rb:670]: <"\x86"> (UTF-8) expected but was <"\x86"> (ASCII-8BIT). 2) Failure: test_decode64(TestBase64) [/Users/watson/src/MacRuby/test_pack.rb:6]: <"\xFF"> (UTF-8) expected but was <"\xFF"> (ASCII-8BIT). 3) Failure: test_strict_decode64(TestBase64) [/Users/watson/src/MacRuby/test_pack.rb:17]: <"\xFF"> (UTF-8) expected but was <"\xFF"> (ASCII-8BIT). 4) Failure: test_urlsafe_decode64(TestBase64) [/Users/watson/src/MacRuby/test_pack.rb:40]: <"\xFF"> (UTF-8) expected but was <"\xFF"> (ASCII-8BIT). 5) Failure: test_pack_N(TestPack) [/Users/watson/src/MacRuby/test_pack.rb:117]: <"\xFF\xFF\xFF\xFE"> (UTF-8) expected but was <"\xFF\xFF\xFF\xFE"> (ASCII-8BIT). --- snip --- }}} -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:4> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): I think pack('U') to be repaired by the following patch. {{{ #!diff diff --git a/encoding.c b/encoding.c index d3295a0..dc590a8 100644 --- a/encoding.c +++ b/encoding.c @@ -519,9 +519,7 @@ rb_enc_get_index(VALUE obj) void rb_enc_set_index(VALUE obj, int encindex) { - if (encindex < ENCODINGS_COUNT) { - return ; - } + assert(encindex >= 0 && encindex < ENCODINGS_COUNT); rb_str_force_encoding(obj, rb_encodings[encindex]); } diff --git a/include/ruby/intern.h b/include/ruby/intern.h index d8b3940..58fc56a 100644 --- a/include/ruby/intern.h +++ b/include/ruby/intern.h @@ -174,6 +174,9 @@ VALUE rb_fiber_resume(VALUE fib, int argc, VALUE *args); VALUE rb_fiber_yield(int argc, VALUE *args); VALUE rb_fiber_current(void); VALUE rb_fiber_alive_p(VALUE); +/* encoding.c */ +void rb_enc_set_index(VALUE obj, int encindex); +int rb_utf8_encindex(void); /* enum.c */ /* error.c */ VALUE rb_exc_new(VALUE, const char*, long); diff --git a/pack.c b/pack.c index 9d073cb..89c9142 100644 --- a/pack.c +++ b/pack.c @@ -446,6 +446,7 @@ pack_pack(VALUE ary, SEL sel, VALUE fmt) char type; long items, len, idx, plen; const char *ptr; + int enc_info = 1; /* 0 - BINARY, 1 - US-ASCII, 2 - UTF-8 */ #ifdef NATINT_PACK int natint; /* native integer */ #endif @@ -508,6 +509,20 @@ pack_pack(VALUE ary, SEL sel, VALUE fmt) } switch (type) { + case 'U': + /* if encoding is US-ASCII, upgrade to UTF-8 */ + if (enc_info == 1) enc_info = 2; + break; + case 'm': case 'M': case 'u': + /* keep US-ASCII (do nothing) */ + break; + default: + /* fall back to BINARY */ + enc_info = 0; + break; + } + + switch (type) { case 'A': case 'a': case 'Z': case 'B': case 'b': case 'H': case 'h': @@ -1014,6 +1029,20 @@ pack_pack(VALUE ary, SEL sel, VALUE fmt) } } + OBJ_INFECT(data, fmt); + switch (enc_info) { + case 1: + // TODO + // ENCODING_CODERANGE_SET(data, rb_usascii_encindex(), ENC_CODERANGE_7BIT); + break; + case 2: + rb_enc_set_index(data, rb_utf8_encindex()); + break; + default: + /* do nothing, keep ASCII-8BIT */ + break; + } + // Taint the ByteString accordingly. if (OBJ_TAINTED(fmt)) { OBJ_TAINT(data); }}} -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:5> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): About other failing tests in test_pack.rb, I guess that depend on #742. Because, the result is not corresponding as follows when running it with Ruby 1.9 specifying Script Encoding of UTF-8. [[BR]] Default Encoding is ASCII-8BIT in test_pack.rb so that Script Encoding is not specified. {{{ # -*- coding: utf-8 -*- p "\377\377\377\376".encoding p [4294967294].pack('N').encoding }}} {{{ $ ruby test.rb #<Encoding:UTF-8> #<Encoding:ASCII-8BIT> }}} -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:6> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): I see. #742 is another story indeed. I have a few points regarding your patch: - Maybe the include/ruby/intern.h changes should go to include/ruby/encoding.h? - I believe the OBJ_INFECT macro does nothing on MacRuby, it's probably better to not use it. Can you try running the rubyspec suite (rake spec:ci) after building with your patch? If the specs pass, feel free to commit. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:7> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): Abort occurs even when "rake spec:ci" is ran with MacRuby Trunk HEAD, and it is not possible to confirm the pass X( -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:8> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): Can you try "rake spec:core" and "rake spec:library"? It might work (I suspect the crash is introduced by a spec:macruby spec). -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:9> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): I tried those commands, and worked using r4433. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:10> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): If this also includes your #pack fix, then feel free to commit :) -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:11> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): I see that you committed the change in r4437, should this bug be closed? -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:12> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): I guess that MacRuby does not throw the exception of this issue if #742 is fixed.[[BR]] To keep seeing the appearance, I think that this issue's status keep "open" until #742 fixed. It may decrease that the same problem is registered again, if status is "open". :) -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:13> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by jhamor@…): I think this might be related to #988 but not sure. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:14> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by watson1978@…): MacRuby does not support Script Encoding now.[[BR]] And can't check validity of Encoding definitely. I think better that MacRuby does not check Encoding, till issues are fixed. It is very likely that the script which stopped by this issue comes to work. {{{ #!diff diff --git a/string.c b/string.c index c224d7f..407ad62 100644 --- a/string.c +++ b/string.c @@ -165,6 +165,11 @@ str_invert_byte_order(rb_str_t *self) static rb_encoding_t * str_compatible_encoding(rb_str_t *str1, rb_str_t *str2) { + // TODO : #742, #849 + // MacRuby does not support Script Encoding now. + // So, can't check validity of Encoding definitely. + return rb_encodings[ENCODING_UTF8]; +#if 0 if (str1->encoding == str2->encoding) { return str1->encoding; } @@ -182,6 +187,7 @@ str_compatible_encoding(rb_str_t *str1, rb_str_t *str2) return str1->encoding; } return NULL; +#endif } static rb_encoding_t * }}} -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:15> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): It may not be a good idea to stop checking encodings, as it might simply hide genuine user-level bugs. Maybe we should fix #742, #849 etc. instead. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:16> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): r4982 should fix many Encoding exceptions. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:17> MacRuby <http://macruby.org/>
#849: MacRuby throws the exception of "Encoding::CompatibilityError" with various cases. ----------------------------------+----------------------------------------- Reporter: watson1978@… | Owner: lsansonetti@… Type: defect | Status: new Priority: blocker | Milestone: Component: MacRuby | Keywords: ----------------------------------+----------------------------------------- Comment(by lsansonetti@…): We need to revisit this ticket. If there are still cases where MacRuby throws an Encoding exception, we should file a ticket per case. -- Ticket URL: <http://www.macruby.org/trac/ticket/849#comment:18> MacRuby <http://macruby.org/>
participants (1)
-
MacRuby