REBOL3 tracker
  0.9.12 beta 		Ticket #0001986 User: anonymous

Project:
Login | View Tickets



rss
TypeWish Statusreviewed Date6-Mar-2013 09:06
Versionr3 master CategoryNative Submitted byrebolek
PlatformAll Severityminor Prioritynormal

Summary Have DEHEX convert UTF-8 sequences from browsers as Unicode
Description Unicode characters sent in forms from web browsers are now commonly encoded as UTF-8 byte sequences in the %.. notation. For example the character "ø" is encoded as %C5%99. DEHEX cannot understand this yet and converts it badly. See the example code.
Example code 
>> dehex "%c5%99"
== "A?"

>> to binary! dehex "%c5%99"
== #{C385C299} ; bad sequence

>> to binary! "ø"
== #{C599} ; good sequence

Assigned ton/a Fixed in- Last Update11-Apr-2014 16:51


Comments
(0003570)
rebolek
6-Mar-2013 09:08

 I see that curecode has its problems with Unicode too. That "ø" should be r with háèek (caron) - HTML entity ř.
(0003571)
rebolek
6-Mar-2013 09:29

 This looks like a fix (done in 10 minutes so it may have problems on its own):

dehex: funct [
    "Converts URL-style hex encoded (%xx) strings. Process Unicode characters properly."
    value [ any-string! ] "The string to dehex"
][
    value: to binary! value
    parse value [
        some [
            to #"%"
            m: ( change/part m load rejoin [ "#{" to string! copy/part next m 2 "}" ] 3 )
            skip
        ]
        to end
    ]
    to string! value
]
(0003579)
BrianH
6-Mar-2013 23:36

 Change "Process Unicode characters properly." to something like "Process UTF-8 sequences as Unicode." to be more specific and avoid arguments over what "properly" means without any context. Otherwise, good idea if this is both common behavior and according to the current relevant standards.

I don't think this is a bug as much as a wish, since it is working as designed, and this would be a change to its design (minor detail). I'll rephrase the ticket accordingly, though it's the same request either way.
(0003580)
rebolek
7-Mar-2013 00:02

 It was really designed to do this?

>> to binary! dehex "Å™"
== #{C385C299}

That looks like a bug, not a design. If it's design, it's very bad one, because it's not predictable.
(0003581)
rebolek
7-Mar-2013 00:04

 Grrr, should be "%c5%99". This is really stupid format...
(0003582)
rgchris
7-Mar-2013 00:15

 URL-encoded strings refer to bytes, not codepoints—url-encoded strings should be converted to binary before dehexing. The current 'dehex converting hex-encoded pairs to codepoints is broken.
(0003583)
rebolek
7-Mar-2013 00:17

 +1 @rgchris
(0003584)
BrianH
7-Mar-2013 00:04

 DEHEX was designed to decode ASCII sequences before Unicode in a browser was even a thing, and before the URL-encoding standards even applied to Unicode characters. If browsers have started encoding Unicode characters that way then that is a relatively new development, and definitely one that we should adapt to.

Don't take offence at the "Wish" designation. In general "Wish" tickets are considered more important than "Bug" tickets. Maybe when we hit beta we will be able to put a higher priority on "Bug" tickets that aren't easy fixes or crashes.
(0003585)
rebolek
7-Mar-2013 00:16

 I did some research ( = read this Wikipedia entry http://en.wikipedia.org/wiki/Percent-encoding ) and there I found that "The generic URI syntax mandates that new URI schemes that provide for the representation of character data in a URI must, in effect, represent characters from the unreserved set without translation, and should convert all other characters to bytes according to UTF-8, and then percent-encode those values. This requirement was introduced in January 2005 with the publication of RFC 3986." So it's actually not that new development as it pre-dates R3.
No offence taken with the "Wish" designation, I just like to see this fixed in R3 rather sooner than later, as it's basic requirement for R3 as CGI in non-ASCII-only parts of world.
(0003586)
BrianH
7-Mar-2013 01:42

 DEHEX hasn't been redesigned for Unicode yet, so it still has the design from R2. I don't want to argue whether that RFC predates R3, because then we'll just get into how many years it takes these standards to be adopted, and how telling it is that before today I had never heard of a browser implementing that standard (not as telling as you'd think since I haven't done web development in a couple years). When it comes down to it it's a good idea to make this change for the reasons you gave above.
(0003587)
abolka
7-Mar-2013 02:54

 "DEHEX was designed [..] before Unicode in a browser was even a thing."

I distinctly remember having to tweak a registry setting in Win95 to tinker with Netscape 3 and Unicode. Wikipedia says, Netscape 3.0 was released in 1996; so maybe, Unicode in a browser existed before REBOL was even a thing :)

More on topic: whether under the flag of Wish or Bug, this change is necessary (*) to support Unicode correctly and consistently. On the whole, current DEHEX in R3 is simply wrong. (If this wrongness can be argued to be "by design", then other changes in R3 simply invalidate the previous design).

(*: Alternative fixes are thinkable, but, I think, unreasonable: error out when DEHEX encounters a value >127. Or change the docstring of DEHEX to remove the "URL-style" reference and simply define DEHEX as an additional Rebol-internal method of escaping the first 256 Unicode codepoints.)

So: +1
(0003597)
BrianH
8-Mar-2013 21:11

 So, looking through the code, it appears that DEHEX can operate on 8-bit-element binary series and 16-bit-element strings. I don't know when the 8-bit DEHEX is ever called (perhaps it's a left-over from when binary! was a part of any-string!), but one thing that is clear is that it doesn't have a way to change the series to 16-bit-elements mid-stream if it encounters a codepoint that is greater than 255. That means that we can't support this for DEHEX running in 8-bit mode, assuming it ever does. However, some theoretical DEHEX of a binary would just generate UTF-8 binaries for this input using the old method, so we'll probably be fine for now.

If we are willing to limit this to 16-bit DEHEX this can be done. If it's OK with you all I'll have it generate the invalid-character codepoint when it decodes a codepoint outside of the BMP, at least as a stopgap until we support those in R3 at all. And there is no reason to complain about overlong UTF-8 encodings since we're generating Unicode strings in the first place - the restriction against overlong encodings is for when you're working with encoded UTF data directly instead of decoding it first.
(0003812)
Ladislav
10-Apr-2013 01:52

 That #"ř" should be #"^(0159)" - as I see it works well in Curecode, so I wonder what exactly Bolek messed up? (probably his browser setting)
(0004401)
abolka
11-Apr-2014 16:51

 In the core-tests suite.

Date User Field Action Change
11-Apr-2014 16:51 abolka Comment : 0004401 Added -
10-Apr-2013 01:53 Ladislav Comment : 0003812 Modified -
10-Apr-2013 01:52 Ladislav Comment : 0003812 Added -
8-Mar-2013 21:11 BrianH Comment : 0003597 Added -
7-Mar-2013 04:33 abolka Comment : 0003587 Modified -
7-Mar-2013 04:32 abolka Comment : 0003587 Modified -
7-Mar-2013 03:00 abolka Comment : 0003587 Modified -
7-Mar-2013 02:54 abolka Comment : 0003587 Modified -
7-Mar-2013 02:54 abolka Comment : 0003587 Added -
7-Mar-2013 01:42 BrianH Comment : 0003586 Added -
7-Mar-2013 00:17 rebolek Comment : 0003583 Added -
7-Mar-2013 00:16 rebolek Comment : 0003585 Added -
7-Mar-2013 00:15 rgchris Comment : 0003582 Added -
7-Mar-2013 00:04 BrianH Comment : 0003584 Added -
7-Mar-2013 00:04 rebolek Comment : 0003581 Added -
7-Mar-2013 00:03 rebolek Comment : 0003580 Modified -
7-Mar-2013 00:03 rebolek Comment : 0003580 Modified -
7-Mar-2013 00:02 rebolek Comment : 0003580 Added -
6-Mar-2013 23:44 BrianH Type Modified Bug => Wish
6-Mar-2013 23:44 BrianH Status Modified submitted => reviewed
6-Mar-2013 23:44 BrianH Code Modified -
6-Mar-2013 23:44 BrianH Description Modified -
6-Mar-2013 23:44 BrianH Summary Modified DEHEX converts Unicode charaters badly. => Have DEHEX convert UTF-8 sequences from browsers as Unicode
6-Mar-2013 23:40 BrianH Comment : 0003579 Modified -
6-Mar-2013 23:38 BrianH Comment : 0003579 Modified -
6-Mar-2013 23:36 BrianH Comment : 0003579 Modified -
6-Mar-2013 23:36 BrianH Comment : 0003579 Added -
6-Mar-2013 09:29 rebolek Comment : 0003571 Added -
6-Mar-2013 09:08 rebolek Comment : 0003570 Added -
6-Mar-2013 09:06 rebolek Ticket Added -
Copyright @ 2007-2010 Softinnov - Powered by Cheyenne