| Type | Bug | Status | reviewed | Date | 4-Apr-2013 10:07 |
|---|---|---|---|---|---|
| Version | r3 master | Category | Datatype | Submitted by | Ladislav |
| Platform | All | Severity | major | Priority | normal |
| Summary | Internal representation of URLs |
|---|---|
| Description |
URLs are sequences of octets. In R3, URL's are currently internally implemented as "sequences of characters", i.e., as a "special string type". As can be observed, the LOAD function transforms the "external representation" to "internal representation", which in Rebol is a kind of a string type, i.e., (in R3) a sequence of Unicode code points. Moreover, observations (see #2011 and #2013) suggest that only the code points in the 0 to 255 range are used by LOAD at present to "internally encode" any "externally represented URL". Unfortunately, the "external to internal format" transformation is incorrect as #2011 proves, not preserving the "encoded delimiters" transforming them to "unencoded delimiters". Thus, LOAD transformation currently misrepresents (loses) external information, namely the information which delimiters were originally "encoded", i.e., not meant as delimiters. In addition to misrepresenting the "external information" the problem also is that in Rebol URLs are mutable values and Rebol mutating actions should be unrestricted, meaning that any modification should be allowed, producing a valid internal representation. Yet another problem with the current internal representation is that it is rather inefficient, representing octets by Unicode characters. |
| Example code |
| Assigned to | n/a | Fixed in | - | Last Update | 13-Jan-2016 05:32 |
|---|
| Comments | |
|---|---|
|
(0003755)
BrianH 4-Apr-2013 20:59 |
URLs outside of Rebol are defined as a series of octets, but inside Rebol they don't necessarily need to be. Many schemes in Rebol could just be URL-looking syntax for something Rebol-specific, like an ODBC scheme. I think that it should be up to the scheme to handle any encoding to octets at runtime, if such a thing is necessary or possible for that particular scheme (for example it would be possible for HTTP, unnecessary and inappropriate for ODBC).
A bigger problem is how we will need to handle hex encoding, because octets or codepoints, we need to put off decoding encoded characters until DECODE-URL breaks the URL into parts first. We need to handle the URL syntax stuff, splitting into domain, user, password, path and fragment, before any hex decoding is done. |
|
(0003757)
Ladislav 4-Apr-2013 21:19 |
" I think that it should be up to the scheme to handle any encoding to octets at runtime, if such a thing is necessary or possible for that particular scheme (for example it would be possible for HTTP, unnecessary and inappropriate for ODBC)." - In this ticket I wanted to discuss other issues, feeling that the internal representation (format) must be chosen and used before any scheme-specific handling is employed. If it is chosen incorrectly, then correct scheme-specific handling may become impossible, in fact. (which is actually happening at present as I demonstrated) Once the LOAD result loses necessary information, no DECODE-URL can recover the lost information any more. |
|
(0003758)
BrianH 4-Apr-2013 22:22 |
Agreed, that is definitely the problem we have here. Actually, we have two separate problems with the current internal representation: Unicode and escaping.
As mentioned in #482, #1986 and #2013, we need an internal representation that can handle Unicode characters in url! values. On a semantic model level, if we want to keep url! as a member of any-string! then it should be logically a series of char! values, which are Unicode codepoints in R3. On a practical level, we want to be able to use PARSE to process url! values, but we also want to be able to support schemes that can in one way or another handle Unicode, such as modern HTTP and ODBC. The actual matter of how an individual scheme processes Unicode data is up to the scheme, but the internal model needs to be able to store the Unicode data for it to process. That means that whatever internal model we choose will need to be encoded from and decoded to Unicode characters when you are accessing the contents of the url!. It would be best if that decoding is done in the url! type actions themselves and just providing a char! API, hiding the actual internal physical representation so as to avoid data corruption. Escaping is a separate matter, but the internal model does need to take it into account or else, like you say, data is lost before we get to DECODE-URL. Also, this escaping problem is actually scheme-independent, a side effect of basing the url! type more-or-less on the web-standard URL/URI model. The URL/URI data model makes a distinction between characters that are syntax and characters that are considered data. When there is a conflict, where you want to use syntax characters as data, they provide another way of encoding that doesn't conflict with the syntax: percent encoding (let's assume that I am talking about the extended percent encoding that includes Unicode support). This gives them two ways to represent the same character, one that is considered to be syntax, and one that is not. This distinction is good to emulate for the url! data model, even though url! is logically a series of codepoints (any-string!) rather than a series of octets. The external syntax is also good to emulate (see #1986 and #2013 for details), though we might want to also support most printable non-whitespace Unicode characters directly in url! syntax (see #482 and #2013 for details). We definitely need an internal escaping method to make the distinction between syntax and data, because the any-string! model is just a series of characters, and the intended use of a character is not part of the model. That doesn't mean that we want to use the URI/URL percent escaping for our internal data model for url! values. As an internal model, percent encoding has several downsides:
One way to deal with these issues is to use a different method for escaping internally: Use a single normally-nonprinting escape character and don't hex-encode the character it is escaping, just put the escaped character there directly. Internally we have no reason to limit ourselves to printing characters, and use of these characters in data is rare. For that matter, ASCII has a character specifically intended to do this kind of escaping: #"^(1B)". We wouldn't need to escape any characters except the escape itself and syntax characters like "%", ":", "@", "/", "?" or "#" when they are used as non-syntax data. We could even expose the escape character in the user data model so the PARSE rules in DECODE-URL can react to it - since no hex or UTF-8 encoding is involved, the escape character can just be considered internal markup. For that matter, since INSERT, APPEND and CHANGE have /only options that aren't otherwise being used for string types, we can even have their default behavior without /only generate the escape characters directly if we like, or insert pre-escaped data with /only. We could even have a function that takes percent-encoded data and returns escaped data (I mean other than LOAD, since it could work on url! fragments, not just full url! syntax), and another that does the opposite. This would hide the difficulties of percent encoding from users that are generating url! values dynamically rather than writing them out in syntax, and it would maintain equivalence between characters that were unnecessarily percent-encoded and those characters not encoded, which particularly comes in handy for Unicode data. If we decided to use percent encoding internally as the escaping method instead of the ESC character proposal, we should still have LOAD, INSERT, APPEND and CHANGE decode the percent encoding for non-syntax characters, including Unicode, with the same pre-encoded-data /only treatment. MOLD could regenerate that percent encoding for Unicode as I mentioned in #2013, but internally all unnecessary percent encoding should be decoded. We don't want users to have to process percent encoding when they don't need to, and don't want to ever require them to do UTF-8 encoding and decoding themselves. |
|
(0003761)
BrianH 4-Apr-2013 23:42 |
The one trick of all this is that while the need for a Unicode encoding and syntax character escaping model is scheme-independent, the actual encoding and escaping model generated within the scheme handlers themselves would be scheme-specific.
The syntax of the url! type that LOAD accepts and MOLD generates would have one escaping model, which woul;d be scheme-independent. The internal data model could have another encoding method for Unicode, also scheme-independent as long as the url! actions returned decoded characters, and a way to mark escaped characters that is also scheme-independent. The only scheme-dependent thing would be how the port scheme handlers translate the scheme-independent escaped data to the differently escaped data model that the implemented protocol requires. As long as the scheme can determine the difference between syntax and data using DECODE-URL, it can know what to translate to what it requires for its own external purposes. |
|
(0003764)
Ladislav 4-Apr-2013 23:58 |
'wouldn't do any octet encoding at all, they would only be concerned with escaping (which is currently done as a "%" and two hex digits). ' - you are just using a terminology incompatible with the RFC. According to the RFC #3986 the usage of '#"%" followed by two hex digits' is called "percent encoding", which is what I am respecting to maintain consistency. |
|
(0003766)
BrianH 5-Apr-2013 00:28 |
you are just using a terminology incompatible with the RFC
There is no RFC for Rebol syntax or semantics. We are not discussing a limited RFC-compatible external definition of a URL, we are discussing Rebol's definition (which I will call url! to avoid further confusion). The scheme handlers that need to generate RFC3986 URLs will convert url! values to the appropriate octet streams. The scheme handlers that wrap Unicode APIs (like ODBC) won't need to do that kind of conversion, and shouldn't. On the other hand, using RFC3986 as the inspiration for the url! type's external representation (its syntax in Rebol source) is not a bad idea, even though url! values would be Unicode. Regardless of how we decide to implement escape characters internally, we can still have MOLD generate percent encoding (including the UTF-8 sequence encoding for Unicode characters, don't know the RFC for that, but see #1986), and LOAD understand it. There are real advantages to internally encoding these escape characters using a different escape value, particularly one of the non-printing characters rather than "%", and some disadvantages, but that is a different issue from how the url! value is treated by LOAD and MOLD. TO-STRING is another issue, which would depend on the internal escaping method for syntax characters in the url! type. |
|
(0003770)
BrianH 5-Apr-2013 01:18 |
Advantage to using percent-encoding internally:
* Bug-for-bug compatibility with R2. * The syntax resembles the internal data, for better or worse (hint: worse). * We can use an 8-bit-element series to encode a url! internally. |
|
(0003772)
BrianH 5-Apr-2013 01:23 |
"LOAD is accepting an "external representation", which is a string (Unicode, in case of R3)."
Um, no. Unicode is not a binary encoding, it is a series of codepoints which may be encoded in one of any number of binary encodings. LOAD only loads UTF-8 binary data. When you LOAD a string! value, it converts the string to UTF-8 binary before it loads it. Just a clarification, it doesn't matter for the purposes of this ticket. "I.e., LOAD does not convert from UTF-8 to Unicode, but, funnily enough, from Unicode to something resembling UTF-8 (it isn't exact UTF-8 due to the decisions I am describing in this ticket)." Well, for the url! type that is a problem. However, if LOAD actually converted to UTF-8 that wouldn't be a problem, as long as escape sequences were handled somehow internally, and as long as any characters in the url! were decoded from UTF-8 before they were returned. No individual bytes visible to Rebol code, all UTF-8 encoding handled by the internal code, a true internally-variable-length string type. PICK, POKE and INDEX? no longer O(1), sorry. Making the url! type use the same internal encoding as string!, plus the internal escape sequence, that would make PICK, POKE and INDEX? O(1) again. Unless we decided to implement url! and string! as UTF-8 or UTF-16 of course. |
|
(0003775)
BrianH 5-Apr-2013 01:49 |
"and where our points differ" - After your comment edits, our points differ in a completely different way now :) |
|
(0004687)
Ladislav 8-Jan-2016 19:28 |
"... whatever internal model we choose will need to be encoded from and decoded to Unicode characters when you are accessing the contents of the url! It would be best if that decoding is done in the url! type actions themselves and just providing a char! API, hiding the actual internal physical representation so as to avoid data corruption." - hmm, this looks natural, but it probably isn't possible. Any Unicode code point we find in the externally represented URL may either be a real Unicode, or it may be a sequence of octets looking like a Unicode code point only by chance. While in the former case it can be handled on a per-codepoint basis, in the latter case it is necessary to handle it on a per octet basis. |
|
(0004689)
Ladislav 13-Jan-2016 05:32 |
Also, when I copy a URL containing Unicode from the address field of the Chrome browser, I get: https://cs.wikipedia.org/wiki/Karel_%C4%8Capek. When you put it to a browser, you get a different look (Unicode) in the address field. |
| Date | User | Field | Action | Change |
|---|---|---|---|---|
| 13-Jan-2016 05:32 | ladislav | Comment : 0004689 | Added | - |
| 8-Jan-2016 19:38 | ladislav | Comment : 0004687 | Modified | - |
| 8-Jan-2016 19:30 | ladislav | Comment : 0004687 | Modified | - |
| 8-Jan-2016 19:29 | ladislav | Comment : 0004687 | Modified | - |
| 8-Jan-2016 19:28 | ladislav | Comment : 0004687 | Added | - |
| 8-Jan-2016 18:57 | ladislav | Description | Modified | - |
| 8-Jan-2016 18:49 | ladislav | Description | Modified | - |
| 8-Jan-2016 18:42 | ladislav | Description | Modified | - |
| 8-Jan-2016 18:42 | ladislav | Status | Modified | submitted => reviewed |
| 8-Jan-2016 18:10 | ladislav | Description | Modified | - |
| 24-May-2013 11:52 | Ladislav | Comment : 0003764 | Modified | - |
| 24-May-2013 11:51 | Ladislav | Comment : 0003774 | Removed | - |
| 10-Apr-2013 11:33 | Ladislav | Comment : 0003813 | Removed | - |
| 10-Apr-2013 02:13 | Ladislav | Comment : 0003813 | Modified | - |
| 10-Apr-2013 02:10 | Ladislav | Comment : 0003813 | Modified | - |
| 10-Apr-2013 02:08 | Ladislav | Comment : 0003813 | Added | - |
| 6-Apr-2013 00:05 | BrianH | Comment : 0003775 | Modified | - |
| 6-Apr-2013 00:04 | BrianH | Comment : 0003773 | Removed | - |
| 5-Apr-2013 23:22 | BrianH | Comment : 0003770 | Modified | - |
| 5-Apr-2013 22:24 | BrianH | Comment : 0003758 | Modified | - |
| 5-Apr-2013 22:18 | BrianH | Comment : 0003761 | Modified | - |
| 5-Apr-2013 22:17 | BrianH | Comment : 0003766 | Modified | - |
| 5-Apr-2013 22:07 | BrianH | Comment : 0003758 | Modified | - |
| 5-Apr-2013 22:01 | BrianH | Comment : 0003761 | Modified | - |
| 5-Apr-2013 22:00 | BrianH | Comment : 0003761 | Modified | - |
| 5-Apr-2013 22:00 | BrianH | Comment : 0003761 | Modified | - |
| 5-Apr-2013 21:45 | BrianH | Comment : 0003758 | Modified | - |
| 5-Apr-2013 21:29 | BrianH | Comment : 0003766 | Modified | - |
| 5-Apr-2013 20:52 | BrianH | Comment : 0003758 | Modified | - |
| 5-Apr-2013 19:20 | BrianH | Comment : 0003770 | Modified | - |
| 5-Apr-2013 11:07 | Ladislav | Comment : 0003768 | Removed | - |
| 5-Apr-2013 11:06 | Ladislav | Comment : 0003765 | Removed | - |
| 5-Apr-2013 11:06 | Ladislav | Comment : 0003769 | Removed | - |
| 5-Apr-2013 11:05 | Ladislav | Comment : 0003763 | Removed | - |
| 5-Apr-2013 11:05 | Ladislav | Comment : 0003762 | Removed | - |
| 5-Apr-2013 11:04 | Ladislav | Comment : 0003757 | Modified | - |
| 5-Apr-2013 11:04 | Ladislav | Comment : 0003759 | Removed | - |
| 5-Apr-2013 11:03 | Ladislav | Comment : 0003760 | Removed | - |
| 5-Apr-2013 11:03 | Ladislav | Comment : 0003757 | Modified | - |
| 5-Apr-2013 11:01 | Ladislav | Comment : 0003756 | Removed | - |
| 5-Apr-2013 11:00 | Ladislav | Description | Modified | - |
| 5-Apr-2013 02:00 | BrianH | Comment : 0003772 | Modified | - |
| 5-Apr-2013 01:49 | BrianH | Comment : 0003775 | Added | - |
| 5-Apr-2013 01:42 | BrianH | Comment : 0003772 | Modified | - |
| 5-Apr-2013 01:33 | Ladislav | Comment : 0003774 | Added | - |
| 5-Apr-2013 01:32 | BrianH | Comment : 0003772 | Modified | - |
| 5-Apr-2013 01:31 | Ladislav | Comment : 0003771 | Removed | - |
| 5-Apr-2013 01:31 | BrianH | Comment : 0003773 | Added | - |
| 5-Apr-2013 01:23 | BrianH | Comment : 0003772 | Added | - |
| 5-Apr-2013 01:22 | Ladislav | Comment : 0003771 | Modified | - |
| 5-Apr-2013 01:19 | BrianH | Comment : 0003770 | Modified | - |
| 5-Apr-2013 01:18 | Ladislav | Comment : 0003771 | Added | - |
| 5-Apr-2013 01:18 | BrianH | Comment : 0003770 | Added | - |
| 5-Apr-2013 01:07 | Ladislav | Comment : 0003769 | Modified | - |
| 5-Apr-2013 01:05 | Ladislav | Comment : 0003769 | Added | - |
| 5-Apr-2013 00:58 | Ladislav | Comment : 0003768 | Added | - |
| 5-Apr-2013 00:37 | BrianH | Comment : 0003766 | Modified | - |
| 5-Apr-2013 00:35 | BrianH | Comment : 0003766 | Modified | - |
| 5-Apr-2013 00:32 | BrianH | Comment : 0003766 | Modified | - |
| 5-Apr-2013 00:28 | BrianH | Comment : 0003766 | Added | - |
| 5-Apr-2013 00:25 | Ladislav | Comment : 0003765 | Added | - |
| 4-Apr-2013 23:58 | Ladislav | Comment : 0003764 | Added | - |
| 4-Apr-2013 23:52 | Ladislav | Comment : 0003763 | Added | - |
| 4-Apr-2013 23:51 | Ladislav | Comment : 0003762 | Added | - |
| 4-Apr-2013 23:46 | Ladislav | Comment : 0003756 | Modified | - |
| 4-Apr-2013 23:45 | Ladislav | Comment : 0003753 | Removed | - |
| 4-Apr-2013 23:44 | BrianH | Comment : 0003761 | Modified | - |
| 4-Apr-2013 23:44 | Ladislav | Description | Modified | - |
| 4-Apr-2013 23:42 | BrianH | Comment : 0003761 | Added | - |
| 4-Apr-2013 23:40 | Ladislav | Description | Modified | - |
| 4-Apr-2013 23:39 | Ladislav | Description | Modified | - |
| 4-Apr-2013 23:38 | Ladislav | Description | Modified | - |
| 4-Apr-2013 23:20 | Ladislav | Comment : 0003760 | Modified | - |
| 4-Apr-2013 23:16 | Ladislav | Comment : 0003759 | Modified | - |
| 4-Apr-2013 23:04 | Ladislav | Description | Modified | - |
| 4-Apr-2013 23:03 | Ladislav | Description | Modified | - |
| 4-Apr-2013 23:02 | Ladislav | Description | Modified | - |
| 4-Apr-2013 23:00 | Ladislav | Comment : 0003760 | Modified | - |
| 4-Apr-2013 22:58 | Ladislav | Comment : 0003760 | Added | - |
| 4-Apr-2013 22:55 | Ladislav | Comment : 0003759 | Modified | - |
| 4-Apr-2013 22:53 | Ladislav | Comment : 0003759 | Modified | - |
| 4-Apr-2013 22:51 | Ladislav | Comment : 0003759 | Added | - |
| 4-Apr-2013 22:49 | Ladislav | Description | Modified | - |
| 4-Apr-2013 22:40 | BrianH | Comment : 0003758 | Modified | - |
| 4-Apr-2013 22:22 | BrianH | Comment : 0003758 | Added | - |
| 4-Apr-2013 21:39 | Ladislav | Comment : 0003757 | Modified | - |
| 4-Apr-2013 21:38 | Ladislav | Comment : 0003757 | Modified | - |
| 4-Apr-2013 21:34 | Ladislav | Comment : 0003757 | Modified | - |
| 4-Apr-2013 21:20 | Ladislav | Comment : 0003756 | Modified | - |
| 4-Apr-2013 21:19 | Ladislav | Comment : 0003757 | Added | - |
| 4-Apr-2013 21:17 | Ladislav | Comment : 0003756 | Added | - |
| 4-Apr-2013 20:59 | BrianH | Comment : 0003755 | Added | - |
| 4-Apr-2013 11:20 | Ladislav | Comment : 0003753 | Modified | - |
| 4-Apr-2013 10:55 | Ladislav | Comment : 0003753 | Modified | - |
| 4-Apr-2013 10:54 | Ladislav | Comment : 0003753 | Modified | - |
| 4-Apr-2013 10:53 | Ladislav | Comment : 0003753 | Modified | - |
| 4-Apr-2013 10:53 | Ladislav | Comment : 0003753 | Added | - |
| 4-Apr-2013 10:28 | Ladislav | Description | Modified | - |
| 4-Apr-2013 10:28 | Ladislav | Description | Modified | - |
| 4-Apr-2013 10:27 | Ladislav | Description | Modified | - |
| 4-Apr-2013 10:26 | Ladislav | Description | Modified | - |
| 4-Apr-2013 10:07 | Ladislav | Ticket | Added | - |