Talk:UTF-8
| This is the talk page for discussing improvements to the UTF-8 article. This is not a forum for general discussion of the subject of the article. |
Article policies
|
| Find sources: Google (books · news · scholar · free images · WP refs) · FENS · JSTOR · TWL |
 | This  level-5 vital article is rated B-class on Wikipedia's content assessment scale.It is of interest to the following WikiProjects: | |||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||
Table should not only use color to encode information (but formatting like bold and underline)
As in a previous comment https://en.wikipedia.org/wiki/Talk:UTF-8/Archive_1#Colour_in_example_table? this has been done before, and is *better* so that everyone can clearly see the different part of the code. Relying on color alone is not good, due to color vision deficiencies and varying color rendition on devices. — Preceding unsigned comment added by 88.219.179.109 (talk • contribs) 02:26, 17 April 2020 (UTC)
Should "The Manifesto" be mentioned somewhere?
More specifically, this one: https://utf8everywhere.org -- Preceding unsigned comment added by Rudxain (talk o contribs) 21:52, 12 July 2024 (UTC)
- Only if it's got significant coverage in reliable sources. Remsense 22:10, 12 July 2024 (UTC)
- It's kind of ahistorical, since the Microsoft decisions that they deplore were made while developing Windows NT 3.1, and UTF-8 wasn't even a standard until Windows NT 3.1 was close to being released. There was more money to be made from East Asian customized computer systems than Unicode computer systems in 1993, so Unicode was probably not their main focus at that time... AnonMoos (talk) 20:30, 15 July 2024 (UTC)
The number of 3 byte encodings is incorrect
This sentence is incorrect:
Three bytes are needed for the remaining 61,440 codepoints...
FFFF - 0800 + 1 = F800 = 63,488 three byte codepoints.
The other calculations for 1, 2, and 4 byte encodings are correct. Bantling66 (talk) 02:56, 23 August 2024 (UTC)
- You forgot to subtract 2048 surrogates in the D800–DFFF range. – MwGamera (talk) 08:58, 23 August 2024 (UTC)
Multi-point flags
I'm struggling to assume good faith here with this edit. A flag which consists of five code points is already sufficiently illustrative of the issue being discussed. That an editor saw fit to first remove that example without discussion, and then to swap it out for the other example when it was pared down to one flag, invites discussion of why that particular flag was removed, and the obvious answer isn't a charitable one. Chris Cunningham (user:thumperward) (talk) 12:35, 17 September 2024 (UTC)
- Yes it was restored to the pride flag for precisely the reasons you state. Spitzak (talk) 20:48, 17 September 2024 (UTC)
- A better, more in-depth explanations of the flags can be found on the articles regional indicator symbol and Tags_(Unicode_block)#Current_use (the mechanism for these specific flags). I don't think it belongs in articles of specific character encodings like UTF-8 at all.
- The fact that one code point does not necessarily produce one grapheme has nothing to do with a specific character encoding like UTF-8. It's a more fundamental property of the text itself and any encoding that can be used to encode some string of characters decodes back to the same characters when decoded back from the binary representation. Although very popular, UTF-8 is just one of the numerous ways to encode text to binary and back.
- I wrote more about this below at Other issues in the article and sadly only then noticed this was already being somewhat discussed here. Mossymountain (talk) 10:45, 20 September 2024 (UTC)
Why was the "heart" of the article, almost the whole section of UTF-8#Encoding (Old revision) removed instead of adding a note?
NOTE: The section seems to have been renamed (UTF-8#Encoding -> UTF-8#Description) in this edit.
I don't understand why such a large part of UTF-8#Encoding (old revision) was suddenly removed in this edit (edit A), and then this edit (edit B) (diff after both edits) instead of either:
- Adding a note about parts of it being written poorly.
- Rewriting some of it. (the best and the most difficult option)
- Carefully considering removing parts that were definitely redundant (such as arguably the latter part of UTF-8#Examples (old revision)).
Both of the edits removed a separate, and quite a well-written example (at least for my brain, these very examples made understanding UTF-8 require significantly less effort spent thinking). I don't think removing them was a good decision. Yes, you could explain basically anything without using examples, but in my experience an example is usually the easiest and fastest way for someone to understand almost any concept, especially when the examples were so visual and beautifully simple. I see it in the same category as a lecturer speaking with his hands and writing+drawing relevant things on a whiteboard versus having to hold the lecture by speaking over the phone.
The 1st, edit A
→Encoding: this entire section is almost completely opaque and its inclusion stymies the addition of some clear prose describing how unicode is decoded — user:Thumperward, (edit A)
To me, this reads as if UTF-8 was accidentally conflated with Unicode, causing a mistake to remove the parts from the wrong article (Having thought about it more, I now think it's) a severe disagreement of article design/presentation style.
(I still think edit notes asking for rewrites would have been the way to go instead of nuking the information, and that for some of the items, an article-like rewrite would be the wrong choice: Some data is way more enjoyable and simple to read visually from a table than it is to glean from written or spoken word and, as such, should be visualized in a table.)
I am strongly of the mind that the deleted parts included the two most important parts of the whole article, that must definitely be included as they are the very core of the article:
- The UTF-8#Codepage layout (old revision), in my opinion the most important part of any article about a character encoding. This part was in my opinion also designed, formatted and written exemplarily well here. The colour palette could be adjusted accordingly if it's a problem for the colour-blind.
- Precedents/Examples in other articles about specific character encodings:- Variable multi-byte ("cousins of UTF-8"):
- single byte:
- ASCII#Character_set (a strict subset of UTF-8)
- Code_page_437#Character_set
- ISO/IEC_8859-1#Code_page_layout
- The first list (numbered 1..7) of UTF-8#Examples (old revision) that clearly, by a singular simple example demonstrates how UTF-8 works. (I agree it could be rewritten, the language used is quite verbose)
Sweeping the less important items under these rugs to make this seem shorter:
The 2nd, edit B
|
|---|
The 2nd, edit B
This edit removed the whole section UTF-8#Overlong encodings (old revision). I disagree with its removal.
|
About removing helper colours and edit C
|
|---|
|
The 3rd, edit CThis is about font colouring on UTF-8#Encoding (old version), it reverts this edit by User:Nsmeds. The textual information stays the same between the two, the edit only removes the custom colours. I would prefer some form of colouring to be added back.
This is attempting to fix a potential issue for the colour-blind, but I think it unfortunately only ends up denying the help the colour was there to provide from both the colour-blind and not. The colours were NEVER the primary way to convey any data, but an additional help to make the parsing of the information faster and less straining to the eye (removing the need to count anything, you don't need to know that a hex digit covers 4 bits, or that the 0x7 on the left column corresponds to the first xxx on the right, and whether you do or don't, you just instantly see the relationship without thinking. This is obviously highly desirable in data visualization. Even without doing anything to suboptimal colours, when they are only potentially hard to distinguish from each other instead of the background, the remaining distinguishable groups still serve the original purpose, only with some of it missing or hard to see. The monochrome version ends up being strictly worse.
I think these should be considered before removing colour outright:
I think the least total effort catch-all long-term solution would be to provide a site-wide toggle on the side that overrides all text and background colouring when you want, probably makes sense beside the existing "Light" and "Dark" mode toggles, to force foreground elements close to the opposite end.  The other solution to fix all of what edit C attempted to fix, (and the solution applicable right here and now) would be to use a palette that is also readable for the colour blind, such as these three palettes found on Color_blindness#Ordered_Information that can be used to produce distinct colours that work no matter of colour-blindness. NOTE: They ALL work for ALL types of colour blindness, it's just a choice of which one looks the nicest.
|
Other issues in the article (solved)
|
|---|
|
The UTF-8 article does talk about generic things about Unicode quite a bit more than I think it should, such as explaining how some "graphical characters can be more than 4 bytes in UTF-8". This is because Unicode (and by extension UTF-8) does not deal in graphemes in the first place, but code points (essentially just numbers to index into Unicode), which can correspond to valid Unicode characters, which in turn can directly correspond to a grapheme. Some characters don't correspond to a grapheme at all (control characters), such as the formatting tag characters used in the flag example, and some combine/join with other character(s) to to produce a combination grapheme (combining/joining characters).
|
Mossymountain (talk) 05:09, 20 September 2024 (UTC) Mossymountain (talk) 17:10, 20 September 2024 (UTC)
- Because the editor was offended that that section used color. Akeosnhaoe (talk) 08:56, 20 September 2024 (UTC)
- It's pretty important that we not communicate information solely through color, but I wonder how we could better do something like that. Remsense ‥ 论 09:02, 20 September 2024 (UTC)
- Most of the information wasn't in the color, it was in the text readable without formatting in monochrome. The color was there just to make it easier to quickly identify which is which.
If what Akeosnhaoe said is the case (which I don't think it is, I think this was an honest misunderstanding with good intentions), obviously the colors should be changed to the intended visibility standard, not the information removed. Mossymountain (talk) 10:17, 20 September 2024 (UTC)
- Most of the information wasn't in the color, it was in the text readable without formatting in monochrome. The color was there just to make it easier to quickly identify which is which.
- It's pretty important that we not communicate information solely through color, but I wonder how we could better do something like that. Remsense ‥ 论 09:02, 20 September 2024 (UTC)
IMHO the edits made by user:thumperward were a good and powerful attempt to remove the obscene bloat of this article. The enourmous complex "examples" with color did not provide any information, and it is quite impossible to figure out what the colors mean without already knowing how UTF-8 works already. Elimination of the "code page" is IMHO a good and daring decision, one I may not have made and I'm glad he tried it. I'd like to continue, pretty much removing the bloated mess of "comparisons" that are either obvious or that nobody cares about, the few useful bits of info there can be merged into the description.Spitzak (talk) 18:13, 20 September 2024 (UTC)
- My most important point, by far, is that I vehemently disagree with the removal of the code page.
- It is the single thing with the most useful information packed on the article and irreplaceable in utility. I don't understand what was wrong with it at all. I see its removal as the same kind of hindrance as deleting all of the drawings that visualize what measurements the letters
h, r, drepresent on a cylinder from that article.
This makes it a lecture where the professor can only attend by talking over the phone. No gestures, no diagrams, nothing. It "does still work", it's just requires more effort from the students (and from the professor, but that's a one-time cost here) Yes, you technically still can glean all of the same information by reading through the article and spending effort to understand what you read, but it would outright DENY the use case where one just looks at a picture or two for a couple of seconds and is already able to close the article, while hindering the rest of the readers by not providing the still useful clarification as study aides. - I'm firmly in the camp that believes that for virtually all human readers, some well thought out visualizations illustrating some concept's defining characteristics only help in understanding, they are the best way to essentially convey "what something looks like", be it logically (like in this case) or physically. I personally have visited the UTF-8 page specifically for the code page for years whenever I needed a refresher when dealing with the encoding. Sure, I could have dug up a cumbersome specification and ^F'd through it to achieve the same thing in at least double the time, but the article was easily the best resource I've found on the internet for understanding UTF-8, largely thanks to how well the code page was thought out and put together.
- I have only read some of the other text on the article previously, never in full before and I agree the article has had problems with bloat. In my mind this still does not mean the most useful thing should be removed in favour of briefness (it's essentially just a picture/diagram, but one that you can interact with to get more out of. The readers can easily identify that rough class of thing and skip it when they don't want to inspect it. It's very obviously not part of the text you're supposed to read out loud for example.) Mossymountain (talk) 05:36, 21 September 2024 (UTC)
I'm not relitigating basic, universally-understood concepts such as "articles should not be hundreds of kilobytes long", "articles should not use colours to convey important information" or "articles are not supposed to be reference textbooks". These are simply settled consensus. The code page table is absolutely useless for any purpose other than implementing handling of the format, which is categorically not the point of an encyclopedia article. What this article should do is explain where UTF-8 fits into the world, how it has been adopted, and how at some basic level it works. Precisely what any given sequence of bytes happens to stand for (other than in explaining how the byte sequence informs multi-byte code points) is not pertinent, especially because the lowest seven bytes were very deliberately copied from ASCII anyway.
Frankly, the major thing I gleaned from the above wall of text (and that on my talk page) is that the editor posting it hasn't actually read the article very closely. A lot of the trimming down that was performed on the text was precisely because the article should put more emphasis on UTF-8's unique features, primarily its variable-length encoding and how multiple code points can be combined into a single glyph. I argued against the (seemingly political) removal of some of that detail in the previous section of this talk page, so it makes no sense to argue that this has somehow been de-emphasised by the removal of unrelated trivia.
This article still needs a lot of work. What it does not need is the re-addition of huge, heavy blocks of content of absolutely no value outside of a reference textbook. Chris Cunningham (user:thumperward) (talk) 11:03, 21 September 2024 (UTC)
- I am not arguing those points. At least I don't think I am. The closest one is probably the third one: "articles are not supposed to be reference textbooks". I will happily concede my positions whenever I get how they break them. (I'm unable to find what you're referencing here, but what I'm arguing for shouldn't be in conflict with it, at least not with what kind of idea I assume the phrase is getting at)
- "How multiple code points can be combined into a single glyph" has nothing to do with UTF-8. I wrote about this at #Other issues in the article above.
- Combining differing amounts of bytes to single code points on the other hand is the defining characteristic of a variable length character encoding, such as UTF-8 and its "cousins", like Shift JIS and GBK. (The links go to the respective code page layout-equivalents on the articles.)
- I have read the full article, as I said here when talking about how the code page has been very useful for me personally; "I have only read some of the other text on the article previously, never in full before and I agree the article has had problems with bloat." (Emphasis added, I didn't catch how ambiguous this was when proofreading!)
- I think one of the best things about such a table/picture is how it helps you build a mental map in order to get a better understanding about what you're reading: It's essentially the "picture" of the thing, what it logically looks like. Especially with colour (or some other way to subconsciously differentiate sections), it's a powerful way to visually identify and to "map" it in the brain for better understanding. This leverages the fact that visual recognition is the single strongest way for humans to match patterns and receive data. This process is largely automatic, and thus requires very little effort in comparison to constructing the "map" from scratch by reading rules about the subject. "A picture is worth a thousand words" etc. etc. This is more true the more complicated a subject is. I compared this to using diagrams on articles about mathematical concepts in the #cylinder example.
- Some topics benefit greatly from such additional illustration and I believe this is one of those cases. I think that articles like this SHOULD at least show the corresponding code page, as it efficiently and intuitively summarizes the encoding. As I wrote above at "#Precedents/examples in other articles", it looks like all similar articles about 8-bit(== such a table is small) character encodings have an equivalent table or picture.
- I previously thought it was neat how UTF-8's table had additional information sprinkled in (like the hover-over Unicode ranges per start byte), but I can see how this is just extra clutter. Shift_JIS#Shift_JIS_byte_map is very clean in comparison, only listing the actual code points as text.
- About the code page being "useless for any purpose other than implementing handling of the format"; I think this is almost the other way around. In comparison to reading about a topic, when programming something I want the written details/rules instead. A picture can also help, but mainly because it helps me understand the thing itself better in general, just like when just reading about it for my own sake.
- I currently interpret the rationale for edits A and B as
Since these poorly laid out sections have both internal and external repetition, while not even close to proper essay form, it should all be removed in order to make it more inviting for someone to later write about things including some of the points from these sections. Currently, virtually no one would probably even attempt to do that because it would always end up repeating these sections, and gradually removing parts from such a consolidated an interdependent form of data is virtually impossible.
- I agree with that in general. It's just that I found the approach almost irresponsibly heavy-handed.
- I think the main disagreement here is whether an appropriate article should include technically redundant (able to be deduced when consciously spending effort to) illustrations or examples, when the rules are already explained in pure writing. I think a tiny number of pertinent examples and clarifying illustrations can greatly enhance the readability/ease of understanding of topics like this. Both to help make readers previously unfamiliar with topic ready to accept the details and to give returning ones a quick refresher, drastically reducing the need to read much of the text itself again. In addition to that, I'd wager most readers don't always read full articles (or even paragraphs), but instead try to skim through to find something they're after and illustrations and examples are precisely those kind of "gold nuggets"; dense, yet easily digestible information. (When time is of the essence, I definitely do this in order to "wring the information out" and these kind of things help a lot.)
- I don't think every guideline about what the ideal article should look like is supposed to be followed as strictly as technically possible and the resulting prototype applied 1:1 on every article to harshly cull the inharmonious parts off.
In regards to colours...(merged to #Helper colours above.) Mossymountain (talk) 06:30, 22 September 2024 (UTC)
Unicode no. of characters wrong
Unicode has 1,111,412 characters. Please make this change. FrierMAnaro (talk) 14:17, 31 October 2024 (UTC)
- 0x110000 is 1,114,112, but the number shown is after subtracting the 2048 surrogate halves (I disagree but the consensus was that they should not count) Spitzak (talk) 17:58, 31 October 2024 (UTC)
- Indeed, the Unicode Standard explicitly states it
contains 1,114,112 code points
right in its introduction, but there are much fewer characters. We're just quite loose in distinguishing between code points, characters, Unicode scalar values, and not well-defined ad-hoc phrases likevalid Unicode code points
as currently used in the second paragraph of the article. UTF-8 does not encode "code points" or "characters" but "Unicode scalar values" (D76). There are 1,112,064 of these. Not all are assigned to characters yet; some are explicitly designated noncharacters. UTF encodings can encode them all, but there are no well-formed sequences of code units that would represent surrogate code points. The wording is grossly imprecise, but the numbers are correct. – MwGamera (talk) 23:10, 31 October 2024 (UTC)- I changed it to say "Unicode scalar values" and added a citation of the Unicode 16.0.0 standard to the reference for the number. Guy Harris (talk) 21:53, 1 November 2024 (UTC)
- Indeed, the Unicode Standard explicitly states it
- Surrogate halves are "code points", but they are not themselves individually "characters" in the most common meaning of the term. They're elements which can be used in pairs to encode characters. AnonMoos (talk) 18:17, 1 November 2024 (UTC)
Tooltips for code points
Can you add a tooltip? Add a tooltip to every cell of the table which shows the range of code points the byte can encode. Also add tooltips for characters beyond the 10FFFF. FrierMAnaro (talk) 07:14, 17 November 2024 (UTC)
Alternative conversion table
I have always found the conversion table a little confusing, so I made a more simple alternative.
https://x.com/LatinSuD/status/1869138590271488375/photo/1
If you like it, I (or somebody) could try to complete it and convert to SVG maybe? LatinSuD (talk) 22:09, 17 December 2024 (UTC)
- We do not recommend additional media rendered as images what should really be text. Remsense ‥ 论 22:28, 17 December 2024 (UTC)
- Looks kind of nice, but there is a desire to keep the table resembling the references, which just use text. Spitzak (talk) 00:30, 18 December 2024 (UTC)
Clarification on overlong encodings being a "vulnerability".
I propose something among the following lines. Overlong encodings on their own are not a vulnerability. What is a vulnerability is (in the example given with ../) trying to match raw ASCII and UTF-8 bytes and thinking they're always equivalent for the same characters, and then sending that incorrectly "sanitized" input to a file system library or something that is UTF-8 aware and will decode it. When doing pattern matching across code points, overlong encoding ceases to be a problem because all strings are already decoded.
So is not an inherent "security flaw" of UTF-8 as a format, rather it's a library bug (or an inherent flaw in merely trying to detect the string .. in paths depending on your view). This does not appear to be reflected in the current section. ThanForIllegitimateAsThisDean (talk) 15:49, 24 February 2026 (UTC)
- Imagine some code that has the job of "don't allow `../` to be in the filename". It can (correctly) just look for this 3-byte pattern. If the filename is then passed to a system that has to decode the UTF-8 (perhaps because the filesystem uses UTF-16), and this one is incorrectly written, it may decode another series of 4-6 bytes into `../` and allow a blackhat to bypass the check. The standard says that it is this translator that is broken, not the checking code. Spitzak (talk) 19:39, 24 February 2026 (UTC)
- You are wrong about the correct behavior. It absolutely is NOT the front ends responsibility to recognize every possible sequence that some broken backend will turn into '.' or '/'. Also you claim that it just has to "decode" the UTF-8 to do this, but that is basically saying the "decoding" has to recognize overlong encodings, along with every other possible mistake that a backend might do. This is impossible. Spitzak (talk) 19:45, 24 February 2026 (UTC)
- You also seem to be confused into thinking that UTF-8 cannot be used unless it is "decoded". That is false. Its entire design is so there is a unique byte pattern for each string, and therefore testing strings for equality, concatenation, and using them to identify objects, can all be done without "decoding". Spitzak (talk) 19:47, 24 February 2026 (UTC)
- For reference, overlong encodings being strictly illegal wasn't specified until well into 2003. The point was that if every string is converted to runes before matching, two options are available: Either the overlong encodings are accepted, and the match is done right anyway (since exactly one kind of rune exists for each character), or the overlong encodings are rejected and the string is declared invalid. Neither of those options are possible with just raw byte matches. ThanForIllegitimateAsThisDean (talk) 22:31, 24 February 2026 (UTC)
- (That's "rune" as in "code point in Unicode".) Not all code converts UTF-8 or UTF-16 to UTF-32 when, for example, checking for "../" in pathnames or URLs, so presumably we're not assuming conversion to runes here. Guy Harris (talk) 21:32, 25 February 2026 (UTC)
- Not in the specific cases where the bug occurs (at least I checked for IIS). And it wasn't really the point of correct behaviour or not I wanted to debate, just an elaboration. Perhaps my other reply was too much of a tangent on the mechanisms of processing encodings or matching, and how to handle incorrect cases. Just wouldn't prefer to label it as an "unicode vulnerability" as the IIS source and the current paragraph in the article appears to present it as. ThanForIllegitimateAsThisDean (talk) 13:55, 26 February 2026 (UTC)
- (That's "rune" as in "code point in Unicode".) Not all code converts UTF-8 or UTF-16 to UTF-32 when, for example, checking for "../" in pathnames or URLs, so presumably we're not assuming conversion to runes here. Guy Harris (talk) 21:32, 25 February 2026 (UTC)
- For reference, overlong encodings being strictly illegal wasn't specified until well into 2003. The point was that if every string is converted to runes before matching, two options are available: Either the overlong encodings are accepted, and the match is done right anyway (since exactly one kind of rune exists for each character), or the overlong encodings are rejected and the string is declared invalid. Neither of those options are possible with just raw byte matches. ThanForIllegitimateAsThisDean (talk) 22:31, 24 February 2026 (UTC)
- I see I'm not the first to take issue with this paragraph. I was going to suggest rewriting it as:
[...] is termed an overlong encoding. Because some software libraries will decode these and others will not, they have been used to bypass simple security checks and perform attacks such as XSS[CN] and path traversal.[IIS ref][Tomcat ref] It's advised that decoders treat overlong encodings as invalid.
- There's now an explicit warning about these in the current version of RFC3629, though really the authors of the mentioned software should have been validating the final value to be used, which clearly they were not. YoshiRulz (talk) 21:05, 24 February 2026 (UTC)
- No, you seem to be misunderstanding and insisting that you must "decode" the UTF-8 to correctly handle it.
- The code that checks for the three bytes `../` is CORRECT
- The code that converts some other sequence of bytes into `../` is WRONG
- Any description must make it clear which code is wrong. It is not "differences in decoding", it is a broken decoder. Spitzak (talk) 19:23, 26 February 2026 (UTC)
- You've assumed that the vulnerable software had separate components for validating the input and adapting it for use with the filesystem, which I think is reasonable, but neglects that someone had to combine those components. And this combining was done with maybe a type check ("both functions take a
u8*, that means string, perfect") but without carefully comparing the invariants of each. - Code which searched only for
[ 0x2E, 0x2E, 0x2F ]would be correct under the assumption that the input was in the form of valid UTF-8 without overlong encodings. Similarly, a UTF-8 to UTF-16 converter which accepted overlong encodings could be correct. It's only when used with input of an unexpected kind that problems arise. I'd hope these invariants were documented, but even the best documentation is no guarantee that a programmer won't accidentally violate them, which is why we have things like refinement types. YoshiRulz (talk) 21:52, 1 March 2026 (UTC)- NO. By definition, a converter that accepts overlong encodings is WRONG. This reduces the number of "variants" anybody needs to consider to exactly one. Anything else is not UTF-8. Spitzak (talk) 19:37, 2 March 2026 (UTC)
- As ThanForIllegitimateAsThisDean noted, the definition wasn't the same in the past (before Unicode 3.1) and so the situation described would indeed be possible historically. – MwGamera (talk) 00:38, 3 March 2026 (UTC)
- NO. By definition, a converter that accepts overlong encodings is WRONG. This reduces the number of "variants" anybody needs to consider to exactly one. Anything else is not UTF-8. Spitzak (talk) 19:37, 2 March 2026 (UTC)
- You've assumed that the vulnerable software had separate components for validating the input and adapting it for use with the filesystem, which I think is reasonable, but neglects that someone had to combine those components. And this combining was done with maybe a type check ("both functions take a
- Perhaps the § Overlong encodings could be phrased more categorically to state that there are simply no such overlong encodings in UTF-8, but that there were in the past and their existence made it trickier to implement some things correctly, leading to security (and other!) problems as per refs. Cf. UTR #36 § 3.1 UTF-8 Exploit. But it's not "advised" to treat them as invalid, it's a conformance requirement for Unicode (clause C10 with ref to D92). – MwGamera (talk) 01:09, 3 March 2026 (UTC)
- No, you seem to be misunderstanding and insisting that you must "decode" the UTF-8 to correctly handle it.
- Alright, I've let this discussion sit for a while (far too long honestly, is everybody still here?) but nothing seems to be happening anymore. At this point I'm starting to think more about what to actually write in the article. The suggestion @YoshiRulz wrote above does work as a base. Of course we'd have to replace "advised" with "mandated by the specification", though I personally put more emphasis on the treatment of overlong encodings in software libraries being inconsistent. Perhaps an example (though not too wordy) can be given in the paragraph as to how exactly this causes XSS, but just having the references there is a start. I'll leave it to you for now. ThanForIllegitimateAsThisDean (talk) 12:28, 25 March 2026 (UTC)
- NO! The code to make sure "../" is not in a string needs to look for exactly ONE 3-byte pattern. This simple validator is CORRECT and is run before (or likely without ever) "converting to final result". If whatever code does "converting to final result" turns some other pattern of bytes into "../" then that code is WRONG, the validator is still CORRECT. Do not put in any text that implies otherwise! Spitzak (talk) 17:53, 25 March 2026 (UTC)
- More specifically, I strongly object to "the authors of the mentioned software should have been validating the final value to be used". That is absolutely wrong. In this case "the authors must fix the 'convert to final result' step". Spitzak (talk) 17:56, 25 March 2026 (UTC)
- At this moment neither of those were in any draft. The first thing that came up was the bugs being present and how they exist. The whole details of who does the reporting on overlong encodings doesn't really matter as long as something does, and the paragraph was to mention that this is not always a given in practice. ThanForIllegitimateAsThisDean (talk) 20:46, 25 March 2026 (UTC)
- What is the problem with the current wording at UTF-8#Overlong encodings? It says that overlong encodings are a security problem and should be considered an error and never decoded. The OP seems to suggest that checking for the three bytes
../is inadequate because subsequent processing by a broken library might accept an overlong encoding. Instead, a checker should decode the UTF-8 into code points, and pass the checked code points for subsequent processing. Regardless of whether that would be desirable, this article has to describe what UTF-8 is. The section is correct as far as I can see. Johnuniq (talk) 02:54, 26 March 2026 (UTC)- Have to admit my original post was poorly worded, but my problems involve how bugs occur. At this point it's only stated that overlong encodings cause bugs, which while having sources does not directly state any details. Which to me was vague because if you decode all your input universally (whether passing overlong encodings or not) you do not have those bugs at all. Pedantic or not, I still prefer a slight clarification. Same with me referring to the vulnerability being in the libraries as opposed to in overlong encodings themselves, as stated right now ("...is termed an overlong encoding. These are a security problem..."), though that is less important.
- The semantics of existing libraries are not really relevant to this paragraph beyond the fact they are inconsistent as stated in the messages above (having both 8-bit clean pattern matching and actually UTF-8 aware reading that happens to pass overlong encodings, which is the crux of the vulnerabilities).
- And as was also noted, it's not that passing overlong encodings merely "should be considered" an error, but it's mandated by spec that it's an error, this is also not mentioned in the current paragraph.
- This doesn't exactly summarise all we were discussing, but it is the gist of my original point plus a bit more. ThanForIllegitimateAsThisDean (talk) 20:00, 26 March 2026 (UTC)
- I agree with your basic sentiment which, I believe, is that security is critical and to suggest that something bad "should" be considered an error is totally inadequate from a security point of view. However, in the context of this article that wording is reasonable. Perhaps something better can be devised but generally articles do not give advice (a Wikipedia defect in a case like this). Looking at it another way, anyone relying on this article when coding a UTF-8 decoder is going to make a mess of it regardless of the wording here. Johnuniq (talk) 00:15, 27 March 2026 (UTC)
- This isn't really wikipedia giving advice. "is an error" is just referring to the specification saying so, explaining how the bugs occur is just explaining it, not in and of itself giving advice. Merely providing a brief summary on what the sourced studies found (and again, how things happened) should be plenty enough for the purposes of this article. ThanForIllegitimateAsThisDean (talk) 20:01, 27 March 2026 (UTC)
- I agree with your basic sentiment which, I believe, is that security is critical and to suggest that something bad "should" be considered an error is totally inadequate from a security point of view. However, in the context of this article that wording is reasonable. Perhaps something better can be devised but generally articles do not give advice (a Wikipedia defect in a case like this). Looking at it another way, anyone relying on this article when coding a UTF-8 decoder is going to make a mess of it regardless of the wording here. Johnuniq (talk) 00:15, 27 March 2026 (UTC)
- I share the sentiment that the section is at best vague, not being helpful for the layman nor for a programmer interested in this class of vulnerabilities. As I see it, the current problems are that 1a) it implies overlong encodings are a security problem, rather than lead to security problems; 1b) it implies overlong encodings are something allowed by the spec, which I suppose was true of the original version, but as mentioned above the spec now explicitly prohibits their use; 2) it states that overlong encodings
allow e.g. malicious JS and
, which IMO simply../to bypass security validationsneeds to be copyeditedneeded a copyedit, now done; and most importantly 3) it states that overlong encodings lead to security problems andshould therefore be considered an error and never decoded
, while in fact they were always erroneous, shouldn't be decoded for spec-conformance reasons, and the security problems arose from the meeting of conformant and non-conformant implementations. YoshiRulz (talk) 00:31, 9 April 2026 (UTC)- Correct: Overlong encodings are not a security problem. But decoding overlong encodings is a security problem. The best way to avoid this security problem is to NOT DECODE. For instance if the file system only looked for the byte that is an ASCII '/' and all other bytes are used unchanged as the "file names", it would work fine. This is despite the fact that it never "decodes" those filenames, and that it does not prevent overlong encodings to be put into those filenames. The problem I have with some of this wording is that it implies you must "decode" in order to avoid errors. In fact the UTF-8 spec is written the way it is so that you don't have to decode. Spitzak (talk) 00:42, 9 April 2026 (UTC)
- This hypothetical implementation of a path traversal checker isn't in the article, so the specifics of that aren't at issue. (I would argue that "not decoding" UTF-8 and interpreting it as an ASCII-like single-byte encoding is still a form of decoding, and non-conformant per later versions of the spec.) You seem to believe that we're trying to put a claim like
All programs must decode UTF-8 to UTF-32 before any further processing
into the article, but that's not the case. The article just needs to reflect the reality that overlong encodings are an inherent part of the encoding scheme, the spec is clear about what to do with them, and when programmers don't follow that exactly then security problems can arise. YoshiRulz (talk) 02:03, 9 April 2026 (UTC)- My proposal is fully conformant with the spec. It does not decode invalid UTF-8 sequences, since such sequences cannot contain the '/' character which is the only character it "decodes". Spitzak (talk) 03:56, 9 April 2026 (UTC)
- This hypothetical implementation of a path traversal checker isn't in the article, so the specifics of that aren't at issue. (I would argue that "not decoding" UTF-8 and interpreting it as an ASCII-like single-byte encoding is still a form of decoding, and non-conformant per later versions of the spec.) You seem to believe that we're trying to put a claim like
- Correct: Overlong encodings are not a security problem. But decoding overlong encodings is a security problem. The best way to avoid this security problem is to NOT DECODE. For instance if the file system only looked for the byte that is an ASCII '/' and all other bytes are used unchanged as the "file names", it would work fine. This is despite the fact that it never "decodes" those filenames, and that it does not prevent overlong encodings to be put into those filenames. The problem I have with some of this wording is that it implies you must "decode" in order to avoid errors. In fact the UTF-8 spec is written the way it is so that you don't have to decode. Spitzak (talk) 00:42, 9 April 2026 (UTC)
- What is the problem with the current wording at UTF-8#Overlong encodings? It says that overlong encodings are a security problem and should be considered an error and never decoded. The OP seems to suggest that checking for the three bytes
- At this moment neither of those were in any draft. The first thing that came up was the bugs being present and how they exist. The whole details of who does the reporting on overlong encodings doesn't really matter as long as something does, and the paragraph was to mention that this is not always a given in practice. ThanForIllegitimateAsThisDean (talk) 20:46, 25 March 2026 (UTC)
Content Disclaimer
Informasi ini disarikan dari Wikipedia dan disajikan kembali untuk tujuan edukasi. Konten tersedia di bawah lisensi CC BY-SA 3.0. Kami tidak bertanggung jawab atas ketidakakuratan data yang bersumber dari kontribusi publik tersebut.
- The information displayed on this website is sourced in part or in whole from Wikipedia and has been adapted for the purpose of restating it. We strive to provide accurate and relevant information, however:
- There is no guarantee of absolute accuracy. Wikipedia is an open, collaborative project that can be edited by anyone, so information is subject to change.
- It is not intended to constitute professional advice. The content displayed is for informational and educational purposes only. For important decisions (e.g., medical, legal, or financial), please consult a professional.
- Content copyright. Wikipedia is licensed under the Creative Commons Attribution-ShareAlike License (CC BY-SA). This means that content may be reused with appropriate attribution and shared under a similar license.
- Responsible use. Any risk arising from the use of information from this website is entirely the responsibility of the user.