You can put two identical-looking versions of the word café next to each other, stare at them until you begin to doubt your eyesight, and still have a computer insist they are different. I find this an impressive amount of disagreement to extract from four letters.
Here they are: café and café. In the first, the é comes as one character. In the second, it comes as an e followed by a separate combining acute accent. Unicode allows both arrangements to represent the same letter.[1] You get the assembled version or the version with a little assembly required.
Neither spelling is wrong. This matters. The person typing their name into a form has not committed a technical offense by possessing an accent, and they should not need to learn the internal representation of a vowel to get their account back.
But a plain Python string comparison checks the sequence of code points, not whether a reasonable person would read both words the same way. I tried those two versions. They looked alike; the comparison returned False.
There is a fix called normalization: put equivalent text into a common representation before comparing it.[1] For this experiment I used the form called NFC, which turns our separate e and accent into the single é. Python has a standard-library function for it.[2] No package installation, no account, no subscription to Accents Pro.
That ought to be the end of a very small story. Except I wanted to try something that sounds perfectly reasonable: check the pieces first, then join them.
They were normal when I checked
I gave Python the word cafe, without its accent, and asked whether it was already in NFC form. Yes.
Then I gave it just the combining accent and asked the same question. Also yes. On its own, the accent has no letter to combine with; there is nothing NFC needs to change.
Two pieces. Both checked. I joined them.
import unicodedata as u
word = "cafe"
accent = "\u0301"
joined = word + accent
print(u.is_normalized("NFC", word))
print(u.is_normalized("NFC", accent))
print(u.is_normalized("NFC", joined))
The result:
True
True
False
I like this result because you can understand the annoyance without reading Python. Both parts passed inspection. Nobody changed either part. Putting them next to each other was enough to make the completed thing fail the same inspection.
Once the accent follows the final e, there is a pair that NFC can combine. The join created that opportunity. It did not perform the combination for us.
The Unicode specification explicitly warns about this: joining normalized strings does not necessarily produce a normalized string.[1] So this isn't a Python bug or a new discovery. It is a small, well-documented reason to distrust the sentence “we already checked that.”
We checked the pieces. We hadn't checked what happened when they met.
Please leave the accent on
For this little word, the fix is to assemble it first and normalize afterward. I did that, and the comparison with the precomposed café finally returned True.
What I would not do is solve the problem by stripping off the accent. That would make the software's life easier by changing what somebody wrote. An accent is part of the text, not packaging you throw away once the letters arrive.
There is a subtler version of that temptation, too. Unicode offers compatibility normalization, which folds away additional distinctions.[1] I tried its NFKC form on m² and got m2. NFC left the superscript alone. Compatibility normalization has legitimate uses, but if I'm reading a measurement, I would prefer the tidy-up operation not flatten part of the notation.
That is what makes this tiny example worth keeping around. “Clean up the text” sounds like housekeeping, but you still have to decide what counts as the same text, what distinctions deserve to survive, and when the cleaning is actually finished.
I started with four letters and an accent. I ended up appreciating how much work it can take for a computer to agree with someone who is plainly right about the word café.