Character Encoding & Unicode Explained: From ASCII to UTF-8, Mojibake, and Beyond

You’ve seen it a hundred times: a CSV that opens as 涓枃 in Excel, a web page showing é where é should be, logs full of question marks where Chinese text used to live. Every one of these mysteries has the same root cause — character encoding. It’s knowledge every programmer uses daily but few ever learned systematically. This guide walks the whole arc from ASCII to Unicode, UTF-8, UTF-16, BOM, and normalization, so the next time text breaks you’ll know exactly where to look.

ASCII and the Code Page Era

ASCII, defined in the 1960s, encodes 128 characters in 7 bits: English letters, digits, punctuation, and control characters. A is 0x41, a is 0x61, space is 0x20. For an English-speaking world, that was plenty.

The world, however, doesn’t only speak English. A byte has 8 bits, leaving 128 spare slots above ASCII, and every region filled them with its own characters: Western Europe used Latin-1 (ISO-8859-1) for é and ü, Russia used KOI8-R for Cyrillic, and mainland China used GB2312 and its superset GBK, which needs two bytes per character because one byte can’t hold thousands of hanzi. These schemes were collectively called code pages, and they had a fatal flaw: the byte 0xE4 means ä in Latin-1 but is half of a hanzi in GBK. Bytes carry no label saying which encoding produced them — the moment the decoder guesses the wrong code page, you get garbage. Unicode exists to end that chaos.

Unicode: One Number per Character

Unicode’s idea is beautifully simple: forget bytes for a moment and give every character in every writing system a single unique number, called a code point, written as U+ followed by hex. A is U+0041, the hanzi 中 is U+4E2D, and 😀 is U+1F600. The code space runs from U+0000 to U+10FFFF, covering nearly every script, symbol, and emoji in use.

Here’s the mental model that makes everything else click: a code point is an abstract number; bytes are the storage format; the mapping between them is the encoding. Saying a file is “in Unicode” is actually meaningless — files always contain bytes, and what determines those bytes is an encoding form like UTF-8 or UTF-16. When you need to look up a character’s code point, the Unicode Character Finder shows its number, official name, and UTF-8 byte sequence in one place.

UTF-8: Variable-Length Done Right

UTF-8 is the dominant encoding on the modern web, and its design is elegant: each code point takes 1 to 4 bytes, with smaller code points using fewer bytes.

U+0000 – U+007F:    0xxxxxxx                              (1 byte)
U+0080 – U+07FF:    110xxxxx 10xxxxxx                     (2 bytes)
U+0800 – U+FFFF:    1110xxxx 10xxxxxx 10xxxxxx            (3 bytes)
U+10000 – U+10FFFF: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx   (4 bytes)

Take 中 (U+4E2D), which falls in the three-byte range. Expanding 4E2D to binary gives 0100 111000 101101; slotting those bits into the template yields 11100100 10111000 10101101 — the byte sequence E4 B8 AD. Every code point from U+0800 to U+FFFF (including all common hanzi) costs 3 bytes, and emoji, sitting above U+FFFF, always cost 4.

UTF-8’s killer feature is full ASCII compatibility: every ASCII character encodes to exactly its original byte. A pure-English UTF-8 file is byte-for-byte identical to an ASCII file, so legacy tools read the English parts fine. Combined with self-synchronization (you can find the next character boundary from any byte), it’s no surprise UTF-8 won. To see the character-to-byte relationship hands-on, feed some text into the Text ↔ Binary Converter and inspect its representation in hex, decimal, or binary.

UTF-16, UCS-2, and Surrogate Pairs

UTF-16 takes a different trade: characters in the Basic Multilingual Plane (BMP, U+0000–U+FFFF) use 2 bytes, and everything above the BMP uses 4. Java and JavaScript strings, and Windows internally, are all UTF-16.

Characters beyond the BMP are encoded as surrogate pairs. Unicode reserves U+D800–U+DFFF as surrogate code points that never represent real characters. An astral code point splits into a high surrogate (U+D800–U+DBFF) followed by a low surrogate (U+DC00–U+DFFF). For example, 😀 (U+1F600) becomes the pair D83D DE00 — two 16-bit code units.

This causes a famous gotcha: JavaScript’s "😀".length returns 2, because length counts UTF-16 code units, not characters. The older UCS-2 encoding didn’t even support characters outside the BMP at all, which is why some legacy systems simply can’t display certain emoji. When iterating strings in modern JS, for...of and the spread operator walk code points rather than code units — the correct default.

Where Mojibake Comes From

Mojibake — the technical term for garbled text — has essentially two causes.

Cause one: decoding with the wrong charset. The bytes E4 B8 AD decode to 中 as UTF-8, but if a reader interprets them as GBK, E4 B8 becomes one hanzi and AD glues onto the next byte — which is exactly how 中文 mutates into the infamous 涓枃. In the other direction, GBK bytes fed to a UTF-8 decoder hit invalid sequences and get replaced with � (U+FFFD, the replacement character). The classic é on web pages is the same accident: é in UTF-8 is C3 A9, and reading those two bytes as Latin-1 produces two separate characters.

Cause two: double encoding. Text is correctly encoded as UTF-8, then those bytes are mistaken for a Latin-1 string and encoded as UTF-8 again. é first becomes é, then four bytes, and the corruption compounds with every round trip. The fix is to reverse the process: decode as Latin-1 back to bytes, then decode as UTF-8 — sometimes twice.

A handy trick when diagnosing mojibake: paste the garbled characters into the Unicode Character Finder and look at their code points — the pattern usually reveals the original encoding chain. And if the text is littered with HTML entities like é or 中, that’s a separate escaping layer; decode it first with the HTML Encoder/Decoder before judging the underlying text.

The BOM and Why UTF-8 BOM Bites

A UTF-16 file starting with the bytes 4E 2D is ambiguous: is that 中 in big-endian order, or U+2D4E in little-endian? The BOM (Byte Order Mark) solves this: encode the code point U+FEFF at the start of the file. UTF-16 big-endian writes FE FF, little-endian writes FF FE, and the first two bytes tell you the order.

UTF-8 has no byte-order problem, yet a UTF-8 BOM — the three bytes EF BB BF — was invented anyway as a “this is UTF-8” signature, largely popularized by Windows Notepad. The trouble is that many tools and protocols don’t expect three extra bytes at the start:

• A PHP file with a BOM produces output before header() runs, triggering “headers already sent”;
• A BOM before a shell script’s #! shebang means the kernel can’t find the interpreter;
• CSV parsers may treat the BOM as part of the first field’s content;
• Concatenating files pushes BOMs into the middle of the text, where they lurk as invisible zero-width characters.

Rule of thumb: write UTF-8 without a BOM by default, unless you know the consumer — certain Excel versions opening UTF-8 CSVs are the usual excuse — genuinely needs it.

Normalization: Is é One Character or Two?

Unicode allows many accented characters to be written two equivalent ways. é can be the single code point U+00E9 (precomposed), or the letter e (U+0065) followed by a combining acute accent U+0301. Both render identically, yet they compare unequal byte-for-byte and code-point-for-code-point. Filesystems, unique database indexes, and password comparisons have all been bitten: two strings that look identical to a human are different to the program.

Unicode defines normalization forms to fix this:

• NFC (Canonical Composition): merge everything mergeable — e + U+0301 becomes U+00E9;
• NFD (Canonical Decomposition): split everything splittable — U+00E9 becomes e + U+0301;
• NFKC/NFKD additionally apply compatibility mappings, turning the fi ligature into fi and full-width characters into half-width ones. These lose information, so use them for matching, not storage.

Practical advice: normalize user input to NFC before storing it, and make sure both sides of any comparison, dedup, or hash use the same form. One real-world trap: macOS’s filesystem leans NFD while Windows and most of the world use NFC, so filenames can subtly change when moving across platforms.

A Practical Debugging Checklist

When encoding trouble strikes, this sequence almost never fails:

1. Check the file’s actual encoding first. On Linux/macOS, file input.txt gives a quick guess; for certainty, hexdump -C file | head shows the opening bytes — EF BB BF means UTF-8 BOM, FF FE means UTF-16 little-endian.
2. For garbled web pages, check the declarations. The HTTP response header Content-Type: text/html; charset=utf-8 outranks the <meta charset="utf-8"> tag in the HTML. A mismatch between them is the classic reason a page looks fine locally but breaks in production.
3. Count bytes to guess the encoding. In UTF-8, hanzi take 3 bytes, ASCII 1, emoji 4; in GBK, hanzi take 2. Long runs of three-byte sequences starting with E4–E9 are a strong tell for UTF-8 Chinese.
4. Convert explicitly instead of guessing. Once the source encoding is known, run iconv -f GBK -t UTF-8 input.txt rather than trusting an editor’s auto-detection.
5. Decode layer by layer. Unwrap HTML entities with the HTML Encoder/Decoder, confirm each character’s true code point with the Unicode Character Finder, and verify the raw bytes with the Text ↔ Binary Converter — inspect all three layers and no encoding bug can hide.

Encoding bugs are insidious precisely because nothing ever throws an error — you just get quietly wrong text. But once you internalize that characters are code points, files are bytes, and encodings are the maps between them — plus the two footnotes of BOM and normalization — nearly every mojibake you’ll ever meet becomes a ten-minute diagnosis instead of an afternoon of despair.