Hex to text conversion gone wrong: the encoding mistakes that turn bytes into garbage

Converting hex to text feels like it should be foolproof: each pair of hex digits is one byte, map the byte to a character, done. In practice it fails constantly, and almost never because the converter is broken — it fails because hex is just a way of writing bytes, and bytes need an agreed-upon encoding before they mean any specific character at all. Here’s what’s actually going wrong when the output is garbled.

Cause 1: wrong character encoding assumed

The single most common cause. The hex bytes were encoded as UTF-8 (where many real-world characters take 2–4 bytes), but the converter is decoding byte-by-byte as if it were Latin-1 or plain ASCII (where every byte is exactly one character). The result: accented letters, curly quotes, em dashes, or any non-English character turn into two or three garbled symbols instead of one correct one.

The fix is always to decode with the same encoding the bytes were originally written in — if you don’t know which one, UTF-8 is the correct default for anything modern (web content, JSON, most APIs); Latin-1/Windows-1252 mostly only shows up in older Windows-originated text files or legacy email headers.

Cause 2: byte order (endianness) mismatches

This one specifically affects multi-byte numeric values (not text), but it’s common enough in hex-to-text tools that handle mixed binary/text data to be worth naming. 00 01 read big-endian is 1; the same two bytes read little-endian is 256. If a hex string represents a numeric length prefix or a binary field embedded in otherwise-textual data, reading it with the wrong byte order doesn’t just misread the number — it throws off every byte position after it, because the tool now thinks the text starts at the wrong offset.

Cause 3: nibble grouping errors

Hex should always come in pairs — two hex digits make one byte. An odd number of hex characters (one stray digit, a copy-paste that dropped a character, a leading 0x that wasn’t stripped before parsing) shifts every subsequent pair by one nibble. Every byte after the error decodes to a completely different, unrelated character — the output isn’t “slightly wrong,” it’s fully scrambled from that point forward, which is actually a useful diagnostic: garbage that starts clean and degrades partway through the string points at a grouping error at that exact position, not a wrong-encoding problem (which corrupts more uniformly throughout).

Cause 4: invisible and non-printable bytes

Not every byte maps to a visible character. Control characters (0x00–0x1F), the byte-order-mark (EF BB BF in UTF-8), and various Unicode formatting characters decode “successfully” but render as nothing, a box, or a question mark depending on the display font — which can look identical to a decoding failure even though the conversion itself was completely correct. If output length looks right but text appears to have missing characters, check for control bytes before assuming the decode logic is wrong.

A quick way to isolate which one you’re looking at

1. Confirm the hex string has an even number of characters — if not, that’s cause 3, fix the input first.
2. Try decoding as UTF-8 first, then Latin-1, and compare — if one produces clean text and the other doesn’t, it was cause 1.
3. If the output is garbled uniformly from the very first character, suspect encoding (cause 1); if it starts clean and degrades partway through, suspect a grouping error (cause 3) at that position.
4. If length matches expectations but specific characters are missing or show as boxes, check for control/formatting bytes (cause 4) rather than re-checking the encoding.

Hex-to-text conversion itself is one line of code in any language — the actual debugging work is almost always in figuring out which of these four assumptions was silently wrong, not in the conversion logic.