82. Unicode

The lesson

Unicode is the universal character standard that assigns a unique number, called a code point, to essentially every character and symbol in use today. A code point is written as U+ followed by four or more hexadecimal digits, for example U+0041 is the Latin capital letter 'A', and U+4E2D is the Chinese character '中' (zhōng). The standard currently defines more than 149,000 characters covering most of the world's writing systems.

The first 128 Unicode code points (U+0000 to U+007F) are exactly the ASCII table. This deliberate design means ASCII text is already valid Unicode text, and it makes migration from the old 7-bit standard trivial. Unicode adds thousands of letters, marks, punctuation, symbols, and emoji beyond that first block.

Unicode itself only assigns numbers; it does not say how those numbers are stored as bytes. The storage scheme is called an encoding. The three common encodings are UTF-8, UTF-16, and UTF-32. UTF-32 uses a fixed 4 bytes for every code point, which is simple but wasteful. UTF-16 uses 2 or 4 bytes and is used internally by Windows and Java. UTF-8 is variable-width and is the dominant encoding on the web and in modern tools.

UTF-8 encodes ASCII characters in a single byte (identical to ASCII), most Latin and Greek letters and Cyrillic in 2 bytes, CJK and most other scripts in 3 bytes, and emoji and rarer characters in 4 bytes. Because it is compact for English and self-synchronising, UTF-8 is the default for JSON, HTML, source code, and most APIs.

This explains a common surprise: an emoji like '😀' (U+1F600) is one character to you but four bytes in UTF-8, while the Chinese character '中' is one character but three bytes. TPEE counts Unicode characters, not bytes, for its 12 prompt sections, so an emoji counts as 1 character even though it uses 4 bytes.

Normalization matters too. Some characters can be written in more than one way — for example 'é' as a single precomposed character U+00E9, or as 'e' (U+0065) followed by a combining acute accent U+0301. Unicode normalization (NFC, NFD, NFKC, NFKD) converts these to a canonical form so that searches and comparisons match correctly.

Why this matters for TPEE: prompts you write may contain accented text, CJK, Arabic, emoji, and other Unicode content. TPEE's font handling and the JSON export must treat these as correct Unicode characters. When you see a garbled 'é' where an 'é' should be, that is usually a UTF-8 vs Latin-1 mismatch, not a data problem. Understanding Unicode helps you write prompts that display and round-trip correctly across languages.

A ward nurse builds a TPEE prompt that keeps accented patient names intact, checking that the intake template displays each character correctly before the form is printed.

A school teacher prepares a bilingual worksheet prompt in TPEE, confirming that Chinese and Arabic characters survive the export so students see the correct text on the handout.

A catalogue librarian writes a TPEE prompt for non-Latin book titles, comparing character counts with byte counts so the records sort correctly in the catalogue file.

A sub-editor at a news outlet checks emoji and accent handling in a TPEE headline prompt before the piece is exported to the publishing system.

A junior bank clerk uses TPEE to draft a customer-address prompt, verifying that international name fields round-trip correctly when the account record is saved.

A hotel front-desk supervisor writes greeting prompts in TPEE that carry guest names in several scripts, checking each one displays correctly before arrival cards are printed.

A gaming community manager builds a TPEE moderator prompt that recognises emoji and non-Latin player handles, noting that one emoji counts as one character but four bytes.

A municipal records officer drafts public notices in TPEE covering several community languages, testing that every character stores and retrieves without corruption.

A logistics documentation lead prepares a TPEE prompt for destination names written in different scripts, checking the export encoding before the shipping labels are generated.

A community language officer at a health service writes translated leaflet prompts in TPEE, checking that accented and non-Latin words survive the copy into the layout tool.

Check yourself