How LLMs See Text: Tokenization

Overview

This lesson introduces tokenization as the step where language models turn written text into token IDs. It includes a short concrete example using the tokenizer associated with gpt-oss-20b, without loading the model weights.

Outcomes

After completing this lesson, you should be able to …

• explain why an LLM does not read text as words in the everyday sense
• describe the basic compression idea behind byte pair encoding
• identify why tokenization affects spelling, multilingual text, cost, and model behavior

Prerequisites

Before starting this tutorial, ensure that …

• you are familiar with tokens and types
• you are familiar with text normalization

Background

When we type a prompt into a large language model, it may feel as though the model receives ordinary words and sentences. Under the hood, most modern language models first convert text into a sequence of tokens. Those tokens may correspond to complete words, common word pieces, punctuation, spaces, bytes, or fragments that do not feel linguistically natural.

This matters because the model does not directly predict the next word. It predicts the next token ID from a fixed vocabulary learned before training.¹

The problem

A tokenizer has to balance several competing goals:

• represent frequent words and patterns compactly
• avoid an enormous vocabulary containing every possible word
• handle rare words, names, numbers, typos, code, emoji, and multilingual text
• keep the representation reversible enough that generated token IDs can be decoded back into text

For writing conventions that use spaces between words, whitespace splitting can be a useful first approximation. Giving each resulting string its own vocabulary entry still leaves us with rare or unseen forms. For other writing conventions, spaces don't provide that starting segmentation at all.

Character- or byte-level encoding can represent unfamiliar strings with a small inventory, but often requires longer sequences than a learned vocabulary of larger pieces. The difference depends on the text, script, encoding, and vocabulary. Subword methods offer a tradeoff: frequent strings can become compact pieces, while other strings can be assembled from smaller units. Those pieces don't necessarily coincide with linguistic words or morphemes.

Byte pair encoding

Byte pair encoding began as a compression algorithm. Let's set aside the idea of bytes for a moment and focus on the core pattern: start with a small set of basic units, count adjacent pairs, and repeatedly merge frequent neighbors into larger pieces.

1. Start with a sequence of small symbols.
2. Count adjacent pairs.
3. Merge the most frequent pair into a new symbol.
4. Repeat until the vocabulary reaches the desired size.

The tokenizer saves the learned pair merges in order. This ordered list is its merge table. Each pair has a rank that determines its priority when encoding new text. Applying the tokenizer uses these saved rules. It doesn't learn a new table by counting pairs in each new sentence.

For text tokenization, this means frequent character or byte sequences become vocabulary items. A word like tokenization might be represented as one token in one tokenizer, but as pieces such as token, ization, or smaller fragments in another.

Imagine we have a small corpus where each document is a sentence:

documents = [
    "low lower lowest",
    "newer wider lower",
    "low low newer",
]

# A toy starting point: split documents into words.
words = [word for document in documents for word in document.split()]

# Then split each word into small symbols.
vocab = [list(word) for word in words]

for word in vocab:
    print(word)

The real training process does not choose merges by looking at one word in isolation. It counts adjacent symbol pairs across the training corpus, merges the most frequent pair wherever it appears, then counts again. After many rounds, the tokenizer has learned a vocabulary of reusable pieces.

Why this affects LLM behavior

Tokenization can make some strings easier or harder for a model to handle.

• Common English words may be represented compactly.
• Rare names may split into several pieces.
• Code and file paths may tokenize very differently from prose.
• Text in some languages may require more tokens for the same amount of visible text.
• Extra whitespace or unusual punctuation may change the token sequence.

A tokenizer's vocabulary also reflects its training data and design choices. Equivalent messages in different languages can require different numbers of tokens, affecting usable context and any per-token cost. This isn't explained by spaces alone: character and byte encodings can differ across scripts too. Petrov et al. (2023) examine these disparities. Compare actual tokenizations rather than treating a word count as a universal measure of how much information fits.

This has implications for prompt formatting. Even if two prompts look similar to a reader, they may not produce the same token sequence.

A first experiment

Try comparing how a tokenizer segments the following strings:

• running
• runner
• unhappiness
• Gus Hahn-Powell
• parsertongue.org
• こんにちは

As you inspect the pieces, ask:

• Which tokens look like words?
• Which tokens look like word parts?
• Which tokens preserve spaces or punctuation?
• Which examples use more tokens than you expected?

An unfamiliar name isn't necessarily one token

Try Shakespeare, Frost, and Darío in the widget below. Inspect raw pieces and IDs rather than guessing from spelling or word boundaries. Change capitalization or add a leading space and compare again. Encoding a name doesn't tell us what the language model has learned about its referent.

Explore learned merge rules

The widget compares the English-focused OLMo-7B-0424-hf tokenizer with the multilingual Qwen3-VL-4B-Instruct tokenizer. Both use byte-level BPE, so we can inspect actual learned merge ranks rather than inventing a toy merge list. Choose Shakespeare’s “Sonnet 18,” Frost’s “Fire and Ice,” Darío’s “Lo fatal,” a short diagnostic example, or your own text. Then compare the highlighted text. Switch model tabs to inspect how the same text is split.

Open the tokenizer widget on its own.

The tables have already been fit. The next operation uses the highest-priority eligible adjacent pair in that tokenizer's learned order. It doesn't count frequencies or learn new merges from your input. The two tokenizers also differ in vocabulary size and preprocessing. We should treat this as an inspection of two complete pipelines, rather than a controlled test of multilingual versus monolingual training. Multilingual tokenizers can differ in how compactly they represent different languages. Try texts in languages you know and compare the pieces produced by each tokenizer.

Reading space markers

A tokenizer viewer may show symbols that weren't in the text you typed. For example, the Qwen tokenizer's raw-piece display represents This is a test. as:

This | Ġis | Ġa | Ġtest | .

The vertical bars above separate tokens. Ġis represents a space followed by is, not a word beginning with the letter Ġ. Likewise, Ġa represents " a", and Ġtest represents " test". Each piece is one token in this example: the space and letters share a token ID. The marker doesn't add another token.

Why Ġ? GPT-2-style byte-level tokenizers use a reversible mapping from bytes to visible characters. In that mapping, the ordinary space byte maps to Ġ. This is a reversible tokenizer convention, not a meaning assigned by Unicode: Ġ is the existing letter U+0120, LATIN CAPITAL LETTER G WITH DOT ABOVE.² Another mapping could choose a different label. Here, the particular label follows from the mapping algorithm. It isn't a linguistic claim about spaces. The raw vocabulary piece exposes that representation, and decoding turns it back into a space.³

Other tokenizers use different conventions. In a Gemma tokenizer's raw-piece display, ▁is also represents a space followed by is. Here ▁ is a low horizontal bar, not the ordinary underscore _. SentencePiece uses this symbol to make whitespace visible.⁴ Some tokenizers also insert an initial space marker during preprocessing, so a marker at the start need not mean you typed a leading space.

Try comparing is (no leading space) with ␠is (one leading space) in the widget. Here ␠ makes an ordinary space visible. Type a space, not that display symbol. Do the pieces or IDs change? Compare within each model: different tokenizers have their own vocabularies, so matching token IDs across models need not mean matching text.

Follow the learned merges

The second widget focuses on the saved merge rules. Enter a short string that forms one pretokenization segment, a chunk identified before BPE. These chunks aren't necessarily linguistic words. Try Escárcega, then switch model tabs and step through the eligible adjacent pairs in learned rank order. The tables have already been fit. Here we apply their rules to your segment.

Open the merge-step widget on its own. Select a raw piece to inspect its token ID and bytes. Use the same input in both models when comparing their rules and boundaries.

A real model tokenizer

The word token does not always mean "word." Some tokenizers are designed to find word-like units. Others are designed to turn text into reusable pieces for a model vocabulary.

Here are two short sentences:

The wizard split the sentence with a spell.
El mago dividió la oración con un hechizo.

If we tokenize them as words, the result looks familiar:

english_words = [
    "The",
    "wizard",
    "split",
    "the",
    "sentence",
    "with",
    "a",
    "spell",
]

spanish_words = [
    "El",
    "mago",
    "dividió",
    "la",
    "oración",
    "con",
    "un",
    "hechizo",
]

If we preserve punctuation as its own token, the period becomes visible:

english_tokens = [
    "The",
    "wizard",
    "split",
    "the",
    "sentence",
    "with",
    "a",
    "spell",
    ".",
]

spanish_tokens = [
    "El",
    "mago",
    "dividió",
    "la",
    "oración",
    "con",
    "un",
    "hechizo",
    ".",
]

A model tokenizer can look different. The tokenizer associated with gpt-oss-20b keeps the English sentence mostly word-like here, but splits parts of the Spanish sentence.

For this example, use the following files:

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

path = hf_hub_download(
    repo_id="openai/gpt-oss-20b",
    filename="tokenizer.json",
)

tokenizer = Tokenizer.from_file(path)

sentences = [
    "The wizard split the sentence with a spell.",
    "El mago dividió la oración con un hechizo.",
]

for sentence in sentences:
    encoding = tokenizer.encode(sentence)
    print(sentence)
    print("Token IDs:")
    print(encoding.ids)
    print("Decoded token strings:")
    print([tokenizer.decode([token_id]) for token_id in encoding.ids])

The wizard split the sentence with a spell.
Token IDs:
[976, 67537, 12648, 290, 21872, 483, 261, 29176, 13]
Decoded token strings:
['The', ' wizard', ' split', ' the', ' sentence', ' with', ' a', ' spell', '.']

El mago dividió la oración con un hechizo.
Token IDs:
[4422, 3284, 78, 29269, 1153, 557, 182414, 406, 537, 139938, 14455, 13]
Decoded token strings:
['El', ' mag', 'o', ' divid', 'ió', ' la', ' oración', ' con', ' un', ' hech', 'izo', '.']

huggingface_hub
tokenizers

python -m pip install -r requirements.txt

The leading spaces are part of the decoded token strings. They show that this tokenizer is preserving information about where pieces occur relative to whitespace.

Different tokenization schemes preserve different information. The important question is not just "what are the tokens?" but "what does this tokenizer make easy or hard for the task I care about?"

Discussion

BPE is not a theory of morphology, writing systems, or human reading. It is an engineering compromise that helps neural language models work with open-ended text using a fixed vocabulary. That compromise is powerful, but it also shapes what the model sees and what it may find easy or difficult.

In later work, this connects to multilingual NLP, low-resource language technology, prompt design, cost estimation, and the design of model-specific evaluation tasks.

Next steps

Take a moment to reflect on how a tokenization algorithm might affect model behavior. To follow token IDs into a model's predictions, continue to How models generate text.

Practice

• How might corpus size affect your choice of vocabulary size?
• Why might a tokenizer avoid merging symbol pairs that appear only once or twice in the training corpus?
• What changes if your starting units are bytes, characters, phones, morphemes, or words?
• When would you prefer a tokenizer that produces many small pieces over one that produces fewer larger pieces?
• Adapt the code snippet above to use the tokenizer for allenai/OLMo-7B-0424-hf, an English language model. You will likely want transformers and AutoTokenizer.from_pretrained("allenai/OLMo-7B-0424-hf"). Try tokenizing a non-English sentence in a language you are familiar with. What do you notice? What problems might this present?