How LLMs See Text: Tokenization

Overview

This lesson introduces tokenization as the step where language models turn written text into token IDs. It includes a short concrete example using the tokenizer associated with gpt-oss-20b, without loading the model weights.

Outcomes

After completing this lesson, you should be able to …

• explain why an LLM does not read text as words in the everyday sense
• describe the basic compression idea behind byte pair encoding
• identify why tokenization affects spelling, multilingual text, cost, and model behavior

Prerequisites

Before starting this tutorial, ensure that …

• you are familiar with tokens and types
• you are familiar with text normalization

Background

When we type a prompt into a large language model, it may feel as though the model receives ordinary words and sentences. Under the hood, most modern language models first convert text into a sequence of tokens. Those tokens may correspond to complete words, common word pieces, punctuation, spaces, bytes, or fragments that do not feel linguistically natural.

This matters because the model does not directly predict the next word. It predicts the next token ID from a fixed vocabulary learned before training.¹

The problem

A tokenizer has to balance several competing goals:

• represent frequent words and patterns compactly
• avoid an enormous vocabulary containing every possible word
• handle rare words, names, numbers, typos, code, emoji, and multilingual text
• keep the representation reversible enough that generated token IDs can be decoded back into text

If we tokenize only by whitespace, rare and unseen words become a problem. If we tokenize only by characters or bytes, every input becomes very long. Subword tokenization is a compromise: common strings can become single tokens, while rare strings can be broken into smaller pieces.

Byte pair encoding

Byte pair encoding began as a compression algorithm. Let's set aside the idea of bytes for a moment and focus on the core pattern: start with a small set of basic units, count adjacent pairs, and repeatedly merge frequent neighbors into larger pieces.

1. Start with a sequence of small symbols.
2. Count adjacent pairs.
3. Merge the most frequent pair into a new symbol.
4. Repeat until the vocabulary reaches the desired size.

For text tokenization, this means frequent character or byte sequences become vocabulary items. A word like tokenization might be represented as one token in one tokenizer, but as pieces such as token, ization, or smaller fragments in another.

Imagine we have a small corpus where each document is a sentence:

documents = [
    "low lower lowest",
    "newer wider lower",
    "low low newer",
]

# A toy starting point: split documents into words.
words = [word for document in documents for word in document.split()]

# Then split each word into small symbols.
vocab = [list(word) for word in words]

for word in vocab:
    print(word)

The real training process does not choose merges by looking at one word in isolation. It counts adjacent symbol pairs across the training corpus, merges the most frequent pair wherever it appears, then counts again. After many rounds, the tokenizer has learned a vocabulary of reusable pieces.

Why this affects LLM behavior

Tokenization can make some strings easier or harder for a model to handle.

• Common English words may be represented compactly.
• Rare names may split into several pieces.
• Code and file paths may tokenize very differently from prose.
• Text in some languages may require more tokens for the same amount of visible text.
• Extra whitespace or unusual punctuation may change the token sequence.

That last point is one reason prompt formatting can matter. Even if two prompts look similar to a reader, they may not produce the same token sequence.

A first experiment

Try comparing how a tokenizer segments the following strings:

• running
• runner
• unhappiness
• Gus Hahn-Powell
• parsertongue.org
• こんにちは

As you inspect the pieces, ask:

• Which tokens look like words?
• Which tokens look like word parts?
• Which tokens preserve spaces or punctuation?
• Which examples use more tokens than you expected?

A real model tokenizer

The word token does not always mean "word." Some tokenizers are designed to find word-like units. Others are designed to turn text into reusable pieces for a model vocabulary.

Here are two short sentences:

The wizard split the sentence with a spell.
El mago dividió la oración con un hechizo.

If we tokenize them as words, the result looks familiar:

english_words = [
    "The",
    "wizard",
    "split",
    "the",
    "sentence",
    "with",
    "a",
    "spell",
]

spanish_words = [
    "El",
    "mago",
    "dividió",
    "la",
    "oración",
    "con",
    "un",
    "hechizo",
]

If we preserve punctuation as its own token, the period becomes visible:

english_tokens = [
    "The",
    "wizard",
    "split",
    "the",
    "sentence",
    "with",
    "a",
    "spell",
    ".",
]

spanish_tokens = [
    "El",
    "mago",
    "dividió",
    "la",
    "oración",
    "con",
    "un",
    "hechizo",
    ".",
]

A model tokenizer can look different. The tokenizer associated with gpt-oss-20b keeps the English sentence mostly word-like here, but splits parts of the Spanish sentence.

For this example, use the following files:

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

path = hf_hub_download(
    repo_id="openai/gpt-oss-20b",
    filename="tokenizer.json",
)

tokenizer = Tokenizer.from_file(path)

sentences = [
    "The wizard split the sentence with a spell.",
    "El mago dividió la oración con un hechizo.",
]

for sentence in sentences:
    encoding = tokenizer.encode(sentence)
    print(sentence)
    print("Token IDs:")
    print(encoding.ids)
    print("Decoded token strings:")
    print([tokenizer.decode([token_id]) for token_id in encoding.ids])

The wizard split the sentence with a spell.
Token IDs:
[976, 67537, 12648, 290, 21872, 483, 261, 29176, 13]
Decoded token strings:
['The', ' wizard', ' split', ' the', ' sentence', ' with', ' a', ' spell', '.']

El mago dividió la oración con un hechizo.
Token IDs:
[4422, 3284, 78, 29269, 1153, 557, 182414, 406, 537, 139938, 14455, 13]
Decoded token strings:
['El', ' mag', 'o', ' divid', 'ió', ' la', ' oración', ' con', ' un', ' hech', 'izo', '.']

huggingface_hub
tokenizers

python -m pip install -r requirements.txt

The leading spaces are part of the decoded token strings. They show that this tokenizer is preserving information about where pieces occur relative to whitespace.

Different tokenization schemes preserve different information. The important question is not just "what are the tokens?" but "what does this tokenizer make easy or hard for the task I care about?"

Discussion

BPE is not a theory of morphology, writing systems, or human reading. It is an engineering compromise that helps neural language models work with open-ended text using a fixed vocabulary. That compromise is powerful, but it also shapes what the model sees and what it may find easy or difficult.

In later work, this connects to multilingual NLP, low-resource language technology, prompt design, cost estimation, and the design of model-specific evaluation tasks.

Next steps

Take a moment to reflect on how a tokenization algorithm might affect model behavior.

Practice

• How might corpus size affect your choice of vocabulary size?
• Why might a tokenizer avoid merging symbol pairs that appear only once or twice in the training corpus?
• What changes if your starting units are bytes, characters, phones, morphemes, or words?
• When would you prefer a tokenizer that produces many small pieces over one that produces fewer larger pieces?
• Adapt the code snippet above to use the tokenizer for allenai/OLMo-7B-0424-hf, an English language model. You will likely want transformers and AutoTokenizer.from_pretrained("allenai/OLMo-7B-0424-hf"). Try tokenizing a non-English sentence in a language you are familiar with. What do you notice? What problems might this present?