One text, two tokenizers
Explore how an English-focused OLMo tokenizer and a multilingual Qwen tokenizer split the same text. Their merge rules are already learned. Nothing is trained here.
Choose or edit the text
William Shakespeare, Sonnet 18 · Source at Project Gutenberg
First comparison loads about 6 MB of tokenizer data. This runs completely locally in your browser.
Select a highlight for its token IDs and bytes. Sometimes the bytes for one character end up in different tokens. We keep the character readable and mark it with stripes. Select it to see those tokens. Text is shown after NFC normalization.
Inspect the raw pieces and merge steps
Merges stay within this pretokenization segment. Try Escárcega, Shakespeare, or a short string with a leading space.
Raw labels such as Ġ stand for bytes. They aren't necessarily characters in the source text. ␠ = space, ↵ = newline, ⇥ = tab. Don't type these display marks literally.
Sources and what to compare
OLMo-7B-0424-hf is English-focused. Qwen3-VL-4B-Instruct is multilingual. They differ in vocabulary size, pretokenization, and training. Token counts alone don't show language proficiency or isolate a training-data effect. These poems are different works, not translations or a controlled language comparison.
Both tokenizers use NFC normalization, which preserves canonical equivalence and can change the code-point sequence. It doesn't simply remove accents. UTF-8 turns code points into bytes. The byte count depends on the code points used. For readable highlighting, we keep the browser's grapheme clusters together. Stripes identify clusters crossing token boundaries. The merge-step widget shows the exact raw pieces and IDs.
Pinned OLMo data · Pinned Qwen data · Attribution · Apache 2.0. Poem wording and punctuation come from the linked Gutenberg editions. Line endings are LF (line feed) and leading layout indentation is removed.