Follow the learned merges
Explore how an English-focused OLMo tokenizer and a multilingual Qwen tokenizer split the same text. Their merge rules are already learned. Nothing is trained here.
Segment input
William Shakespeare, Sonnet 18 · Source at Project Gutenberg
First comparison loads about 6 MB of tokenizer data. This runs completely locally in your browser.
Select a highlight for its token IDs and bytes. Sometimes the bytes for one character end up in different tokens. We keep the character readable and mark it with stripes. Select it to see those tokens. Text is shown after NFC normalization.
Inspect the raw pieces and merge steps
Merges stay within this pretokenization segment. Try Escárcega, Shakespeare, or a short string with a leading space.
Raw labels such as Ġ stand for bytes. They aren't necessarily characters in the source text. ␠ = space, ↵ = newline, ⇥ = tab. Don't type these display marks literally.
Sources and what to compare
OLMo-7B-0424-hf is English-focused. Qwen3-VL-4B-Instruct is multilingual. They differ in vocabulary size, pretokenization, and training. Token counts alone don't show language proficiency or isolate a training-data effect. Enter one pretokenization segment to inspect its byte pieces and the order of the saved merges. If either tokenizer splits your input into several segments, try a shorter string. BPE does not merge across those boundaries.
Both tokenizers use NFC normalization, which preserves canonical equivalence and can change the code-point sequence. It doesn't simply remove accents. UTF-8 turns code points into bytes. The byte count depends on the code points used. Use Previous, Next merge, or the slider to inspect the sequence. The highlighted pair has the next eligible learned rank. Select a raw piece to see its ID and bytes. Compare the same segment in both model tabs.
Pinned OLMo data · Pinned Qwen data · Attribution · Apache 2.0.