Comparative Tokenizer Study

Problem & motivation

A browser model has a limited context window, but a character count is not a token budget. The same text can occupy different amounts of space under different tokenizers. I investigated those differences while working on a local JEE model.

A tokenizer converts text into the pieces a model processes. The study asks how much the count changes when the input stays fixed, and whether the original text survives decoding.

Controls & architecture

1. Freeze inputs100 published samples; pinned source metadata
2. Encode & checkEight configurations; Python and JavaScript
3. Compare & verifySame-input ratios, paired JSON, independent arithmetic

The corpus has 20 samples each of English documentation, code, JSON, HTML, and mixed-script documentation. Forty compact/indented JSON variants make 140 inputs. These are not 140 independent source samples. Eight configurations produce 1,120 recorded encodings; DeepSeek V3 and R1 share core components.

Source hashes, tokenizer revisions, runtime versions, and configuration are recorded. The initial sampling plan was fixed before encoding; the later DeepSeek extension uses an already inspected corpus.

For the JSON pairs, values, key order, array order, and scalar spelling stay fixed. Only whitespace outside strings changes. The analysis uses the geometric mean of same-input ratios, rather than dividing pooled token totals.

Initial controls · DeepSeek extension protocol.

Token-count results

DeepSeek’s indented JSON uses 35.2% more tokens than compact JSON with the same values. V3 and R1 produce identical sequences on all 140 corpus inputs, but tokenize special reasoning markers differently. Matching content counts do not make chat templates interchangeable.

Language differences need context

In the earlier six-profile comparison, LFM uses 2.780× Qwen’s count on the ten Hindi–English samples and 3.988× on the ten Tamil–English samples. Gemma uses 0.721× and 0.651× respectively. These are different lessons, not matched translations: the gap cannot be attributed to language alone.

Language-stratified results and reconstruction findings.

Validation & unexpected findings

All 1,120 recorded encodings agree between Python and JavaScript by sequence hash. The independent arithmetic verifier checks metadata, grouped summaries, and ratios with a separate implementation. The 280 new DeepSeek encodings were measured in both runtimes; earlier results were reused with pinned hashes.

Encoding agreement is not the same as text reconstruction. Decoding was checked in Python, not independently in JavaScript. In the earlier six-profile run, the five BPE profiles reconstructed all 140 inputs exactly. XLM-R reconstructed none of the 100 originals exactly, but all 20 compact JSON variants. Most differences were whitespace; one also involved a byte-order mark. Its low formatting overhead therefore needs to be read alongside text preservation, not as an unqualified advantage.

Verification record · Reconstruction diagnostic.

Limitations

The corpus favors technical sources and includes repeated templates, unequal lengths, and automatically translated documentation. Some source and length groups are small. Eight configurations are not eight independent tokenizer families; one Unigram representative cannot establish a general algorithm effect.

Counts generally omit special tokens and exclude complete chat templates. The study does not measure browser startup, latency, memory, model quality, or provider billing. MiniCPM’s Hugging Face/GGUF equivalence remains unverified. FLORES+ remains evaluation-only; it was not accessed in this extension.

Raw source text and reconstructable token sequences are excluded from the repository. Full reproduction needs the separately reconstructed source inputs and pinned tokenizer assets.

What I learned & next steps

I learned to separate tokenizer configuration identity, content counts, and full-prompt costs. The paired JSON test isolates a formatting change; the mixed-language groups do not isolate language.

I would broaden the source sample and measure full chat prompts, browser startup, and memory before choosing a product strategy. No estimator coefficients were fitted by this study.

ContextFair is a separate application that explores conversation budgeting. Its estimates and exact Qwen content counting should not be presented as a live version of this research benchmark.

Inspect & reproduce

Read the extension report and saved summaries first. The pinned setup guide and experiment report give the rerun sequence. Use fresh output directories to preserve frozen runs.

This is a research repository with published artifacts; it does not provide a hosted tokenizer-study application.