Annotated Transformer Vocab Size Investigation: 60K vs 8K
Background
The Annotated Transformer tutorial reports vocabulary sizes of 59,981 (DE) / 36,745 (EN) when building vocab from Multi30k with min_freq=2. While reproducing the tutorial, I got 8,316 (DE) / 6,384 (EN) using the same dataset and parameters. This writeup investigates the discrepancy.
Setup
The reference code at https://nlp.seas.harvard.edu/annotated-transformer/ uses:
torchtext.datasets.Multi30k(language_pair=("de", "en"))build_vocab_from_iterator(..., min_freq=2, specials=["<s>", "</s>", "<blank>", "<unk>"])- spaCy tokenizers:
de_core_news_sm/en_core_web_sm - Vocab built on combined train + val + test splits
My reproduction uses the same parameters but loads data via load_dataset("bentrevett/multi30k") from HuggingFace, with a custom Vocab class that mirrors build_vocab_from_iterator.
Experiment
Created a separate venv with torch==2.1.2, torchtext==0.16.2, and torchdata==0.7.1 to run the reference code directly against torchtext.datasets.Multi30k.
Results
| torchtext Multi30k | bentrevett/multi30k | Reference claim | |
|---|---|---|---|
| Train pairs | 29,001 | 29,000 | — |
| Val pairs | 1,015 | 1,014 | — |
| Test pairs | ~1,000 (download broken) | 1,000 | — |
| Total | ~31,016 | 31,014 | — |
| Unique DE tokens | 19,617 | 19,949 | — |
| Unique EN tokens | 11,006 | 11,154 | — |
| DE vocab (min_freq=2) | 8,185 | 8,316 | 59,981 |
| EN vocab (min_freq=2) | 6,291 | 6,384 | 36,745 |
The small differences between the two Multi30k sources come from:
- Off-by-one in pair counts (trailing newlines in torchtext's raw files)
- Missing test split in the torchtext run (the Multi30k test server at
quest.dcs.shef.ac.ukreturns a corrupted archive)
The Math
With ~31K sentence pairs, the maximum possible unique German tokens is ~19,600–19,950. After filtering to min_freq>=2, only ~8,200–8,300 survive. Getting 59,981 unique tokens with min_freq=2 is mathematically impossible from this dataset.
To reach 59,981 DE vocab tokens with min_freq=2, you'd need millions of sentence pairs — consistent with WMT14 (~4.5M training pairs), not Multi30k (~29K).
Conclusion
The 59,981/36,745 numbers shown on the Annotated Transformer page were not produced from Multi30k. They were likely generated from an earlier experiment using WMT14 (the page's vocab.pt was cached and never regenerated after the dataset was changed). The correct vocab sizes for Multi30k with min_freq=2 and spaCy tokenization are approximately 8,300 DE / 6,400 EN.
Both torchtext.datasets.Multi30k and bentrevett/multi30k on HuggingFace contain the same underlying data (WMT16 Multimodal Translation Task 1 / Flickr30k) and produce equivalent vocabularies.