# Annotated Transformer Vocab Size Investigation: 60K vs 8K

## Background

The [Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/) tutorial reports vocabulary sizes of **59,981 (DE) / 36,745 (EN)** when building vocab from Multi30k with `min_freq=2`. While reproducing the tutorial, I got **8,316 (DE) / 6,384 (EN)** using the same dataset and parameters. This writeup investigates the discrepancy.

## Setup

The reference code at https://nlp.seas.harvard.edu/annotated-transformer/ uses:
- `torchtext.datasets.Multi30k(language_pair=("de", "en"))`
- `build_vocab_from_iterator(..., min_freq=2, specials=["<s>", "</s>", "<blank>", "<unk>"])`
- spaCy tokenizers: `de_core_news_sm` / `en_core_web_sm`
- Vocab built on combined train + val + test splits

My reproduction uses the same parameters but loads data via `load_dataset("bentrevett/multi30k")` from HuggingFace, with a custom `Vocab` class that mirrors `build_vocab_from_iterator`.

## Experiment

Created a separate venv with `torch==2.1.2`, `torchtext==0.16.2`, and `torchdata==0.7.1` to run the reference code directly against `torchtext.datasets.Multi30k`.

### Results

| | torchtext Multi30k | bentrevett/multi30k | Reference claim |
|---|---|---|---|
| Train pairs | 29,001 | 29,000 | — |
| Val pairs | 1,015 | 1,014 | — |
| Test pairs | ~1,000 (download broken) | 1,000 | — |
| **Total** | **~31,016** | **31,014** | — |
| Unique DE tokens | 19,617 | 19,949 | — |
| Unique EN tokens | 11,006 | 11,154 | — |
| **DE vocab (min_freq=2)** | **8,185** | **8,316** | **59,981** |
| **EN vocab (min_freq=2)** | **6,291** | **6,384** | **36,745** |

The small differences between the two Multi30k sources come from:
- Off-by-one in pair counts (trailing newlines in torchtext's raw files)
- Missing test split in the torchtext run (the Multi30k test server at `quest.dcs.shef.ac.uk` returns a corrupted archive)

### The Math

With ~31K sentence pairs, the maximum possible unique German tokens is ~19,600–19,950. After filtering to `min_freq>=2`, only ~8,200–8,300 survive. **Getting 59,981 unique tokens with `min_freq=2` is mathematically impossible from this dataset.**

To reach 59,981 DE vocab tokens with `min_freq=2`, you'd need millions of sentence pairs — consistent with **WMT14** (~4.5M training pairs), not Multi30k (~29K).

## Conclusion

The 59,981/36,745 numbers shown on the [Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/) page were **not produced from Multi30k**. They were likely generated from an earlier experiment using WMT14 (the page's `vocab.pt` was cached and never regenerated after the dataset was changed). The correct vocab sizes for Multi30k with `min_freq=2` and spaCy tokenization are approximately **8,300 DE / 6,400 EN**.

Both `torchtext.datasets.Multi30k` and `bentrevett/multi30k` on HuggingFace contain the same underlying data (WMT16 Multimodal Translation Task 1 / Flickr30k) and produce equivalent vocabularies.