## Transformer Model Evolution: Architecture & Parameter Comparison

### Architecture Overview

| | Transformer Base (2017) | BERT-Base (2018) | GPT-2 124M (2019) | GPT-3 175B (2020) | LLaMA 7B (2023) |
|---|---|---|---|---|---|
| Type | Encoder-Decoder | Encoder-only | Decoder-only | Decoder-only | Decoder-only |
| d_model | 512 | 768 | 768 | 12,288 | 4,096 |
| d_ff | 2,048 | 3,072 | 3,072 | 49,152 | 11,008 |
| Heads | 8 | 12 | 12 | 96 | 32 |
| Layers | 6 enc + 6 dec | 12 enc | 12 dec | 96 dec | 32 dec |
| Vocab | 37,000 | 30,522 | 50,304 | 50,257 | 32,000 |
| Context | 256 | 512 | 1,024 | 2,048 | 2,048 |
| Position | Sinusoidal | Learned | Learned | Learned | RoPE |
| Norm | Post-LN | Post-LN | Pre-LN | Pre-LN | RMSNorm (Pre) |
| FFN activation | ReLU | GELU | GELU | GELU | SwiGLU |
| Bias | Yes | Yes | Yes | Yes | **No** |
| Weight tying | Embed + Generator | -- | Embed + lm_head | Embed + lm_head | **None** |
| **Total params** | **63M** | **110M** | **124M** | **175B** | **6.7B** |

---

### Detailed Parameter Calculations

#### 1. Transformer Base (this project) -- 63M

Config: `d=512, d_ff=2048, h=8, N=6+6, V=37000`

| Component | Calculation | Params |
|---|---|---|
| **Embedding** | 37000 x 512 (shared, tied w/ generator) | 18,944,000 |
| **Attention** (6 enc layers) | 6 x 4 x (512x512 + 512) | 6,303,744 |
| **Attention** (6 dec layers, self+cross) | 6 x 2 x 4 x (512x512 + 512) | 12,607,488 |
| **FFN** (12 layers total) | 12 x [(512x2048+2048) + (2048x512+512)] | 25,196,544 |
| **LayerNorm** | 6x2x1024 + 6x3x1024 + 2x1024 | 32,768 |
| **Total** | | **63,084,544** |

| Category | Params | % |
|---|---|---|
| Embedding | 18.9M | 30.0% |
| Attention | 18.9M | 30.0% |
| FFN | 25.2M | 40.0% |
| Norm | 0.03M | 0.1% |

---

#### 2. BERT-Base -- 110M

Config: `d=768, d_ff=3072, h=12, N=12, V=30522`

| Component | Calculation | Params |
|---|---|---|
| **Token embed** | 30522 x 768 | 23,440,896 |
| **Position embed** | 512 x 768 | 393,216 |
| **Segment embed** | 2 x 768 | 1,536 |
| **Embed LN** | 768 + 768 | 1,536 |
| **Attention** (12 layers) | 12 x 4 x (768x768 + 768) | 28,348,416 |
| **FFN** (12 layers) | 12 x [(768x3072+3072) + (3072x768+768)] | 56,669,184 |
| **LayerNorm** (12 layers) | 12 x 2 x (768+768) | 36,864 |
| **Pooler** | 768x768 + 768 | 590,592 |
| **Total** | | **109,482,240** |

| Category | Params | % |
|---|---|---|
| Embedding | 23.8M | 21.8% |
| Attention | 28.3M | 25.9% |
| FFN | 56.7M | 51.8% |
| Pooler + Norm | 0.6M | 0.6% |

---

#### 3. GPT-2 124M -- 124M

Config: `d=768, d_ff=3072, h=12, N=12, V=50304`

| Component | Calculation | Params |
|---|---|---|
| **Token embed** | 50304 x 768 (tied w/ lm_head) | 38,633,472 |
| **Position embed** | 1024 x 768 | 786,432 |
| **Attention** (12 layers) | 12 x [(768x2304+2304) + (768x768+768)] | 28,348,416 |
| **FFN** (12 layers) | 12 x [(768x3072+3072) + (3072x768+768)] | 56,669,184 |
| **LayerNorm** | 12 x 2 x 1536 + 1536 | 38,400 |
| **Total** | | **124,475,904** |

| Category | Params | % |
|---|---|---|
| Embedding | 39.4M | 31.7% |
| Attention | 28.3M | 22.8% |
| FFN | 56.7M | 45.5% |
| Norm | 0.04M | 0.0% |

---

#### 4. GPT-3 175B -- 175B

Config: `d=12288, d_ff=49152, h=96, N=96, V=50257`

| Component | Calculation | Params |
|---|---|---|
| **Token embed** | 50257 x 12288 (tied w/ lm_head) | 617,558,016 |
| **Position embed** | 2048 x 12288 | 25,165,824 |
| **Attention** (96 layers) | 96 x [(12288x36864+36864) + (12288x12288+12288)] | 57,986,777,088 |
| **FFN** (96 layers) | 96 x [(12288x49152+49152) + (49152x12288+12288)] | 115,970,015,232 |
| **LayerNorm** | 96 x 2 x 24576 + 24576 | 4,743,168 |
| **Total** | | **174,604,259,328** |

| Category | Params | % |
|---|---|---|
| Embedding | 642.7M | 0.4% |
| Attention | 58.0B | 33.2% |
| FFN | 116.0B | 66.4% |
| Norm | 4.7M | 0.0% |

---

#### 5. LLaMA 7B -- 6.7B

Config: `d=4096, d_ff=11008, h=32, N=32, V=32000, no bias`

| Component | Calculation | Params |
|---|---|---|
| **Token embed** | 32000 x 4096 | 131,072,000 |
| **lm_head** (NOT tied) | 32000 x 4096 | 131,072,000 |
| **Attention** (32 layers) | 32 x 4 x (4096x4096) | 2,147,483,648 |
| **SwiGLU FFN** (32 layers) | 32 x 3 x (4096x11008) | 4,328,521,728 |
| **RMSNorm** | 32 x 2 x 4096 + 4096 | 266,240 |
| **Total** | | **6,738,415,616** |

| Category | Params | % |
|---|---|---|
| Embedding + lm_head | 262.1M | 3.9% |
| Attention | 2,147.5M | 31.9% |
| FFN (SwiGLU, 3 matrices) | 4,328.5M | 64.2% |
| Norm | 0.3M | 0.0% |

Note: SwiGLU uses **3** weight matrices (gate, up, down) instead of the standard 2, which is why FFN dominates even more.

---

### Key Trends

| Trend | Detail |
|---|---|
| **FFN dominates** | FFN grows from ~40% (small) to ~66% (large) of total params as models scale |
| **Embedding shrinks** | From 30% (Transformer) to <1% (GPT-3) -- embedding is fixed-cost, doesn't scale with depth |
| **Encoder-Decoder to Decoder-only** | Every major model after T5 is decoder-only; simpler, scales better |
| **Bias removed** | LLaMA drops all bias terms -- saves params, no quality loss |
| **SwiGLU replaces ReLU/GELU** | 3 matrices instead of 2, but better quality per FLOP |
| **Learned norms to RMSNorm** | Cheaper, no mean computation, half the params |
| **Learned position to RoPE** | No position embedding params, extrapolates to longer sequences |
| **Weight tying abandoned** | LLaMA untied lm_head -- at 7B scale the 131M overhead is negligible |