Architecture Overview
|
Transformer Base (2017) |
BERT-Base (2018) |
GPT-2 124M (2019) |
GPT-3 175B (2020) |
LLaMA 7B (2023) |
| Type |
Encoder-Decoder |
Encoder-only |
Decoder-only |
Decoder-only |
Decoder-only |
| d_model |
512 |
768 |
768 |
12,288 |
4,096 |
| d_ff |
2,048 |
3,072 |
3,072 |
49,152 |
11,008 |
| Heads |
8 |
12 |
12 |
96 |
32 |
| Layers |
6 enc + 6 dec |
12 enc |
12 dec |
96 dec |
32 dec |
| Vocab |
37,000 |
30,522 |
50,304 |
50,257 |
32,000 |
| Context |
256 |
512 |
1,024 |
2,048 |
2,048 |
| Position |
Sinusoidal |
Learned |
Learned |
Learned |
RoPE |
| Norm |
Post-LN |
Post-LN |
Pre-LN |
Pre-LN |
RMSNorm (Pre) |
| FFN activation |
ReLU |
GELU |
GELU |
GELU |
SwiGLU |
| Bias |
Yes |
Yes |
Yes |
Yes |
No |
| Weight tying |
Embed + Generator |
-- |
Embed + lm_head |
Embed + lm_head |
None |
| Total params |
63M |
110M |
124M |
175B |
6.7B |
Detailed Parameter Calculations
1. Transformer Base (this project) -- 63M
Config: d=512, d_ff=2048, h=8, N=6+6, V=37000
| Component |
Calculation |
Params |
| Embedding |
37000 x 512 (shared, tied w/ generator) |
18,944,000 |
| Attention (6 enc layers) |
6 x 4 x (512x512 + 512) |
6,303,744 |
| Attention (6 dec layers, self+cross) |
6 x 2 x 4 x (512x512 + 512) |
12,607,488 |
| FFN (12 layers total) |
12 x [(512x2048+2048) + (2048x512+512)] |
25,196,544 |
| LayerNorm |
6x2x1024 + 6x3x1024 + 2x1024 |
32,768 |
| Total |
|
63,084,544 |
| Category |
Params |
% |
| Embedding |
18.9M |
30.0% |
| Attention |
18.9M |
30.0% |
| FFN |
25.2M |
40.0% |
| Norm |
0.03M |
0.1% |
2. BERT-Base -- 110M
Config: d=768, d_ff=3072, h=12, N=12, V=30522
| Component |
Calculation |
Params |
| Token embed |
30522 x 768 |
23,440,896 |
| Position embed |
512 x 768 |
393,216 |
| Segment embed |
2 x 768 |
1,536 |
| Embed LN |
768 + 768 |
1,536 |
| Attention (12 layers) |
12 x 4 x (768x768 + 768) |
28,348,416 |
| FFN (12 layers) |
12 x [(768x3072+3072) + (3072x768+768)] |
56,669,184 |
| LayerNorm (12 layers) |
12 x 2 x (768+768) |
36,864 |
| Pooler |
768x768 + 768 |
590,592 |
| Total |
|
109,482,240 |
| Category |
Params |
% |
| Embedding |
23.8M |
21.8% |
| Attention |
28.3M |
25.9% |
| FFN |
56.7M |
51.8% |
| Pooler + Norm |
0.6M |
0.6% |
3. GPT-2 124M -- 124M
Config: d=768, d_ff=3072, h=12, N=12, V=50304
| Component |
Calculation |
Params |
| Token embed |
50304 x 768 (tied w/ lm_head) |
38,633,472 |
| Position embed |
1024 x 768 |
786,432 |
| Attention (12 layers) |
12 x [(768x2304+2304) + (768x768+768)] |
28,348,416 |
| FFN (12 layers) |
12 x [(768x3072+3072) + (3072x768+768)] |
56,669,184 |
| LayerNorm |
12 x 2 x 1536 + 1536 |
38,400 |
| Total |
|
124,475,904 |
| Category |
Params |
% |
| Embedding |
39.4M |
31.7% |
| Attention |
28.3M |
22.8% |
| FFN |
56.7M |
45.5% |
| Norm |
0.04M |
0.0% |
4. GPT-3 175B -- 175B
Config: d=12288, d_ff=49152, h=96, N=96, V=50257
| Component |
Calculation |
Params |
| Token embed |
50257 x 12288 (tied w/ lm_head) |
617,558,016 |
| Position embed |
2048 x 12288 |
25,165,824 |
| Attention (96 layers) |
96 x [(12288x36864+36864) + (12288x12288+12288)] |
57,986,777,088 |
| FFN (96 layers) |
96 x [(12288x49152+49152) + (49152x12288+12288)] |
115,970,015,232 |
| LayerNorm |
96 x 2 x 24576 + 24576 |
4,743,168 |
| Total |
|
174,604,259,328 |
| Category |
Params |
% |
| Embedding |
642.7M |
0.4% |
| Attention |
58.0B |
33.2% |
| FFN |
116.0B |
66.4% |
| Norm |
4.7M |
0.0% |
5. LLaMA 7B -- 6.7B
Config: d=4096, d_ff=11008, h=32, N=32, V=32000, no bias
| Component |
Calculation |
Params |
| Token embed |
32000 x 4096 |
131,072,000 |
| lm_head (NOT tied) |
32000 x 4096 |
131,072,000 |
| Attention (32 layers) |
32 x 4 x (4096x4096) |
2,147,483,648 |
| SwiGLU FFN (32 layers) |
32 x 3 x (4096x11008) |
4,328,521,728 |
| RMSNorm |
32 x 2 x 4096 + 4096 |
266,240 |
| Total |
|
6,738,415,616 |
| Category |
Params |
% |
| Embedding + lm_head |
262.1M |
3.9% |
| Attention |
2,147.5M |
31.9% |
| FFN (SwiGLU, 3 matrices) |
4,328.5M |
64.2% |
| Norm |
0.3M |
0.0% |
Note: SwiGLU uses 3 weight matrices (gate, up, down) instead of the standard 2, which is why FFN dominates even more.
Key Trends
| Trend |
Detail |
| FFN dominates |
FFN grows from ~40% (small) to ~66% (large) of total params as models scale |
| Embedding shrinks |
From 30% (Transformer) to <1% (GPT-3) -- embedding is fixed-cost, doesn't scale with depth |
| Encoder-Decoder to Decoder-only |
Every major model after T5 is decoder-only; simpler, scales better |
| Bias removed |
LLaMA drops all bias terms -- saves params, no quality loss |
| SwiGLU replaces ReLU/GELU |
3 matrices instead of 2, but better quality per FLOP |
| Learned norms to RMSNorm |
Cheaper, no mean computation, half the params |
| Learned position to RoPE |
No position embedding params, extrapolates to longer sequences |
| Weight tying abandoned |
LLaMA untied lm_head -- at 7B scale the 131M overhead is negligible |