Modern LLM Technology Knowledge Framework

@hao CC BY 4.0

================================================================================
MODERN LLM TECHNOLOGY KNOWLEDGE FRAMEWORK

Background: Built GPT-2 124M (nanoGPT), fine-tuned with Dolly 15k using SFT

================================================================================
COMPLETE STACK DIAGRAM

                        ┌─────────────────────────────────────┐
                        │         MODERN LLM STACK            │
                        └─────────────────────────────────────┘
                                        │
    ┌───────────────────────────────────┼───────────────────────────────────┐
    │                                   │                                   │
    ▼                                   ▼                                   ▼

┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ 1. ARCHITECTURE │ │ 2. TRAINING │ │ 3. ALIGNMENT │
│ INNOVATIONS │ │ AT SCALE │ │ (Post-Training) │
└───────────────────┘ └───────────────────┘ └───────────────────┘
│ │ │
├─ ★ RoPE (positions) ├─ ★ Distributed Training ├─ ★ SFT ✓ (done)
├─ ★ GQA/MQA │ (DP, TP, PP, FSDP) ├─ ★ RLHF
├─ ★ Flash Attention ├─ ★ Mixed Precision (BF16) │ ├─ ★ Reward Model
├─ ★ KV Cache ├─ ★ ZeRO Optimizer │ └─ PPO
├─ Mixture of Experts ├─ Gradient Checkpointing ├─ ★ DPO
├─ Sliding Window ├─ Large Batch Training ├─ RLAIF
└─ ALiBi └─ Curriculum Learning └─ Constitutional AI

    ┌───────────────────────────────────┼───────────────────────────────────┐
    │                                   │                                   │
    ▼                                   ▼                                   ▼

┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ 4. INFERENCE │ │ 5. EFFICIENT │ │ 6. EXTENDED │
│ OPTIMIZATION │ │ FINE-TUNING │ │ CAPABILITIES │
└───────────────────┘ └───────────────────┘ └───────────────────┘
│ │ │
├─ ★ Quantization ├─ ★ LoRA / QLoRA ├─ ★ Long Context
│ (INT8, INT4, AWQ, GPTQ) ├─ Adapters │ (RoPE scaling)
├─ Speculative Decoding ├─ Prefix Tuning ├─ ★ Chain-of-Thought
├─ ★ Continuous Batching └─ DoRA ├─ ★ Tool Use / Agents
├─ ★ KV Cache Paging ├─ ★ RAG
└─ vLLM / TensorRT-LLM └─ Multi-modal (Vision)

    ┌───────────────────────────────────┼───────────────────────────────────┐
    │                                   │                                   │
    ▼                                   ▼                                   ▼

┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐
│ 7. DATA │ │ 8. EVALUATION │ │ 9. TOKENIZATION │
│ ENGINEERING │ │ & SAFETY │ │ │
└───────────────────┘ └───────────────────┘ └───────────────────┘
│ │ │
├─ ★ Data Filtering/Quality ├─ ★ Benchmarks (MMLU, ├─ ★ BPE ✓ (done)
├─ ★ Deduplication │ HumanEval, MATH) ├─ SentencePiece
├─ Data Mixing Ratios ├─ Red Teaming ├─ Tiktoken
└─ Synthetic Data Gen └─ Adversarial Evaluation └─ Unigram

════════════════════════════════════════════════════════════════════════════════
▼ FOUNDATIONS ▼
════════════════════════════════════════════════════════════════════════════════

┌─────────────────────────────────────────────────────────────────────────────────┐
│ 0. TRANSFORMER FUNDAMENTALS │
│ (Learned in nanoGPT) │
└─────────────────────────────────────────────────────────────────────────────────┘

┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ ATTENTION │ │ ARCHITECTURE │ │ OPTIMIZATION │
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
│ ★ Self-Attention │ │ ★ Transformer Block │ │ ★ Adam/AdamW │
│ ★ Multi-Head Attn │ │ ★ Residual Conn │ │ ★ Cross-Entropy │
│ ★ Scaled Dot-Product│ │ ★ Layer Norm │ │ ★ Backprop │
│ ★ Causal Masking │ │ ★ FFN (MLP) │ │ ★ LR Scheduling │
│ Attention Scores │ │ ★ Embeddings │ │ Gradient Clipping │
│ Softmax │ │ Dropout │ │ Weight Decay │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘

┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ MATH/THEORY │ │ LANGUAGE MODEL │ │ SCALING │
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
│ ★ Matrix Multiply │ │ ★ Next-Token Pred │ │ ★ Scaling Laws │
│ Linear Algebra │ │ ★ Autoregressive │ │ Chinchilla │
│ Probability │ │ ★ Perplexity │ │ Compute-Optimal │
│ Information Theory│ │ Temperature │ │ Emergent Abilities│
│ Gradient Descent │ │ Top-k / Top-p │ │ │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘

================================================================================
LEGEND

★ = Core technique - essential to understand modern LLMs
✓ = Already learned/implemented

================================================================================
PRIORITY TIERS FOR LEARNING

TIER 1 (Must Know) TIER 2 (Important) TIER 3 (Advanced)
───────────────── ────────────────── ─────────────────
★ Foundations (done) ★ Flash Attention Mixture of Experts
★ SFT (done) ★ Distributed Training Speculative Decoding
★ RLHF / DPO ★ ZeRO Constitutional AI
★ RoPE ★ Continuous Batching Sliding Window
★ GQA ★ Long Context RLAIF
★ KV Cache ★ Tool Use Synthetic Data
★ LoRA ★ Data Quality Multi-modal
★ Quantization ★ Benchmarking
★ RAG
★ Chain-of-Thought

================================================================================
RECOMMENDED LEARNING PATHS

If you want to... Start with
───────────────── ──────────
Make your model actually helpful 3. Alignment → RLHF or DPO
Understand Llama/Mistral internals 1. Architecture → RoPE, GQA, Flash Attention
Train larger models 2. Training at Scale → Distributed training
Run models efficiently 4. Inference → Quantization, vLLM
Fine-tune cheaply 5. Efficient Fine-tuning → LoRA/QLoRA
Build ChatGPT-like apps 6. Capabilities → Tool use, RAG

================================================================================
SUGGESTED NEXT STEPS (POST-SFT)

  1. RLHF/DPO - Make your model align to human preferences
  2. LoRA - Fine-tune efficiently without full training
  3. Architecture - Understand Llama/Mistral improvements over GPT-2

================================================================================
KEY PAPERS TO READ

FOUNDATIONS:

  • "Attention Is All You Need" (2017) - Original Transformer

ARCHITECTURE:

  • "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021)
  • "GQA: Training Generalized Multi-Query Transformer" (2023)
  • "FlashAttention: Fast and Memory-Efficient Exact Attention" (2022)

ALIGNMENT:

  • "Training language models to follow instructions with human feedback" (2022) - InstructGPT/RLHF
  • "Direct Preference Optimization" (2023) - DPO

EFFICIENT FINE-TUNING:

  • "LoRA: Low-Rank Adaptation of Large Language Models" (2021)
  • "QLoRA: Efficient Finetuning of Quantized LLMs" (2023)

SCALING:

  • "Scaling Laws for Neural Language Models" (2020) - OpenAI
  • "Training Compute-Optimal Large Language Models" (2022) - Chinchilla

INFERENCE:

  • "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (2022)
  • "Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023) - vLLM