================================================================================
TIER 1 DEEP DIVE: CORE LLM TECHNIQUES

Prerequisites: Transformer fundamentals, SFT (assumed known from nanoGPT + Dolly)

Topics Covered:

1. RLHF (Reinforcement Learning from Human Feedback)
2. DPO (Direct Preference Optimization)
3. RoPE (Rotary Position Embeddings)
4. GQA (Grouped Query Attention)
5. KV Cache
6. LoRA (Low-Rank Adaptation)
7. Quantization
8. RAG (Retrieval Augmented Generation)
9. Chain-of-Thought (CoT)

################################################################################

1. RLHF - REINFORCEMENT LEARNING FROM

HUMAN FEEDBACK

################################################################################

WHAT IS IT?
───────────
RLHF is the technique that transformed GPT-3 into ChatGPT. It aligns language
models to human preferences using reinforcement learning, making them helpful,
harmless, and honest.

WHY IT MATTERS
──────────────

• Base LLMs just predict next tokens - they don't know what humans actually want
• SFT teaches format but not nuanced preferences
• RLHF teaches the model to optimize for human satisfaction
• This is THE key innovation behind ChatGPT's success

THE THREE-STAGE PIPELINE
────────────────────────

┌─────────────────────────────────────────────────────────────────────────────┐
│ Stage 1: Supervised Fine-Tuning (SFT) │
│ ───────────────────────────────────── │
│ • Start with pretrained base model │
│ • Fine-tune on high-quality demonstration data │
│ • Human labelers write ideal responses to prompts │
│ • Standard cross-entropy loss │
│ • Result: Model that can follow instructions (you did this!) │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Stage 2: Reward Model Training │
│ ───────────────────────────────── │
│ • Collect comparison data: humans rank multiple outputs │
│ • For prompt X, model generates responses A and B │
│ • Human says: "A is better than B" (or vice versa) │
│ • Train reward model to predict human preferences │
│ │
│ Architecture: │
│ ┌──────────┐ ┌──────────┐ ┌────────┐ │
│ │ Prompt │───▶│ LLM │───▶│ Scalar │ (reward score) │
│ │+Response │ │ (frozen) │ │ Head │ │
│ └──────────┘ └──────────┘ └────────┘ │
│ │
│ Loss function (Bradley-Terry model): │
│ L = -log(σ(r(x,y_w) - r(x,y_l))) │
│ │
│ Where: │
│ r(x,y) = reward for response y given prompt x │
│ y_w = preferred (winning) response │
│ y_l = rejected (losing) response │
│ σ = sigmoid function │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ Stage 3: RL Fine-Tuning with PPO │
│ ──────────────────────────────── │
│ • Use reward model to score generations │
│ • Optimize policy (the LLM) to maximize reward │
│ • PPO (Proximal Policy Optimization) prevents too-large updates │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Prompt │────▶│ Policy │────▶│ Response │────▶│ Reward │ │
│ │ │ │ (LLM) │ │ │ │ Model │ │
│ └──────────┘ └──────────┘ └──────────┘ └────┬─────┘ │
│ ▲ │ │
│ │ ┌──────────┐ │ │
│ └─────────│ PPO │◀──────────┘ │
│ update │ Optimize │ reward │
│ └──────────┘ │
│ │
│ Key components: │
│ • Policy: The LLM being trained │
│ • Reference model: Frozen copy of SFT model (prevents drift) │
│ • KL penalty: Keeps policy close to reference │
│ │
│ Objective: │
│ maximize E[r(x,y)] - β * KL(π_θ || π_ref) │
│ │
│ The KL term prevents reward hacking (gaming the reward model) │
└─────────────────────────────────────────────────────────────────────────────┘

PPO SPECIFICS
─────────────
PPO is used because it's stable and sample-efficient:

L_PPO = min(r_t(θ) * A_t, clip(r_t(θ), 1-ε, 1+ε) * A_t)

Where:

• r_t(θ) = π_θ(a|s) / π_old(a|s) (probability ratio)
• A_t = advantage estimate
• ε = clipping parameter (typically 0.2)

The clipping prevents the policy from changing too drastically in one update.

PRACTICAL CONSIDERATIONS
────────────────────────
• Reward hacking: Model finds exploits in reward model

• Solution: KL penalty, reward model ensembles
  • Labeler disagreement: Humans often disagree
• Solution: Multiple labelers, clear guidelines
  • Compute intensive: Need to run inference during training
• Solution: Efficient batching, model parallelism
  • Mode collapse: Model outputs become repetitive
• Solution: Entropy bonus, diverse prompts

KEY PAPERS
──────────
• "Training language models to follow instructions with human feedback" (2022)

• The InstructGPT paper from OpenAI
  • "Learning to summarize from human feedback" (2020)
• Earlier work that laid groundwork

################################################################################

2. DPO - DIRECT PREFERENCE OPTIMIZATION

################################################################################

WHAT IS IT?
───────────
DPO is a simpler alternative to RLHF that skips the reward model entirely.
It directly optimizes the language model on preference data.

WHY IT MATTERS
──────────────
• RLHF is complex: 3 models (policy, reference, reward), PPO is finicky
• DPO achieves similar results with just supervised learning
• Much easier to implement and debug
• More stable training dynamics
• Becoming the preferred method for many practitioners

THE KEY INSIGHT
───────────────
The optimal policy under RLHF has a closed-form solution!

Given reward r(x,y) and reference policy π_ref:

π*(y|x) = (1/Z(x)) * π_ref(y|x) * exp(r(x,y)/β)

This means we can express the reward in terms of policies:

r(x,y) = β * log(π*(y|x) / π_ref(y|x)) + β * log(Z(x))

DPO substitutes this into the Bradley-Terry preference model and derives
a loss that only depends on the policy (no explicit reward model needed).

DPO LOSS FUNCTION
─────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ L_DPO(π_θ; π_ref) = -E[ log σ( β * (log π_θ(y_w|x)/π_ref(y_w|x) │
│ - log π_θ(y_l|x)/π_ref(y_l|x)) ) ] │
│ │
│ Simplified: │
│ L = -log σ( β * (Δlog_prob_winner - Δlog_prob_loser) ) │
│ │
│ Where: │
│ - y_w = preferred response │
│ - y_l = rejected response │
│ - β = temperature parameter (controls deviation from reference) │
│ - Δlog_prob = log π_θ(y|x) - log π_ref(y|x) │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

INTUITION
─────────
The loss increases the probability of preferred responses while decreasing
the probability of rejected responses, relative to the reference model.

• If model prefers winner over loser → loss is low ✓
• If model prefers loser over winner → loss is high ✗

The reference model acts as an anchor to prevent the model from
degenerating (e.g., assigning all probability to one token).

TRAINING PROCEDURE
──────────────────

1. Start with SFT model (this becomes both π_θ and π_ref initially)
2. Freeze a copy as π_ref
3. For each preference pair (x, y_w, y_l):
   a. Compute log probs under π_θ for both responses
   b. Compute log probs under π_ref for both responses (cached, no grad)
   c. Compute DPO loss
   d. Update π_θ

COMPARISON: RLHF vs DPO
───────────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ RLHF DPO │
├─────────────────────────────────────────────────────────────────────────────┤
│ Models needed 3 (policy, ref, reward) 2 (policy, ref) │
│ Training RL (PPO) - unstable Supervised - stable │
│ Hyperparameters Many (PPO has ~10+) Few (mainly β) │
│ Compute High (online generation) Lower (offline) │
│ Implementation Complex Simple │
│ Performance Gold standard Comparable │
│ Data efficiency Can use same data many Fixed dataset │
│ times via online sampling │
└─────────────────────────────────────────────────────────────────────────────┘

VARIANTS
────────
• IPO (Identity Preference Optimization): Removes log sigmoid for stability
• KTO (Kahneman-Tversky Optimization): Works with just good/bad labels
• ORPO: Combines SFT and preference optimization in one step
• SimPO: Simplified version without reference model

PRACTICAL TIPS
──────────────
• β typically 0.1 to 0.5 (higher = more conservative)
• Learning rate: lower than SFT (1e-6 to 1e-5)
• Batch size matters: larger is generally better
• Data quality > quantity for preference pairs
• Reference model can be periodically updated (iterative DPO)

KEY PAPER
─────────
• "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023)

################################################################################

3. RoPE - ROTARY POSITION EMBEDDINGS

################################################################################

WHAT IS IT?
───────────
RoPE encodes position information by rotating the query and key vectors
in attention. It's used in most modern LLMs (Llama, Mistral, etc.).

WHY IT MATTERS
──────────────
• Original transformer used absolute position embeddings (limited)
• RoPE provides relative position information naturally
• Better extrapolation to longer sequences than seen in training
• Enables efficient long-context extensions
• Now the de facto standard for decoder-only models

THE PROBLEM WITH ABSOLUTE POSITIONS
───────────────────────────────────
Original transformer: add position embedding to token embedding

x_i = token_embed(i) + pos_embed(i)

Problems:
• Model sees absolute positions, not relative distances
• Fixed maximum length (e.g., 512 or 2048)
• Poor generalization to longer sequences

THE ROPE INSIGHT
────────────────
Key idea: Encode position through ROTATION in 2D subspaces.

For position m, rotate the embedding by angle m*θ:

┌ ┐ ┌ ┐ ┌ ┐
│ q'{2i} │ │ cos(mθ_i) -sin(mθ_i)│ │q{2i}│
│ │ = │ │ │ │
│q'{2i+1}│ │ sin(mθ_i) cos(mθ_i)│ │q{2i+1}│
└ ┘ └ ┘ └ ┘

Where θ_i = 10000^(-2i/d) (similar to sinusoidal encoding)

WHY ROTATION WORKS
──────────────────
When computing attention between positions m and n:

q_m · k_n = (R_m * q) · (R_n * k)
= q · (R_m^T * R_n) · k
= q · R_{n-m} · k (rotation matrices compose!)

The dot product only depends on RELATIVE position (n-m), not absolute positions!

IMPLEMENTATION
──────────────
In practice, implemented as element-wise operations (more efficient):

def apply_rope(x, cos, sin):
    # x shape: (batch, seq_len, n_heads, head_dim)
    # Split into pairs
    x1 = x[..., ::2]   # even indices
    x2 = x[..., ::2]   # odd indices

    # Rotate
    rotated = torch.cat([
        x1 * cos - x2 * sin,
        x1 * sin + x2 * cos
    ], dim=-1)
    return rotated

# Precompute frequencies
def precompute_freqs(dim, max_seq_len, base=10000):
    freqs = 1.0 / (base ** (torch.arange(0, dim, 2) / dim))
    t = torch.arange(max_seq_len)
    freqs = torch.outer(t, freqs)  # (seq_len, dim/2)
    cos = freqs.cos()
    sin = freqs.sin()
    return cos, sin

COMPARISON: POSITION ENCODING METHODS
─────────────────────────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ Method Relative? Extrapolate? Used In │
├─────────────────────────────────────────────────────────────────────────────┤
│ Absolute Learned No No GPT-2, BERT │
│ Sinusoidal No Somewhat Original Transformer │
│ ALiBi Yes Yes BLOOM, MPT │
│ RoPE Yes Yes* Llama, Mistral, Qwen │
│ T5 Relative Bias Yes No T5, Flan │
└─────────────────────────────────────────────────────────────────────────────┘

• With appropriate scaling techniques

EXTENDING CONTEXT LENGTH WITH ROPE
──────────────────────────────────
RoPE enables various context extension techniques:

1. Position Interpolation (PI):

   • Scale positions: pos' = pos * (L_train / L_target)
   • Example: trained on 4K, extend to 16K by scaling by 0.25
   • Requires some fine-tuning

2. NTK-aware Scaling:

   • Modify the base frequency: base' = base * α^(dim/(dim-2))
   • Better preserves high-frequency information

3. YaRN (Yet another RoPE extensioN):

   • Combines interpolation with NTK scaling
   • Different scaling for different frequency bands
   • State-of-the-art for context extension

4. Dynamic NTK:

   • Adjust scaling based on actual sequence length
   • No fine-tuning needed (but less optimal)

KEY PAPER
─────────
• "RoFormer: Enhanced Transformer with Rotary Position Embedding" (2021)

################################################################################

4. GQA - GROUPED QUERY ATTENTION

################################################################################

WHAT IS IT?
───────────
GQA is an attention variant where multiple query heads share the same
key-value heads. It's a middle ground between MHA and MQA.

WHY IT MATTERS
──────────────
• KV cache is the memory bottleneck for long sequences
• GQA reduces KV cache size significantly
• Minimal quality loss compared to full MHA
• Used in Llama 2 70B, Mistral, and most modern large models
• Enables longer contexts and larger batch sizes

THE EVOLUTION
─────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ MHA (Multi-Head Attention) - Original Transformer │
│ ───────────────────────────────────────────────── │
│ • Each head has its own Q, K, V projections │
│ • n_heads queries, n_heads keys, n_heads values │
│ • Full expressiveness, high memory │
│ │
│ Q heads: [Q1] [Q2] [Q3] [Q4] [Q5] [Q6] [Q7] [Q8] │
│ K heads: [K1] [K2] [K3] [K4] [K5] [K6] [K7] [K8] │
│ V heads: [V1] [V2] [V3] [V4] [V5] [V6] [V7] [V8] │
│ │
│ MQA (Multi-Query Attention) - Google 2019 │
│ ──────────────────────────────────────────── │
│ • All query heads share ONE key-value head │
│ • n_heads queries, 1 key, 1 value │
│ • Massive memory savings, some quality loss │
│ │
│ Q heads: [Q1] [Q2] [Q3] [Q4] [Q5] [Q6] [Q7] [Q8] │
│ K heads: [─────────────── K1 ───────────────────] │
│ V heads: [─────────────── V1 ───────────────────] │
│ │
│ GQA (Grouped Query Attention) - Google 2023 │
│ ─────────────────────────────────────────────── │
│ • Query heads grouped, each group shares K-V │
│ • n_heads queries, n_kv_heads keys, n_kv_heads values │
│ • Balance of quality and efficiency │
│ │
│ Q heads: [Q1] [Q2] [Q3] [Q4] | [Q5] [Q6] [Q7] [Q8] │
│ K heads: [───── K1 ─────────] [───── K2 ─────────] │
│ V heads: [───── V1 ─────────] [───── V2 ─────────] │
│ └── group 1 ───────┘ └─── group 2 ──────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

MEMORY ANALYSIS
───────────────
KV cache size per token = 2 * n_layers * n_kv_heads * head_dim * bytes

Example: Llama 2 70B
• 80 layers, 64 query heads, 8 KV heads (GQA with 8 groups)
• head_dim = 128
• KV cache per token = 2 * 80 * 8 * 128 * 2 bytes = 328 KB

If it were MHA (64 KV heads):
• KV cache per token = 2 * 80 * 64 * 128 * 2 bytes = 2.6 MB

That's 8x memory savings! For a 4096 context, that's 10GB vs 1.3GB.

IMPLEMENTATION
──────────────

class GroupedQueryAttention(nn.Module):
    def __init__(self, d_model, n_heads, n_kv_heads):
        super().__init__()
        self.n_heads = n_heads
        self.n_kv_heads = n_kv_heads
        self.n_groups = n_heads // n_kv_heads  # queries per KV head
        self.head_dim = d_model // n_heads

        self.q_proj = nn.Linear(d_model, n_heads * self.head_dim)
        self.k_proj = nn.Linear(d_model, n_kv_heads * self.head_dim)
        self.v_proj = nn.Linear(d_model, n_kv_heads * self.head_dim)
        self.o_proj = nn.Linear(n_heads * self.head_dim, d_model)

    def forward(self, x):
        B, L, _ = x.shape

        q = self.q_proj(x).view(B, L, self.n_heads, self.head_dim)
        k = self.k_proj(x).view(B, L, self.n_kv_heads, self.head_dim)
        v = self.v_proj(x).view(B, L, self.n_kv_heads, self.head_dim)

        # Expand K, V to match Q heads (repeat for each group)
        k = k.repeat_interleave(self.n_groups, dim=2)
        v = v.repeat_interleave(self.n_groups, dim=2)

        # Standard attention from here
        # ...

COMMON CONFIGURATIONS
─────────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ Model n_heads n_kv_heads Ratio Type │
├─────────────────────────────────────────────────────────────────────────────┤
│ GPT-2 12 12 1:1 MHA │
│ Llama 1 32 32 1:1 MHA │
│ Llama 2 7B 32 32 1:1 MHA │
│ Llama 2 70B 64 8 8:1 GQA │
│ Mistral 7B 32 8 4:1 GQA │
│ Falcon 40B 64 1 64:1 MQA │
└─────────────────────────────────────────────────────────────────────────────┘

CONVERTING MHA TO GQA
─────────────────────
You can convert existing MHA models to GQA:

1. Group the KV heads
2. Average or select representative heads within each group
3. Fine-tune briefly to recover quality

KEY PAPER
─────────
• "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints" (2023)

################################################################################

5. KV CACHE

################################################################################

WHAT IS IT?
───────────
KV Cache stores the key and value tensors from previous tokens during
autoregressive generation, avoiding redundant computation.

WHY IT MATTERS
──────────────
• Without KV cache: O(n²) computation per token (recompute all attention)
• With KV cache: O(n) computation per token
• Essential for practical inference speeds
• Main memory bottleneck for long sequences
• Understanding it is key to optimizing inference

THE PROBLEM
───────────
Autoregressive generation: predict one token at a time

"The cat" → "sat" → "on" → "the" → "mat"

Naive approach: For each new token, recompute attention over ALL previous tokens

Step 1: Attend over [The] - 1 token
Step 2: Attend over [The, cat] - 2 tokens
Step 3: Attend over [The, cat, sat] - 3 tokens
...
Step n: Attend over all n tokens - n tokens

Total attention computations: 1 + 2 + 3 + ... + n = O(n²)

THE SOLUTION: KV CACHE
──────────────────────
Key insight: K and V for previous tokens DON'T CHANGE!

Attention(Q, K, V) = softmax(QK^T / √d) V

For token at position i:
• Q_i depends only on token i (the new token)
• K_j, V_j for j < i were computed in previous steps

So: Cache all K and V, only compute new K_i, V_i, Q_i

┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ Step 1: Input "The" │
│ ────────────────── │
│ Compute: Q₁, K₁, V₁ │
│ Cache: K_cache = [K₁], V_cache = [V₁] │
│ Attend: Q₁ @ [K₁]ᵀ │
│ │
│ Step 2: Input "cat" │
│ ─────────────────── │
│ Compute: Q₂, K₂, V₂ (only for new token!) │
│ Cache: K_cache = [K₁, K₂], V_cache = [V₁, V₂] │
│ Attend: Q₂ @ [K₁, K₂]ᵀ │
│ │
│ Step 3: Input "sat" │
│ ─────────────────── │
│ Compute: Q₃, K₃, V₃ (only for new token!) │
│ Cache: K_cache = [K₁, K₂, K₃], V_cache = [V₁, V₂, V₃] │
│ Attend: Q₃ @ [K₁, K₂, K₃]ᵀ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

Each step only computes 1 new Q, K, V instead of all previous ones!

MEMORY REQUIREMENTS
───────────────────
KV cache size = 2 * n_layers * seq_len * n_kv_heads * head_dim * bytes

Example: Llama 2 7B, 4096 context, FP16
• 32 layers, 32 KV heads, head_dim = 128
• Size = 2 * 32 * 4096 * 32 * 128 * 2 = 2.1 GB

This is why KV cache is often the bottleneck, not model weights!

IMPLEMENTATION SKETCH
─────────────────────

class CachedAttention(nn.Module):
    def forward(self, x, kv_cache=None, use_cache=True):
        B, L, D = x.shape

        q = self.q_proj(x)
        k = self.k_proj(x)
        v = self.v_proj(x)

        if kv_cache is not None:
            # Concatenate with cached K, V
            k_cache, v_cache = kv_cache
            k = torch.cat([k_cache, k], dim=1)
            v = torch.cat([v_cache, v], dim=1)

        # Standard attention computation
        attn_out = scaled_dot_product_attention(q, k, v)

        if use_cache:
            new_cache = (k, v)
            return attn_out, new_cache
        return attn_out, None

ADVANCED KV CACHE TECHNIQUES
────────────────────────────

1. PagedAttention (vLLM):
   • Manage KV cache like virtual memory pages
   • Non-contiguous allocation reduces fragmentation
   • Enables efficient batching of variable-length sequences

2. Sliding Window Cache:
   • Only keep last W tokens in cache
   • Used in Mistral (W=4096)
   • O(W) memory instead of O(n)

3. KV Cache Quantization:
   • Store cached K, V in lower precision (INT8, INT4)
   • Reduces memory by 2-4x
   • Minimal quality impact

4. KV Cache Compression:
   • Learned compression of cached states
   • Trade compute for memory

5. Speculative Decoding:
   • Use smaller model to draft multiple tokens
   • Verify with large model in parallel
   • Amortizes KV cache updates

PREFILL VS DECODE
─────────────────
Two distinct phases in generation:

Prefill (prompt processing):
• Process entire prompt at once
• Compute-bound (matrix multiplications)
• KV cache is populated

Decode (token generation):
• Generate one token at a time
• Memory-bound (KV cache reads)
• Much slower per-token than prefill

This is why "time to first token" differs from "tokens per second"!

################################################################################

6. LoRA - LOW-RANK ADAPTATION

################################################################################

WHAT IS IT?
───────────
LoRA fine-tunes LLMs by adding small trainable low-rank matrices to frozen
pretrained weights. It's the most popular parameter-efficient fine-tuning method.

WHY IT MATTERS
──────────────
• Full fine-tuning requires storing full model gradients (huge memory)
• LoRA reduces trainable parameters by 10,000x
• Can fine-tune 65B models on consumer GPUs
• Multiple LoRA adapters can be hot-swapped
• Quality comparable to full fine-tuning

THE CORE IDEA
─────────────
Weight updates during fine-tuning are low-rank!

Instead of updating full weight W:
W' = W + ΔW

Decompose ΔW into low-rank matrices:
ΔW = BA where B ∈ R^(d×r), A ∈ R^(r×k), r << min(d,k)

┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ Full Fine-tuning: LoRA: │
│ ───────────────── ───── │
│ │
│ ┌───────────────┐ ┌───────────────┐ │
│ │ │ │ │ │
│ │ W + ΔW │ │ W (frozen) │───┐ │
│ │ (d × k) │ │ (d × k) │ │ │
│ │ │ │ │ │ │
│ └───────────────┘ └───────────────┘ │ ┌─────┐ │
│ ├──│ + │──▶ │
│ Trainable: d × k ┌─────┐ │ └─────┘ │
│ (millions of params) │ B │─────────────┘ │
│ │(d×r)│ │
│ └──┬──┘ │
│ │ │
│ ┌──┴──┐ │
│ │ A │ │
│ │(r×k)│ ◀── trainable │
│ └─────┘ │
│ │
│ Trainable: r×(d+k) │
│ (thousands of params) │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

PARAMETER SAVINGS
─────────────────
Original matrix: d × k parameters
LoRA: r × d + r × k = r(d + k) parameters

Example: d=4096, k=4096, r=8
• Full: 16.7M parameters
• LoRA: 65K parameters (256x reduction!)

WHICH LAYERS TO ADAPT?
──────────────────────
Typically applied to attention projection matrices:
• Q projection (most common)
• V projection (most common)
• K projection (sometimes)
• O projection (sometimes)
• FFN layers (less common, but can help)

Common configs:
• Attention only: q_proj, v_proj
• All attention: q_proj, k_proj, v_proj, o_proj
• Full: + FFN layers

IMPLEMENTATION
──────────────

class LoRALinear(nn.Module):
    def __init__(self, original_layer, r=8, alpha=16):
        super().__init__()
        self.original = original_layer
        self.original.weight.requires_grad = False  # Freeze

        d, k = original_layer.weight.shape
        self.lora_A = nn.Parameter(torch.randn(r, k) * 0.01)
        self.lora_B = nn.Parameter(torch.zeros(d, r))

        self.scaling = alpha / r  # Scale factor

    def forward(self, x):
        # Original path (frozen)
        original_out = self.original(x)

        # LoRA path (trainable)
        lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling

        return original_out + lora_out

KEY HYPERPARAMETERS
───────────────────
• r (rank): 4, 8, 16, 32 typical. Higher = more capacity, more params
• alpha: Scaling factor, typically 16 or 2*r. Affects learning rate effectively
• Target modules: Which layers to apply LoRA to
• Learning rate: Usually higher than full fine-tuning (1e-4 to 3e-4)

MERGING LORA
────────────
After training, LoRA can be merged into base weights (no inference overhead):

W_merged = W + BA * (alpha/r)

This means:
• Training: Keep LoRA separate
• Deployment: Merge for zero overhead OR keep separate for hot-swapping

QLORA: COMBINING WITH QUANTIZATION
──────────────────────────────────
QLoRA = 4-bit quantized base + LoRA adapters

• Base model in 4-bit NormalFloat (NF4)
• LoRA adapters in FP16/BF16
• Double quantization for further savings
• Fine-tune 65B models on single 48GB GPU!

VARIANTS
────────
• LoRA+: Different learning rates for A and B
• DoRA: Decomposes into magnitude and direction
• AdaLoRA: Adaptive rank allocation
• QLoRA: Quantized base model
• LongLoRA: Efficient long-context fine-tuning

KEY PAPERS
──────────
• "LoRA: Low-Rank Adaptation of Large Language Models" (2021)
• "QLoRA: Efficient Finetuning of Quantized LLMs" (2023)

################################################################################

7. QUANTIZATION

################################################################################

WHAT IS IT?
───────────
Quantization reduces model precision from FP32/FP16 to INT8/INT4, making
models smaller and faster while maintaining acceptable quality.

WHY IT MATTERS
──────────────
• 70B parameter model in FP16 = 140GB (won't fit on most hardware)
• Same model in INT4 = 35GB (fits on high-end consumer GPU)
• 2-4x speedup from reduced memory bandwidth
• Enables local deployment of large models
• Critical for edge/mobile deployment

PRECISION FORMATS
─────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ Format Bits Range Use Case │
├─────────────────────────────────────────────────────────────────────────────┤
│ FP32 32 ±3.4×10³⁸ Training (legacy) │
│ FP16 16 ±65504 Training/Inference │
│ BF16 16 ±3.4×10³⁸ Training (better range than FP16) │
│ INT8 8 -128 to 127 Inference │
│ INT4 4 -8 to 7 Inference (aggressive) │
│ NF4 4 Normal dist QLoRA (better for weights) │
│ FP8 8 Varies Emerging for training │
└─────────────────────────────────────────────────────────────────────────────┘

BASIC QUANTIZATION MATH
───────────────────────
Convert floating point to integer:

x_quant = round(x / scale) + zero_point
x_dequant = (x_quant - zero_point) * scale

Where:
scale = (max_val - min_val) / (2^bits - 1)
zero_point = round(-min_val / scale)

QUANTIZATION GRANULARITY
────────────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ Per-tensor: One scale for entire tensor │
│ ──────────────────────────────────────── │
│ ┌─────────────────────────┐ │
│ │ scale = 0.5 │ Simple but less accurate │
│ │ ░░░░░░░░░░░░░░░░░░░░░ │ │
│ └─────────────────────────┘ │
│ │
│ Per-channel: One scale per output channel │
│ ───────────────────────────────────────── │
│ ┌─────────────────────────┐ │
│ │ scale = [0.5, 0.3, 0.7]│ Better accuracy, standard for weights │
│ │ ░░░░░░░░░░░░░░░░░░░░░ │ │
│ │ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ │ │
│ │ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ │ │
│ └─────────────────────────┘ │
│ │
│ Per-group: One scale per group of values (e.g., 128 values) │
│ ─────────────────────────────────────────────────────────── │
│ ┌─────────────────────────┐ │
│ │ [s1][s2][s3][s4]... │ Best accuracy for INT4, used in GPTQ/AWQ │
│ │ ░░░░ ▓▓▓▓ ▒▒▒▒ ████ │ │
│ └─────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

QUANTIZATION METHODS
────────────────────

1. POST-TRAINING QUANTIZATION (PTQ)
   Quantize after training, no retraining needed

   a) Naive Round-to-Nearest (RTN):
   • Just round weights to nearest quantized value
   • Fast but lower quality

   b) GPTQ (GPT Quantization):
   • Uses calibration data to minimize quantization error
   • Quantizes weights one at a time, adjusting remaining weights
   • Based on Optimal Brain Quantization
   • Very popular for INT4

   c) AWQ (Activation-aware Weight Quantization):
   • Identifies important weights based on activation magnitudes
   • Keeps important weights at higher precision
   • Often better than GPTQ

2. QUANTIZATION-AWARE TRAINING (QAT)
   Simulate quantization during training

   • Forward: Use quantized weights
   • Backward: Use straight-through estimator (STE)
   • Better quality but requires training

WEIGHT VS ACTIVATION QUANTIZATION
─────────────────────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ Weights (easy): Activations (hard): │
│ • Static, known at compile time • Dynamic, vary per input │
│ • Gaussian-ish distribution • Can have outliers │
│ • INT4 works well • Usually need INT8 or higher │
│ │
│ Common setups: │
│ • W8A8: INT8 weights, INT8 activations (balanced) │
│ • W4A16: INT4 weights, FP16 activations (popular for inference) │
│ • W4A8: INT4 weights, INT8 activations (aggressive) │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

THE OUTLIER PROBLEM
───────────────────
LLMs have outlier features (very large activation values in few dimensions).
These break naive quantization.

Solutions:
• LLM.int8(): Mixed precision for outlier channels
• SmoothQuant: Migrate quantization difficulty from activations to weights
• GPTQ/AWQ: Handle outliers through calibration

PRACTICAL QUANTIZATION TOOLS
────────────────────────────
• bitsandbytes: Easy INT8/INT4 with LLM.int8(), used by QLoRA
• AutoGPTQ: GPTQ implementation
• AutoAWQ: AWQ implementation
• llama.cpp: GGML/GGUF formats, CPU-optimized
• TensorRT-LLM: NVIDIA's optimized inference

QUALITY vs SIZE TRADEOFF
────────────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ Method Size Reduction Quality Loss Notes │
├─────────────────────────────────────────────────────────────────────────────┤
│ FP16→INT8 2x ~0-1% Very safe │
│ FP16→INT4 4x ~1-3% Good with GPTQ/AWQ │
│ FP16→INT3 5.3x ~3-5% Noticeable degradation │
│ FP16→INT2 8x ~10%+ Significant loss │
└─────────────────────────────────────────────────────────────────────────────┘

Quality loss depends heavily on model size (larger models quantize better).

KEY PAPERS
──────────
• "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale" (2022)
• "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers" (2022)
• "AWQ: Activation-aware Weight Quantization for LLM Compression" (2023)

################################################################################

8. RAG - RETRIEVAL AUGMENTED GENERATION

################################################################################

WHAT IS IT?
───────────
RAG enhances LLMs by retrieving relevant documents from an external knowledge
base and including them in the prompt context.

WHY IT MATTERS
──────────────
• LLMs have fixed knowledge (training cutoff date)
• Fine-tuning for every knowledge update is expensive
• RAG provides "infinite" updatable knowledge
• Reduces hallucinations by grounding in sources
• Enables domain-specific applications
• Core technique for most production LLM applications

THE RAG PIPELINE
────────────────
┌─────────────────────────────────────────────────────────────────────────────┐
│ │
│ INDEXING (Offline) │
│ ────────────────── │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │Documents │───▶│ Chunk │───▶│ Embed │───▶│ Vector │ │
│ │ │ │ │ │ │ │ DB │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ RETRIEVAL + GENERATION (Online) │
│ ─────────────────────────────── │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Query │───▶│ Embed │───▶│ Retrieve │───▶│ Rerank │ │
│ │ │ │ Query │ │ Top-K │ │(optional)│ │
│ └──────────┘ └──────────┘ └──────────┘ └────┬─────┘ │
│ │ │
│ ▼ │
│ ┌──────────┐ ┌──────────────────────────────────────────┐ │
│ │ Response │◀───│ LLM: "Given context: {docs}, answer: {q}"│ │
│ │ │ │ │ │
│ └──────────┘ └──────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘

COMPONENT DEEP DIVE
───────────────────

1. CHUNKING
   Split documents into smaller pieces for embedding

   Strategies:
   • Fixed size: Every N tokens (simple, can break mid-sentence)
   • Recursive: Split by paragraphs, then sentences, then words
   • Semantic: Split at topic boundaries
   • Sentence-based: Keep sentence integrity

   Typical chunk size: 256-512 tokens
   Overlap: 10-20% (prevents losing context at boundaries)

2. EMBEDDING
   Convert text to dense vectors for similarity search

   Popular models:
   • OpenAI text-embedding-ada-002/3-small/3-large
   • Cohere embed-v3
   • BGE (open source, competitive)
   • E5 (open source)
   • GTE (open source)

   Dimension: 384-3072 (higher = more capacity, more compute)

3. VECTOR DATABASE
   Store and search embeddings efficiently

   Options:
   • Pinecone (managed, scalable)
   • Weaviate (open source, feature-rich)
   • Milvus (open source, scalable)
   • Chroma (open source, simple)
   • Qdrant (open source, fast)
   • pgvector (PostgreSQL extension)
   • FAISS (library, not a DB)

4. RETRIEVAL
   Find relevant chunks for a query

   Methods:
   • Dense: Embedding similarity (cosine, dot product)
   • Sparse: BM25, TF-IDF (keyword matching)
   • Hybrid: Combine dense + sparse (often best)

   Typical: Retrieve top 3-10 chunks

5. RERANKING (Optional but recommended)
   Re-score retrieved chunks with a more powerful model

   • Cross-encoders score (query, document) pairs
   • Much more accurate than bi-encoders
   • But too slow for initial retrieval
   • Examples: Cohere Rerank, BGE Reranker

PROMPT CONSTRUCTION
───────────────────

System: You are a helpful assistant. Answer based on the provided context.
If the answer is not in the context, say "I don't know."

Context:
[Retrieved chunk 1]
[Retrieved chunk 2]
[Retrieved chunk 3]

User: {user_question}

ADVANCED RAG TECHNIQUES
───────────────────────

1. Query Transformation:
   • Query expansion: Add synonyms, related terms
   • HyDE: Generate hypothetical answer, embed that instead
   • Multi-query: Generate multiple query variations

2. Hierarchical Retrieval:
   • Summary-level → Document-level → Chunk-level
   • Retrieve parents of matched chunks for more context

3. Self-RAG:
   • Model decides when to retrieve
   • Model evaluates retrieval quality
   • Model generates citations

4. Agentic RAG:
   • Multi-step retrieval with reasoning
   • Query planning and decomposition
   • Tool use for structured data

5. Graph RAG:
   • Build knowledge graph from documents
   • Retrieve via graph traversal
   • Better for multi-hop questions

EVALUATION METRICS
──────────────────
Retrieval:
• Recall@K: % of relevant docs in top K
• MRR: Mean Reciprocal Rank
• NDCG: Normalized Discounted Cumulative Gain

Generation:
• Faithfulness: Does answer match retrieved context?
• Relevance: Does answer address the question?
• RAGAS: Popular RAG evaluation framework

COMMON PITFALLS
───────────────
• Chunk size too large: Dilutes relevance signal
• Chunk size too small: Loses context
• No reranking: Embedding similarity != relevance
• Ignoring metadata: Dates, sources help filtering
• No citation: Users can't verify answers

KEY PAPERS
──────────
• "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020)
• "Self-RAG: Learning to Retrieve, Generate, and Critique" (2023)

################################################################################

9. CHAIN-OF-THOUGHT (CoT)

################################################################################

WHAT IS IT?
───────────
Chain-of-Thought prompting encourages LLMs to generate intermediate reasoning
steps before giving a final answer, dramatically improving performance on
complex tasks.

WHY IT MATTERS
──────────────
• Enables reasoning capabilities not seen in base models
• Can turn a failing model into a succeeding one
• Simple to implement (just add "Let's think step by step")
• Foundational for more advanced reasoning techniques
• Key capability difference between GPT-3 and ChatGPT-era models

THE CORE INSIGHT
────────────────
LLMs struggle with multi-step reasoning when asked directly:

Q: "If John has 3 apples and buys 2 more, then gives half to Mary,
how many does John have?"

Without CoT: "2" (wrong, guessing)
With CoT: "John starts with 3 apples. He buys 2 more, so 3+2=5.
He gives half to Mary, so 5/2=2.5. John has 2.5 apples." ✓

Generating intermediate steps:

1. Breaks complex problems into simpler sub-problems
2. Provides "working memory" in the context
3. Allows error detection and correction
4. Matches how humans solve problems

COT PROMPTING METHODS
─────────────────────

1. ZERO-SHOT COT
   Just add "Let's think step by step"

   ┌─────────────────────────────────────────────────────────────────────────┐
   │ Q: [Complex question] │
   │ │
   │ A: Let's think step by step. │
   │ [Model generates reasoning...] │
   │ Therefore, the answer is [X]. │
   └─────────────────────────────────────────────────────────────────────────┘

   Effectiveness varies by model. Works well on GPT-4, Claude, etc.

2. FEW-SHOT COT
   Provide examples with reasoning chains

   ┌─────────────────────────────────────────────────────────────────────────┐
   │ Q: Roger has 5 tennis balls. He buys 2 more cans of balls. Each can │
   │ has 3 balls. How many tennis balls does he have now? │
   │ A: Roger started with 5 balls. He bought 2 cans of 3 balls each, │
   │ so 2 * 3 = 6 balls. 5 + 6 = 11. The answer is 11. │
   │ │
   │ Q: [New question] │
   │ A: │
   └─────────────────────────────────────────────────────────────────────────┘

3. SELF-CONSISTENCY
   Generate multiple CoT paths, take majority vote

   ┌─────────────────────────────────────────────────────────────────────────┐
   │ Question │
   │ │ │
   │ ┌─────────────┼─────────────┐ │
   │ ▼ ▼ ▼ │
   │ CoT Path 1 CoT Path 2 CoT Path 3 │
   │ Answer: A Answer: B Answer: A │
   │ │ │ │ │
   │ └─────────────┼─────────────┘ │
   │ ▼ │
   │ Majority Vote: A │
   └─────────────────────────────────────────────────────────────────────────┘

   Use temperature > 0 for diverse paths. Significantly improves accuracy.

4. TREE OF THOUGHTS (ToT)
   Explore multiple reasoning branches, backtrack if needed

   ┌─────────────────────────────────────────────────────────────────────────┐
   │ Problem │
   │ │ │
   │ ┌─────────────┼─────────────┐ │
   │ ▼ ▼ ▼ │
   │ Thought 1 Thought 2 Thought 3 │
   │ (score: 8) (score: 3) (score: 7) │
   │ │ ✗ │ │
   │ ┌────┴────┐ ┌───────┴────┐ │
   │ ▼ ▼ ▼ ▼ │
   │ T1.1 T1.2 T3.1 T3.2 │
   │ (score:9) (score:4) (score:6) (score:8) │
   │ │ ✗ ✗ │ │
   │ ▼ ▼ │
   │ Solution 1 Solution 2 │
   └─────────────────────────────────────────────────────────────────────────┘

   Uses LLM to evaluate promising paths. Good for planning problems.

ADVANCED REASONING TECHNIQUES
─────────────────────────────

1. Least-to-Most Prompting:
   • Decompose complex problem into sub-problems
   • Solve sub-problems in order
   • Use previous answers for later sub-problems

2. Program-Aided Language (PAL):
   • Generate code instead of natural language reasoning
   • Execute code to get answer
   • More reliable for math/logic

3. ReAct (Reasoning + Acting):
   • Interleave reasoning with tool use
   • Thought → Action → Observation → Thought...
   • Foundation for agents

4. Reflection/Self-Critique:
   • Generate answer
   • Critique the answer
   • Revise based on critique
   • Repeat

WHEN COT HELPS
──────────────
✓ Math word problems
✓ Multi-step logic
✓ Commonsense reasoning
✓ Code generation
✓ Planning tasks

WHEN COT DOESN'T HELP
─────────────────────
✗ Simple factual recall
✗ Tasks where direct answer is obvious
✗ Very small models (need sufficient capability)
✗ When speed matters (generates more tokens)

COT IN TRAINING
───────────────
Modern models are trained with CoT:
• Include reasoning traces in training data
• Reward models score reasoning quality, not just final answer
• "Process supervision" vs "outcome supervision"

This is why ChatGPT-era models are better at reasoning than GPT-3.

IMPLEMENTATION TIP
──────────────────
For production systems, often use:

1. System prompt: "Think through problems step by step before answering."
2. Structured output: Have model output {"reasoning": "...", "answer": "..."}
3. Hide reasoning: Show user only the answer (CoT happens internally)

KEY PAPERS
──────────
• "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (2022)
• "Self-Consistency Improves Chain of Thought Reasoning" (2022)
• "Tree of Thoughts: Deliberate Problem Solving with LLMs" (2023)
• "ReAct: Synergizing Reasoning and Acting in Language Models" (2022)

################################################################################

SUMMARY: YOUR LEARNING PATH

################################################################################

You've now covered the theory for all Tier 1 topics. Here's a suggested
practical progression:

COMPLETED (from nanoGPT + Dolly):
✓ Transformer fundamentals
✓ SFT (Supervised Fine-Tuning)
✓ BPE Tokenization

RECOMMENDED NEXT IMPLEMENTATIONS:

1. LoRA Fine-tuning
   • Use HuggingFace PEFT library
   • Fine-tune Llama 7B on your dataset
   • Experiment with rank, alpha, target modules

2. DPO Alignment
   • Use TRL library
   • Create preference dataset from your SFT model outputs
   • Train DPO on top of your SFT model

3. RAG System
   • Use LangChain or LlamaIndex
   • Build document Q&A over your own documents
   • Experiment with chunking, embedding models, reranking

4. Quantization
   • Quantize your fine-tuned model with AutoGPTQ or bitsandbytes
   • Compare inference speed and quality
   • Try different bit widths (8, 4)

5. Architecture Study
   • Read Llama source code
   • Implement RoPE from scratch
   • Implement GQA attention

RESOURCES:
• HuggingFace Transformers: https://github.com/huggingface/transformers
• TRL (Training with RL): https://github.com/huggingface/trl
• PEFT (LoRA etc): https://github.com/huggingface/peft
• LangChain: https://github.com/langchain-ai/langchain
• vLLM: https://github.com/vllm-project/vllm