LLM Basics Series 3: Transformer Attention Intuition

RNN: sequential computation dependency. In an RNN, each hidden state is a function of the previous one:

The GPU cannot start computing $h_{3}$ until $h_{2}$ is a finished number, because $h_{3}$ takes $h_{2}$ as input. This is a hard sequential dependency in the computation itself, not just in the data. Even during training when the full input sequence is known, the hidden states must be computed one after another.

Transformer: parallel across all positions. Self-attention computes the relationship between every pair of tokens in one matrix operation. The queries, keys, and values are all derived from the input sequence at once:

The attention score matrix $QK^\top$ has shape $(T \times T)$, where entry $(i, j)$ is how much token $i$ attends to token $j$. All $T^2$ entries are computed simultaneously in a single matrix multiply.

Two types of attention masks. There are two settings:

• Bidirectional (encoder, BERT-style). The full $T \times T$ matrix. Every token attends to every other token. No masking.
• Causal/autoregressive (decoder, GPT-style). A lower-triangular mask. Row $i$ only attends to tokens $1, \ldots, i$. Future positions are set to $-\infty$ before softmax, zeroing them out. This is what LLMs use.

In practice, the causal mask is implemented as an additive mask $M$ applied to the attention logits before softmax:

where $M$ has 0 at allowed positions and $-\infty$ at blocked positions:

Adding $-\infty$ makes $e^{-\infty} = 0$ inside softmax, so those positions receive exactly zero attention weight. This is more numerically stable than multiplying by a binary 0/1 mask after softmax, because it guarantees the attention weights over allowed positions still sum to 1.

The causal mask does not make computation sequential. Row 3 of the attention matrix uses keys from tokens 1, 2, 3. But it does not depend on the result of row 1 or row 2. All rows read from the same input vectors $k_{1}, k_{2}, \ldots, k_{T}$, which are all available from the start. The mask restricts what each row sees, but all rows are computed in one operation:

• Row 1: $[q_{1} \cdot k_{1}]$
• Row 2: $[q_{2} \cdot k_{1}, \; q_{2} \cdot k_{2}]$
• Row 3: $[q_{3} \cdot k_{1}, \; q_{3} \cdot k_{2}, \; q_{3} \cdot k_{3}]$
• Row 4: $[q_{4} \cdot k_{1}, \; q_{4} \cdot k_{2}, \; q_{4} \cdot k_{3}, \; q_{4} \cdot k_{4}]$

All four rows are computed simultaneously. The causal mask is a data restriction (cannot peek at the future), not a computation dependency (must wait for a previous result).

Contrast during training.

• RNN training: still sequential. $h_{3}$ literally cannot be computed until $h_{2}$ is finished, because $h_{3} = f(x_{3}, h_{2})$.
• Transformer training: the entire $T \times T$ masked attention matrix is one batched matrix multiply. The full input sequence is known from the training data, so all positions are processed in parallel.

What about inference? During generation (inference), GPT-style models do produce tokens one at a time, because each new token depends on the model's own previous output. That part is inherently sequential. But training is fully parallel, which is why transformers are so much faster to train on long sequences and scale so well with GPU hardware.

End of expanded note.

The root cause in RNNs. An RNN applies the same transformation at every time step:

During backpropagation through time, the gradient from step $T$ back to step 0 involves a product of Jacobians:

Each factor depends on $W$. If the spectral norm of $W$ is consistently less than 1, this product shrinks exponentially (vanishing gradients). If greater than 1, it grows exponentially (exploding gradients). The critical point: sequence length = depth of the computation graph. A 1,000-token sequence means multiplying through 1,000 Jacobians.

LSTMs partially fix this. The LSTM cell state provides an additive path:

When the forget gate $f_{t} \approx 1$ and the input gate $i_{t} \approx 0$, the cell state passes through nearly unchanged, so gradients can survive longer chains. But the gates are themselves functions of $W$ and $h_{t-1}$, so the gradient path still involves multiplicative factors at every time step. For very long sequences, the product of gate values can still decay or grow, and the model must learn to hold gates open for exactly the right duration. LSTMs reduce the problem but do not eliminate it.

What transformers change. Transformers do not propagate information sequentially across time. The entire sequence is processed in parallel through self-attention. The computation graph depth equals the number of layers (e.g., 12, 24, 96), not the sequence length (which could be 1,000 or 100,000 tokens).

The gradient path length is bounded and fixed, regardless of how long the input is. A 24-layer transformer processing a 10,000-token sequence still only backpropagates through 24 layers. An RNN processing the same sequence would backpropagate through 10,000 steps.

But fixed depth alone is not enough. A plain 24-layer network without skip connections can still have vanishing or exploding gradients (the same issue from MLP training in Series 2). Transformers stay stable because of three additional design choices working together:

(1) Residual connections. Each sub-layer computes

The gradient through a residual block is $I + \frac{\partial \text{SubLayer}}{\partial x_{l}}$, where the identity $I$ provides a direct path for gradients to flow unchanged. Even if the sub-layer gradient is small, the identity term prevents vanishing. This is the same mechanism discussed in the MLP initialization section of Series 2.

(2) Layer normalization. Applied before or after each sub-layer, LayerNorm keeps activations well-scaled and prevents values from drifting to extreme ranges where gradients saturate.

(3) Attention as a short path. In an RNN, information from token 1 must pass through every intermediate hidden state to reach token 1,000. In a transformer, self-attention connects every token pair directly in a single layer. The gradient from any output token to any input token passes through at most $L$ layers (the network depth), regardless of how far apart they are in the sequence. There is no chain of $T$ sequential multiplications.

Summary. The accurate statement is: transformers avoid the vanishing/exploding gradient problem because (1) computation depth is fixed and independent of sequence length (no sequential recurrence), (2) residual connections provide direct gradient paths through every layer, and (3) layer normalization keeps activations stable. It is not simply that "the depth is pre-specified"; it is that the depth does not grow with input length, and the architecture includes mechanisms that keep gradients well-behaved across the fixed number of layers.

End of expanded note.

The intuition comes from binary counting. In binary, the rightmost bit flips every step, the next bit flips every 2 steps, the next every 4, and so on. Each digit oscillates at a different frequency, and the combination uniquely identifies every number. Sinusoidal positional encoding does the same thing with smooth waves instead of binary flips: dimension pair $i = 0$ oscillates fastest (like the least significant bit), and later pairs oscillate progressively slower (like more significant bits).

Why both sin and cos. Using both functions for each frequency allows the model to compute relative positions via a linear transformation. The trigonometric addition formulas give:

For any fixed offset $k$, there exists a linear transformation (a rotation matrix) that maps $P_{t}$ to $P_{t+k}$. The model can learn this linear map, which means it can learn "the token $k$ positions to the left" using a simple matrix multiply. If only sine were used without cosine, the model would need nonlinear operations to recover relative positions.

This rotation matrix is the same 2D rotation matrix that RoPE uses. Both methods share the same mathematical foundation.

End of expanded note.

Where RoPE operates. Unlike sinusoidal and learned encodings which are added at the $d_{model}$ level before attention, RoPE is applied inside the attention mechanism, after the Q and K projections have already split the $d_{model}$-dimensional vector into $H$ heads of $d_{k}$ dimensions each:

1. Embedding: $(T, d_{model})$. Sinusoidal or learned PE would be added here.
2. Q and K projections: $(T, d_{model}) \to H$ heads of $(T, d_{k})$ each, where $d_{k} = d_{model}/H$.
3. RoPE is applied here, to each head's $(T, d_{k})$ query and key vectors independently.

So RoPE operates at the per-head dimension $d_{k}$, not $d_{model}$.

The three quantities in $\theta = t \cdot \omega_{i}$:

• $t$ is the position index of the token in the sequence. $t = 0$ for the first token, $t = 1$ for the second, and so on. Tokens further in the sequence get rotated by a larger angle.
• $i$ is the dimension pair index within one token's vector. $i = 0$ for dimensions $(0, 1)$, $i = 1$ for dimensions $(2, 3)$, etc. A 2D rotation takes exactly 2 numbers as input, so the $d_{k}$ dimensions are grouped into $d_{k}/2$ pairs.
• $\omega_{i} = 1/10000^{2i/d_{k}}$ is the frequency for pair $i$. Pair $i = 0$ has $\omega_{0} = 1$ (fastest, encodes fine-grained local position). The last pair has $\omega \approx 1/10000$ (slowest, encodes coarse long-range position). Position $t$ determines how much to rotate; frequency $\omega_{i}$ determines how sensitive that pair is to position changes.

Worked example. Consider $T = 3$ tokens ("I", "like", "cats") with $d_{model} = 8$ and $H = 2$ heads, so $d_{k} = 4$ per head.

Start from the token embeddings $X \in \mathbb{R}^{3 \times 8}$. The full Q projection gives $Q_{\text{full}} = XW_{Q} \in \mathbb{R}^{3 \times 8}$. Split the 8 columns into $H = 2$ heads of $d_{k} = 4$:

Q_full:
         [  head 0 (cols 0-3)  |  head 1 (cols 4-7)  ]
"I":     [  q00, q01, q02, q03 |  q04, q05, q06, q07 ]
"like":  [  q10, q11, q12, q13 |  q14, q15, q16, q17 ]
"cats":  [  q20, q21, q22, q23 |  q24, q25, q26, q27 ]

For head 0, the query matrix is $(3, 4)$. RoPE groups the $d_{k} = 4$ dimensions into $d_{k}/2 = 2$ pairs: dimensions $(0, 1)$ form pair 0, and dimensions $(2, 3)$ form pair 1:

            pair i=0       pair i=1
            dim 0, dim 1   dim 2, dim 3
t=0  "I":   (q00, q01)     (q02, q03)
t=1  "like":(q10, q11)     (q12, q13)
t=2  "cats":(q20, q21)     (q22, q23)

RoPE rotates each pair by angle $t \times \omega_{i}$:

• Pair $i=0$: frequency $\omega_{0} = 1$ (fast). Token "cats" at $t=2$ gets rotated by $2 \times 1 = 2$ radians.
• Pair $i=1$: frequency $\omega_{1} = 1/10000^{2/4} = 0.01$ (slow). Token "cats" at $t=2$ gets rotated by $2 \times 0.01 = 0.02$ radians.

Early pairs capture fine-grained position differences; later pairs capture coarse, long-range patterns. In Qwen3-8B with $d_{k} = 128$, there are 64 pairs spanning frequencies from fast to slow.

End of expanded note.

Write the rotated query and key as $R_{t} q_{t}$ and $R_{s} k_{s}$, where $R_{t}$ is the block-diagonal rotation matrix for position $t$. The attention score between tokens at positions $t$ and $s$ is:

The last step uses the fact that rotation matrices satisfy $R_{t}^\top R_{s} = R_{s-t}$. The absolute positions $t$ and $s$ disappear, and only the gap $s - t$ remains. The model does not care whether "cat sat" appears at positions (3, 4) or (500, 501); it sees the same relative distance of 1 in both cases.

Why this matters:

• Relative position awareness. The model naturally learns distance-based patterns rather than memorizing absolute position slots.
• Length generalization. Since attention scores depend on relative distance, a model trained on 4K-length sequences can potentially generalize to 8K or longer at inference time.
• No extra parameters. RoPE is deterministic with no learned parameters.
• Applied to Q and K only. The value vectors $V$ are not rotated, because position information only needs to affect which tokens attend to each other, not the content that gets aggregated.

End of expanded note.

The default base value of 10,000 in $\omega_{i} = 1/\text{base}^{2i/d_{k}}$ was set in the original RoPE paper and works well for models trained on context lengths up to about 4K-8K tokens. For longer contexts, the base often needs adjustment.

Why the base matters. A larger base stretches out the rotation frequencies, making all dimension pairs rotate more slowly per position step. This prevents the high-frequency pairs from "wrapping around" (completing full rotations within the context window), which causes the model to confuse distant positions. If you train at 4K but want to infer at 32K, the default base produces rotation angles the model never saw during training, and attention patterns degrade.

Common approaches to context extension:

• Linear scaling (position interpolation). Divide the position index by a factor $s$ so that position 32K maps to the same angle that position 4K would have at the original base. Simple but compresses all positions, reducing the model's ability to distinguish nearby tokens at long contexts.
• NTK-aware scaling. Instead of scaling positions, increase the base: $\text{base}' = \text{base} \times s^{d_{k}/(d_{k} - 2)}$. This spreads the frequency spectrum without compressing any single dimension pair. A model trained at 4K with base 10,000 and extended to 32K ($s = 8$) might use a base around 500,000. This is what LLaMA-based long-context models commonly adopt.
• YaRN (Yet another RoPE extensioN). Applies different scaling strategies to different frequency bands: high-frequency pairs (which capture local position) are left unchanged, and low-frequency pairs (which capture global position) are interpolated. This preserves local attention sharpness while extending the effective context. YaRN also adds a temperature correction to the attention logits to compensate for the changed dot-product magnitudes.

Practical defaults. If you are fine-tuning an existing model for longer context, start with NTK-aware scaling (multiply the base by the extension ratio raised to $d_{k}/(d_{k}-2)$) and fine-tune on a small amount of long-context data (a few hundred steps is often sufficient). If training from scratch with a target context length $T$, many recent models (Qwen3, LLaMA 3.1) simply train with a large base from the start (e.g., 1,000,000) and use progressive context extension during training (start at 4K, then increase to 32K, then 128K). Changing the base without any fine-tuning gives partial improvements but usually requires at least light continued training for best results.

End of expanded note.

The problem with raw embeddings. The simplest attention would compute similarity between token embeddings directly ($XX^\top$), then use that to mix them. But this forces each token's vector to serve two conflicting roles simultaneously: (1) describe what the token is looking for, and (2) describe what the token offers to others. These are different things. When you read "The cat sat on the mat", the word "sat" needs to look for a subject (who sat?) but offer an action (what happened?). One vector cannot optimally do both.

The database analogy. Think of a key-value database. You search with a query, the system compares it against all keys to find relevant entries, and returns the matching values. In self-attention:

• Query ($Q$): what this token is looking for. ("I need a subject.")
• Key ($K$): what this token advertises to others. ("I am a noun/agent.")
• Value ($V$): the actual content to pass along if selected. ("Here is my semantic representation.")

Each is a different learned linear projection of the same embedding, so the same token produces three different vectors serving three different roles:

$W_{Q}$ learns to extract "what am I looking for?", $W_{K}$ learns to extract "what should I advertise?", and $W_{V}$ learns to extract "what content should I pass along?"

Example 1: "The cat sat on the mat." Consider "sat" building its contextualized representation. Without Q/K/V separation, "sat" has one vector that must simultaneously be similar to "cat" (to find the subject) and "mat" (to find the location), but "cat" and "mat" are very different tokens. With Q/K/V separation:

• "sat" produces a query $q_{\text{sat}}$ encoding "I need a noun/subject."
• "cat" produces a key $k_{\text{cat}}$ encoding "I am a noun/agent" and a value $v_{\text{cat}}$ carrying the semantic content of "cat."
• "the" produces a key $k_{\text{the}}$ encoding "I am a determiner."

The score $q_{\text{sat}} \cdot k_{\text{cat}}$ is high, so the output for "sat" receives a large weight of $v_{\text{cat}}$. Now the representation of "sat" carries information about its subject.

Example 2: "She gave him the book because he asked for it." The pronoun "it" needs to resolve what it refers to. Its query encodes "I need a concrete noun/object." The keys for "book", "she", "him" all advertise their roles. The key for "book" (concrete object) matches the query for "it" most strongly, so "it" receives the value of "book" and its output representation carries the meaning of what "it" refers to. Meanwhile "he" might produce a query encoding "I need a person/subject", which matches the key of "she" or "him" depending on the context. Different tokens ask different questions through their queries, and the same token can be a good answer for one query but not another.

What if you removed Q/K separation? Using $XX^\top$ for scores makes attention symmetric: "cat" attending to "sat" would have the same score as "sat" attending to "cat". But these should be different. "sat" looks for a subject (high attention to "cat"), while "cat" looks for a predicate (high attention to "sat" for a different reason). Separate $W_{Q}$ and $W_{K}$ projections break this symmetry, allowing asymmetric attention patterns.

What if you removed V? If you use $K$ as $V$ (i.e., the output is a weighted sum of key vectors), then the information used for matching is the same information passed along. Keys and values encode fundamentally different things:

• Keys encode a token's role: what kind of token it is (noun, verb, subject, object, animate, etc.), so that queries can find it. Tokens with similar roles should have similar keys.
• Values encode a token's semantic meaning: what the word actually means (cat = furry animal; mat = flat surface). Tokens with the same role can have completely different meanings.

The separation matters because role similarity and meaning similarity are independent. Consider "The cat sat on the mat." Both "cat" and "mat" are nouns, so they have similar keys (both match a "looking for a noun" query). But they mean entirely different things, so they need different values.

With separate V, "sat" attends to "cat" with weight 0.7 and "mat" with weight 0.2 (via $QK^\top$), and the output is:

The output carries mostly "cat" meaning. The model knows which noun "sat" relates to.

Without V ($K$ used as $V$), the output is a weighted sum of keys:

You get a blurry mix of roles, not meanings. The model knows "sat" relates to something that is mostly a subject-noun, but it has lost which noun, because both "cat" and "mat" advertised similar role information as nouns. The sentences "the cat sat" and "the mat sat" would produce nearly identical outputs for "sat."

The core principle: role similarity (both nouns, similar keys) is exactly what makes matching work, but it is also what would make the output indistinguishable if keys were reused as values. Separate $V$ breaks this by allowing "similar keys, different values" to coexist. The matching function ($QK^\top$) and the content function ($V$) are fully decoupled.

End of expanded note.

The formulas above show a single sequence for clarity, but in practice a batch of $N$ sequences is processed together. All tensors gain a leading batch dimension:

• Embedding lookup: $N$ sequences of $T$ token IDs $\to (N, T, d_{model})$.
• Q, K, V projections: the matrix multiply broadcasts over $N$ and $T$. $(N, T, d_{model}) \times (d_{model}, d_{k}) \to (N, T, d_{k})$ per head.
• Attention scores: $QK^\top$ is computed per sample. $(N, T, d_{k}) \times (N, d_{k}, T) \to (N, T, T)$. Each sample gets its own $(T \times T)$ attention matrix. Sample 1's tokens never attend to sample 2's tokens.
• Causal mask: the same $(T \times T)$ mask $M$ is broadcast identically across all $N$ samples.
• FFN: applied independently to each token in each sample. $(N, T, d_{model}) \to (N, T, d_{ffn}) \to (N, T, d_{model})$.
• LayerNorm: normalizes across $d_{model}$ for each token in each sample independently. It does not mix across the batch or across positions.

With multi-head attention, the full working shape is $(N, H, T, d_{k})$: batch, heads, sequence length, per-head dimension. All four dimensions are handled in parallel by the GPU.

The training loss is averaged over both positions and the batch:

The batch dimension is purely for GPU parallelism and training efficiency. Each sample's attention and gradient are independent.

End of expanded note.

Both normalize activations to stabilize training, but they normalize across different dimensions.

BatchNorm normalizes across the batch dimension. For a batch of $N$ samples, each with $d$ features, it computes the mean and variance of each feature across all samples in the batch:

Feature $j$ is normalized using statistics from all $N$ samples. This works well when the batch is large enough that the per-feature statistics are stable.

LayerNorm normalizes across the feature dimension. In a transformer, this means per-token normalization. Each token's $d_{model}$-dimensional vector is normalized independently using its own mean and variance:

For a batch with shape $(N, T, d_{model})$, there are $N \times T$ independent normalizations, each over $d_{model}$ values. No information crosses between tokens, and no information crosses between samples:

Sample 1, Token 0: [0.5, -1.2, 0.8, ..., -0.3]  ← normalize these d_model values
Sample 1, Token 1: [1.1,  0.3, 2.1, ...,  0.7]  ← normalize these d_model values
Sample 1, Token 2: [0.2, -0.5, 0.1, ...,  1.4]  ← normalize these d_model values
...
Sample N, Token T: [...]                           ← normalize these d_model values

Each row is standardized to mean 0 and variance 1 on its own.

Visually comparing the two, for a tensor of shape $(N, T, d_{model})$:

BatchNorm: normalizes across N (and T) for each feature dimension
           ↓ down the batch axis
           [sample 1, token 1, feature j]
           [sample 2, token 1, feature j]  ← compute mean/var of feature j
           [sample 3, token 1, feature j]
           ...

LayerNorm: normalizes across d_model for each token in each sample
           → across the feature axis
           [sample 1, token 1, feature 0, feature 1, ..., feature d]
                                ← compute mean/var across all features

Why transformers use LayerNorm:

• Variable sequence lengths. In a batch of sequences, different samples may have different lengths (padded to the same $T$). BatchNorm would compute statistics mixing real tokens with padding, corrupting the normalization. LayerNorm operates per token independently, so padding in other samples does not matter.
• Small or variable batch sizes. BatchNorm needs a reasonably large batch for stable statistics. During inference (batch size = 1) or with small batches, the estimates become noisy. LayerNorm works identically regardless of batch size, because it only looks within one sample.
• Autoregressive generation. During token-by-token generation, each new token must be normalized. LayerNorm can normalize one token using its own $d_{model}$ features. BatchNorm would need statistics from a batch of sequences at the same position, which is not naturally available during generation.
• No running statistics needed. BatchNorm maintains running mean/variance for inference (computed during training). LayerNorm computes everything on the fly from the current input, making it simpler and avoiding train/inference mismatch.

Both methods also learn a per-feature scale $\gamma$ and shift $\beta$ parameter after normalization: $y_{j} = \gamma_{j} \hat{x}_{j} + \beta_{j}$. These let the model undo the normalization if needed.

When to use which:

• BatchNorm: standard MLPs and CNNs with large, fixed-size batches (image classification, tabular data). As discussed in Series 2.
• LayerNorm: transformers, RNNs, and any setting with variable-length sequences or small batches. All modern LLMs use LayerNorm.

End of expanded note.

The autoregressive formulation. A decoder-only LLM models the probability of a sequence by factoring it into a chain of next-token predictions:

At each position $t$, the model predicts the next token using only the tokens before it. This is enforced by the causal attention mask.

How the causal mask creates $T$ training examples from one sequence. During training, the full sequence is known. A key question is: does the model run $T$ separate forward passes, one for each prefix? No. The causal mask achieves the same effect in a single forward pass.

Consider the input sequence [The, cat, sat, on, the, mat] with $T = 6$ tokens. The model runs one forward pass through all $L$ transformer blocks. At every attention layer, the $QK^\top$ score matrix has shape $(6, 6)$. The causal mask $M$ is added before softmax:

After softmax, the attention weight matrix looks like (schematically, with $w_{ij}$ denoting the attention weight from token $i$ to token $j$):

          The   cat   sat   on    the   mat
The   [  1.0    0     0     0     0     0  ]
cat   [  0.3   0.7    0     0     0     0  ]
sat   [  0.1   0.5   0.4    0     0     0  ]
on    [  0.05  0.15  0.6   0.2    0     0  ]
the   [  0.02  0.08  0.1   0.3   0.5    0  ]
mat   [  0.01  0.04  0.05  0.1   0.3   0.5]

Each row sums to 1 over its allowed positions. The zeros above the diagonal come from the $-\infty$ mask making $e^{-\infty} = 0$ in softmax. The critical property: row $t$'s output depends only on tokens $0, \ldots, t$, even though all rows are computed in the same matrix multiply. Row 2 ("sat") is computed using only "The", "cat", "sat", exactly as if the model only saw those three tokens.

After all $L$ transformer blocks, the model produces an output vector $h_{t} \in \mathbb{R}^{d_{model}}$ for each position $t$. Each $h_{t}$ was built using only tokens $0, \ldots, t$ (the mask was applied at every layer). Now each position independently predicts the next token:

Position 0: h_0 was built from [The]                    → predicts "cat"
Position 1: h_1 was built from [The, cat]               → predicts "sat"
Position 2: h_2 was built from [The, cat, sat]          → predicts "on"
Position 3: h_3 was built from [The, cat, sat, on]      → predicts "the"
Position 4: h_4 was built from [The, cat, sat, on, the] → predicts "mat"

One forward pass, $T$ predictions. The model computes attention once with the $(T \times T)$ masked matrix, passes through all blocks once, and produces $T$ output vectors. Each output vector is equivalent to what you would get if you ran the model separately on that prefix. The causal mask guarantees this equivalence because information cannot leak from future positions.

The output head and loss. After the final transformer block, each position's output is projected to vocabulary size $V$:

The training loss is the average cross-entropy across all $T$ positions:

So one sequence of $T$ tokens produces $T$ training signals (one per position), all from a single forward pass. This is extremely efficient compared to running $T$ separate forward passes for each prefix.

The output projection $W_{\text{head}} \in \mathbb{R}^{d_{model} \times V}$ often shares weights with the token embedding table $W_{e} \in \mathbb{R}^{V \times d_{model}}$ (called weight tying: $W_{\text{head}} = W_{e}^\top$). This reduces parameters and ties the input/output representations together.

Inference (generation). At inference time, the model generates tokens one at a time. Starting from a prompt, it runs a forward pass, takes the logits at the last position, samples or picks the highest-probability token, appends it to the sequence, and repeats. Each step is autoregressive: the output of step $t$ becomes part of the input for step $t+1$. This is inherently sequential, unlike training where all positions are computed in parallel.

End of expanded note.

Bidirectional attention. In an encoder-only model like BERT, there is no causal mask. Every token attends to every other token, including tokens that come after it in the sequence. The full $T \times T$ attention matrix is used with no entries masked out.

This means each token's output representation is informed by the entire sequence in both directions. For a sentence like "The bank by the river", the word "bank" can attend to "river" to disambiguate its meaning (riverbank, not financial bank), even though "river" appears later.

Training objective: masked language modeling (MLM). Since the model sees all tokens, it cannot be trained to predict the next token (it would just look ahead and copy the answer). Instead, BERT randomly masks ~15% of input tokens (replacing them with a [MASK] token) and trains the model to predict the original token at those masked positions:

Input:  [The, cat, [MASK], on, the, [MASK]]
Target:  predict "sat" at position 2, "mat" at position 5

The model must use the full bidirectional context to fill in the blanks.

Use cases. Encoder-only models produce a rich contextual representation for each token, making them well-suited for tasks where you need to understand the full input: text classification, named entity recognition, semantic similarity, and extractive question answering. They are not designed for open-ended text generation.

End of expanded note.

Two stacks. An encoder-decoder model has two separate stacks of transformer blocks. The encoder processes the input sequence (e.g., a French sentence) with bidirectional attention. The decoder generates the output sequence (e.g., the English translation) autoregressively with causal attention.

Cross-attention. The decoder has an additional attention layer that connects it to the encoder. In each decoder block, the structure is:

1. Causal self-attention (decoder attends to its own previous tokens).
2. Cross-attention (decoder attends to the encoder's output).
3. FFN.

In cross-attention, the queries come from the decoder and the keys and values come from the encoder:

This lets each decoder token ask: "Which parts of the input are relevant for generating my next word?" For translation, a decoder token generating the English word "cat" would attend strongly to the French encoder tokens for "chat."

No causal mask on cross-attention. The decoder can attend to all encoder positions (the full input is known). The causal mask only applies to the decoder's self-attention (cannot peek at future output tokens).

Use cases. Translation, summarization, speech-to-text, and any task where the input and output are different sequences. The original "Attention Is All You Need" transformer was an encoder-decoder model for machine translation. T5 and BART are well-known encoder-decoder models.

Why decoder-only won for LLMs. Encoder-decoder requires a clear separation of "input" and "output", which fits translation but not open-ended conversation. Decoder-only models treat everything as one continuous sequence (prompt + response), which is simpler and more flexible. At sufficient scale, decoder-only models match or exceed encoder-decoder performance on most tasks, so the added complexity of two stacks is rarely justified for general-purpose LLMs.

End of expanded note.