TL;DR — Self-attention lets every token look at every other token and decide, per token, which ones matter. Each token emits a query (what it's looking for), a key (what it offers), and a value (what it contributes); scores are query·key, softmaxed into weights, and the output is the weighted sum of values. In the worked example below, the pronoun "it" attends to "animal" 25.7× more strongly than to "street" — the grammatically correct referent, not the nearer noun — and nothing about that is a rule. Change one input vector and the resolution moves, because it is learned geometry. The cost is the whole story of modern LLM engineering: every token compares with every other, so attention is quadratic — 10,000 tokens is 100 million comparisons, which is why long contexts are expensive and why so much research targets exactly this term. It stops explaining things when the question is why scale produces capability; the mechanism is legible, the emergence is not.
1. Simple explanation
Older sequence models read left to right, carrying a summary forward. That summary is a bottleneck: by word fifty, word three is a faint smudge. Long-range connections — a pronoun and the noun it refers to, a subject and its verb across a clause — get lost.
Attention removes the bottleneck by letting every position look directly at every other position. No relaying, no forgetting. When the model processes "it", it can look straight at "animal" eleven words back, with no degradation from distance.
The clever part is that each token decides for itself what to look at. It broadcasts a description of what it needs, every other token advertises what it has, and the match strength becomes a weight. Tokens that match get attended to; tokens that don't get ignored.
Analogy — a room of specialists rather than a game of telephone. In telephone, a message passes person to person and degrades. In a room, you say what you need — "who knows about the animal?" — everyone signals whether they can help, and you listen in proportion to how well they match. Distance across the room is irrelevant. And the useful part of the analogy is the cost: for everyone to hear everyone, you need every pair to interact, which is why the room stops working when it gets very large.
2. Diagram
EVERY TOKEN PRODUCES THREE VECTORS
token "it" ──┬── W_Q ──▶ QUERY "I need something animate, in a state"
├── W_K ──▶ KEY "here is what I offer"
└── W_V ──▶ VALUE "here is what I contribute if chosen"
the projections W_Q, W_K, W_V are LEARNED. That is the whole training.
ONE ATTENTION STEP, for the query "it"
query("it") · key(token) softmax weighted sum
↓ ↓ ↓
the 0.4 ──┐ 0.001 ┐
animal 9.1 ──┤ 0.569 ████████ │ output for "it"
cross 2.2 ──┤ scale by 0.013 ├─ = Σ weight × value
street 3.0 ──┤ √d, then 0.022 ▌ │
it 5.6 ──┤ softmax 0.134 ██ │
tired 7.8 ──┘ 0.235 ███ ┘
'animal' 0.569 vs 'street' 0.022 -> 25.7x
▲
the correct referent, not the nearer noun.
No grammar rule was written. It is geometry.
WHY IT COSTS WHAT IT COSTS
every token compares with every token:
12 tokens -> 144 comparisons
100 -> 10,000
1,000 -> 1,000,000
10,000 -> 100,000,000 ← quadratic
This single fact drives context pricing, latency,
and most architecture research since 2017.
3. How it works
3.1 Query, key, value
Each token's embedding is multiplied by three learned matrices to produce three vectors:
- Query — what this token is looking for.
- Key — what this token offers to others looking.
- Value — what this token contributes when it is attended to.
Splitting "what I'm looking for" from "what I offer" is what makes the mechanism asymmetric and useful: "it" can search for an animate noun without itself being one.
The database metaphor is close enough to be worth stating: you issue a query, match it against keys, and retrieve values — except the match is soft, so you get a weighted blend of every value rather than one row.
3.2 Scores, scaling, softmax
score(q, k) = q · k / sqrt(d)
weights = softmax(all scores)
output = Σ weight_i × value_i
The dot product measures alignment. The √d divisor matters more than it looks: without it, dot products in high dimensions grow large, softmax saturates, and gradients vanish — the model stops learning. It's a one-symbol fix for a training-stability problem.
Softmax then turns scores into a distribution summing to 1, so "how much attention" is a proportion rather than an arbitrary magnitude.
3.3 Why the weights are computed, not looked up
The example in §4 resolves a pronoun correctly. It is worth being precise about what did not happen: there is no rule saying "pronouns refer to animate nouns", no parse tree, no coreference table.
The query for "it" happens to weight the dimensions that "animal" and "tired" score highly on, so those tokens win the softmax. Change the input vector for "it" and the winner changes to "street" — no rule was edited, because there was no rule. That's the property that makes transformers general: the same mechanism handles syntax, coreference, and topic without anyone specifying which is which.
3.4 Multiple heads, and the rest of the block
One attention pattern can only express one kind of relationship at a time. Multi-head attention runs several in parallel with different learned projections, so one head can track syntactic agreement while another tracks coreference and a third tracks topic. Their outputs are concatenated and projected back down.
A full transformer block wraps attention in three more pieces, each solving a specific problem:
| piece | what it's for |
|---|---|
| feed-forward network | per-token processing; attention mixes across positions, this transforms within one |
| residual connections | add the input back to the output, so gradients reach early layers in a deep stack |
| layer normalisation | keeps activation scales stable, which is what makes deep stacks trainable |
Stack dozens of those blocks and you have the architecture.
3.5 Position has to be added deliberately
Attention as described is permutation-invariant — shuffle the tokens and the set of query-key comparisons is unchanged. "dog bites man" and "man bites dog" would be identical.
So position is injected explicitly, either as a positional encoding added to the embeddings or as a rotation applied inside the attention computation (RoPE, in most current models). This is not a detail: how positions are represented determines how far a model can extrapolate beyond the context length it trained on.
3.6 The quadratic cost, and where the frame ends
Every token attends to every token, so compute and memory scale with the square of sequence length. Double the context and you quadruple the attention cost. That single fact explains long-context pricing, why latency grows faster than input length, and a large fraction of published architecture research — sparse attention, linear approximations, and IO-aware exact methods like FlashAttention, which doesn't change the maths but makes it dramatically cheaper in practice.
Where this stops helping: the mechanism explains what a transformer computes and says nothing about why scale produces capability. Nothing in §4 predicts that a large enough stack follows instructions or writes working code. It also describes the encoder-style bidirectional case; decoder-only models used for generation add causal masking so a token can only attend backwards. And it says nothing about how the projections come to hold useful values, which is training, not architecture.
4. The math
4.1 Scaled dot-product attention
Q = X · W_Q K = X · W_K V = X · W_V
Attention(Q, K, V) = softmax( Q · K^T / sqrt(d_k) ) · V
where Q·K^T is (n × n) <- the quadratic term
4.2 Complexity
time and memory ~ O(n² · d) n = sequence length, d = model dimension
n = 1,000 -> 1,000,000 pairwise scores
n = 10,000 -> 100,000,000
4.3 Worked example
The sentence is a classic ambiguity: "the animal did not cross the street because it was too tired." Does it mean the animal or the street? Only the animal can be tired.
Four interpretable dimensions — [animate, place, motion, state] — and projections chosen so W_Q emphasises what a token seeks while W_K advertises what it offers.
What does 'it' attend to? (attention weights, one head)
0.569 animal ██████████████████████████████████
0.235 tired ██████████████
0.134 it ████████
0.022 street █
0.013 cross
0.005 was
'animal' 0.569 vs 'street' 0.022 -> ratio 25.72x
The pronoun attends most strongly to animal, then to tired — the two tokens carrying the animate/state signal its query asks for. street, which is closer in the sentence and grammatically plausible, gets 2% of the attention.
Now change only the input vector for the pronoun, tilting it toward "place":
CHANGE ONE WORD and the answer moves -- no rule was edited:
0.497 street
0.182 it
0.087 animal
street now wins. Nothing in the code changed — no rule, no condition, no table. The resolution is a consequence of where the vectors point, which is exactly why the mechanism generalises to relationships nobody enumerated.
4.4 What it costs
tokens comparisons
12 144
100 10,000
1000 1,000,000
10000 100,000,000
5. Real code
"""Self-attention from scratch: why 'it' resolves to 'animal' and not 'street'."""
import math
SENTENCE = ["the", "animal", "did", "not", "cross", "the", "street",
"because", "it", "was", "too", "tired"]
# Hand-made 4-D embeddings so this runs anywhere. SCALE lifts them to a realistic
# norm -- real embeddings are not bounded to [0,1], and small vectors make the
# softmax nearly uniform, which hides the effect this example is about.
SCALE = 3.0
# [animate, place, motion, state]
EMB = {
"the": [0.02, 0.02, 0.02, 0.02],
"animal": [0.95, 0.05, 0.20, 0.30],
"did": [0.05, 0.02, 0.40, 0.05],
"not": [0.02, 0.02, 0.10, 0.10],
"cross": [0.10, 0.35, 0.92, 0.05],
"street": [0.03, 0.94, 0.25, 0.05],
"because": [0.02, 0.02, 0.02, 0.15],
"it": [0.50, 0.45, 0.10, 0.35], # ambiguous by itself
"was": [0.05, 0.05, 0.05, 0.30],
"too": [0.02, 0.02, 0.02, 0.25],
"tired": [0.60, 0.02, 0.05, 0.95], # a STATE only an animate thing has
}
EMB = {w: [x * SCALE for x in v] for w, v in EMB.items()}
D = 4
def matmul_vec(M, v):
return [sum(row[i] * v[i] for i in range(len(v))) for row in M]
def softmax(xs):
m = max(xs)
e = [math.exp(x - m) for x in xs]
s = sum(e)
return [x / s for x in e]
# Learned projections. Real ones are trained; these are chosen so the example is
# legible -- W_q emphasises what a token is LOOKING FOR, W_k what it OFFERS.
W_Q = [[1.0, 0.0, 0.0, 0.9], # a query weights 'animate' and 'state'
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0],
[0.0, 0.0, 0.0, 1.0]]
W_K = [[1.0, 0.0, 0.0, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0],
[0.9, 0.0, 0.0, 1.0]] # a key advertises 'animate' via its state
W_V = [[1.0, 0.0, 0.0, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0],
[0.0, 0.0, 0.0, 1.0]]
def attend(query_word: str, tokens: list[str]) -> list[tuple[float, str]]:
"""One attention head: score the query against every token, then softmax."""
q = matmul_vec(W_Q, EMB[query_word])
scores = []
for t in tokens:
k = matmul_vec(W_K, EMB[t])
# scaled dot product: divide by sqrt(d) to keep the softmax from saturating
scores.append(sum(a * b for a, b in zip(q, k)) / math.sqrt(D))
weights = softmax(scores)
return sorted(zip(weights, tokens), reverse=True)
print("SENTENCE:", " ".join(SENTENCE))
print("\nWhat does 'it' attend to? (attention weights, one head)")
ranked = attend("it", SENTENCE)
for w, t in ranked[:6]:
bar = "█" * int(w * 60)
print(f" {w:.3f} {t:<9} {bar}")
top_word = ranked[0][1]
animal_w = next(w for w, t in ranked if t == "animal")
street_w = next(w for w, t in ranked if t == "street")
print(f"\n 'animal' {animal_w:.3f} vs 'street' {street_w:.3f}"
f" -> ratio {animal_w / street_w:.2f}x")
print("\nWHY: the query for 'it' asks for animate+state; 'animal' and 'tired'")
print(" advertise exactly that. 'street' scores on 'place', which the")
print(" query does not weight. Resolution is learned geometry, not a rule.")
print("\nCHANGE ONE WORD and the answer moves -- no rule was edited:")
EMB["it2"] = [x * SCALE for x in [0.20, 0.85, 0.10, 0.10]] # 'it' leaning to a place
alt = attend("it2", SENTENCE)
for w, t in alt[:3]:
print(f" {w:.3f} {t}")
print("\nCOST: attention compares EVERY token with every other token")
print(f" {'tokens':>8} {'comparisons':>13}")
for n in (12, 100, 1_000, 10_000):
print(f" {n:>8} {n * n:>13,}")
print(" -> quadratic. This is why long contexts are expensive, and why")
print(" so much research targets exactly this term.")
# 'it' resolves to the animate referent, not the nearer noun 'street'.
assert animal_w > street_w
assert top_word in ("animal", "tired", "it")
# The weights are a probability distribution over the sequence.
assert abs(sum(w for w, _ in ranked) - 1.0) < 1e-9
# Every token is compared with every other -- the quadratic term.
assert len(ranked) == len(SENTENCE)
print("\nall assertions passed")
# Output:
# SENTENCE: the animal did not cross the street because it was too tired
#
# What does 'it' attend to? (attention weights, one head)
# 0.569 animal ██████████████████████████████████
# 0.235 tired ██████████████
# 0.134 it ████████
# 0.022 street █
# 0.013 cross
# 0.005 was
#
# 'animal' 0.569 vs 'street' 0.022 -> ratio 25.72x
#
# WHY: the query for 'it' asks for animate+state; 'animal' and 'tired'
# advertise exactly that. 'street' scores on 'place', which the
# query does not weight. Resolution is learned geometry, not a rule.
#
# CHANGE ONE WORD and the answer moves -- no rule was edited:
# 0.497 street
# 0.182 it
# 0.087 animal
#
# COST: attention compares EVERY token with every other token
# tokens comparisons
# 12 144
# 100 10,000
# 1000 1,000,000
# 10000 100,000,000
# -> quadratic. This is why long contexts are expensive, and why
# so much research targets exactly this term.
#
# all assertions passed
The projections are hand-chosen so the example is readable; in a real model they are learned, and the dimensions correspond to nothing nameable. The mechanism — project to Q/K/V, scaled dot product, softmax, weighted sum — is exactly what runs in production.
6. Real-world example
A team fine-tuned a model on documents averaging 800 tokens and it worked well. They then ingested a new source averaging 6,000 tokens, expecting roughly 7× the cost per document.
The bill went up far more than 7×, and latency went up worse. Nothing had changed except input length.
The cause is §3.6. The attention term scales with the square of sequence length, so 7.5× the tokens is about 56× the attention computation. The feed-forward layers scale linearly and the rest of the pipeline barely moved, so the total didn't rise by the full 56× — but it rose enough to break a forecast built on a per-token assumption.
Two things had hidden it. Their cost model was linear in tokens, which is a good approximation until it isn't. And their tests used short documents exclusively, so the non-linearity never appeared before production.
The fix was mostly not architectural: chunk the long documents and process them independently, which converts one quadratic cost into several small ones — n chunks of length n/k cost roughly k × (n/k)² = n²/k, a factor-of-k saving. They also moved to a serving stack with a memory-efficient attention implementation, which cut the constant substantially without changing the asymptotics.
The general lesson: "cost per token" is a useful shorthand that quietly assumes linearity. Anywhere sequence length varies by an order of magnitude, model the quadratic term explicitly.
7. Interview questions companies actually ask
Q1. What problem does attention solve? The bottleneck in sequential models, where information passes position to position and degrades with distance. Attention lets every position read every other position directly, so a pronoun can reference a noun fifty tokens back with no loss from distance. It also makes the computation parallel across positions, which is what made large-scale training practical.
Q2. What are query, key, and value? Three learned projections of each token's embedding. The query is what the token is looking for, the key is what it offers to others, the value is what it contributes when attended to. The split matters because it makes the relationship asymmetric — a pronoun can search for an animate noun without being one.
Q3. Why divide by √d? Because dot products grow with dimension, and large scores saturate the softmax so that one weight approaches 1 and the rest approach 0. When that happens the gradients vanish and the model stops learning. The scaling keeps scores in a range where softmax stays informative — a one-symbol fix for a training-stability problem.
Q4. Why multiple heads? One attention pattern expresses one kind of relationship. Multiple heads with different learned projections can attend to different things simultaneously — one tracking syntactic agreement, another coreference, another topic. Their outputs are concatenated and projected back down.
Q5. Why do transformers need positional encoding? Because attention is permutation-invariant: shuffle the tokens and the set of query-key comparisons is identical, so "dog bites man" and "man bites dog" would be indistinguishable. Position is injected explicitly, either added to the embeddings or applied as a rotation inside attention. How it's represented determines how well the model extrapolates past its trained context length.
Q6. What is the computational complexity, and why does it matter? O(n²·d) in sequence length — every token compares with every token. Doubling the context quadruples the attention cost. That's the driver behind long-context pricing, why latency grows faster than input length, and most architecture research since 2017: sparse attention, linear approximations, and IO-aware exact implementations like FlashAttention that keep the maths and cut the constant.
Q7. Does understanding attention explain why large models are capable? No, and it's worth saying so. The mechanism explains what is computed — a weighted blend of values, selected by learned similarity. Nothing in it predicts that a sufficiently large stack follows multi-step instructions or writes working code. That's an empirical result about scale, not a consequence of the architecture you can read off the equations.
8. When to use / tradeoffs
Understanding this matters when:
- Forecasting cost or latency for variable-length inputs
- Deciding between long context and chunking
- Choosing a serving stack (attention implementations differ hugely in constant factors)
- Debugging why a model ignores something present in the prompt
- Evaluating claims about "linear attention" alternatives
You can treat it as a black box when:
- Inputs are short and uniform
- You're consuming an API and cost is comfortably within budget
| Situation | Why the naive view breaks | Think instead |
|---|---|---|
| Cost modelled per token | Attention is quadratic, not linear | Model n² where lengths vary |
| Long docs cost "proportionally more" | 7.5× tokens ≈ 56× attention | Chunk: n²/k instead of n² |
| Tested only on short inputs | Non-linearity never appears | Test at the real length distribution |
| Expecting order sensitivity for free | Attention is permutation-invariant | Positional encoding does that job |
| Assuming a fact in context will be used | Attention weights decide, and can be low | Check position; see lost-in-the-middle |
| "Linear attention solves everything" | Approximations trade recall for speed | Measure on your task |
Honest limits. The example uses 4 interpretable dimensions and hand-picked projections so the result is legible — real models have hundreds to thousands of dimensions that correspond to nothing nameable, and the clean 25.7× separation here is a property of vectors chosen to produce it. The SCALE factor exists for the same reason: without it the softmax is nearly uniform and the effect is invisible, which is itself a real lesson about how sensitive attention is to input magnitude. This is also a single head in a single layer, whereas real behaviour emerges from dozens of layers of heads composing, and interpretability research shows individual heads are usually far less clean than this. The article covers bidirectional attention; decoder-only generation adds causal masking so tokens attend only backwards. And nothing here explains training, which is where the projections acquire their values.
9. Summary + related articles
- Attention lets every token read every other token directly, removing the distance bottleneck of sequential models.
- Each token emits a query (what it seeks), a key (what it offers), and a value (what it contributes). All three are learned projections.
softmax(Q·Kᵀ / √d)·V. The √d prevents softmax saturation and vanishing gradients — a one-symbol training fix.- Measured: "it" attends to "animal" 25.7× more than to "street" — the correct referent, not the nearer noun.
- No rule did that. Change one input vector and the resolution flips to
street. It is learned geometry, which is why the mechanism generalises. - Multiple heads capture different relationships at once; feed-forward, residuals, and layer norm complete the block.
- Attention is permutation-invariant, so position must be added deliberately — and how it's added determines context extrapolation.
- Quadratic cost: 10,000 tokens is 100 million comparisons. This drives long-context pricing, latency, and most architecture research.
- Chunking converts
n²inton²/k, which is often the cheapest available fix. - The mechanism does not explain why scale produces capability.
Related:
- How LLMs Work — the next-token loop this architecture serves
- Tokenization — what the tokens are before any attention happens
- Embeddings and Cosine Similarity — the input vectors, and what "direction means meaning" implies
- Context Fundamentals — the practical consequence of the quadratic term
- Sampling and Temperature — what happens after the final layer
- Reasoning Budgets: When Thinking Tokens Are Waste — why generated tokens cost more than input tokens
- Vector Search for Retrieval — dot-product similarity in a different setting
- Cost-Latency Tradeoffs — where the quadratic term lands on the bill
Resources
- Vaswani et al. (2017) — Attention Is All You Need, arXiv:1706.03762 — the paper; §3.2's scaled dot product and multi-head attention are defined here: https://arxiv.org/abs/1706.03762
- Bahdanau, Cho & Bengio (2014) — Neural Machine Translation by Jointly Learning to Align and Translate, arXiv:1409.0473 — attention before transformers, and the bottleneck it was solving: https://arxiv.org/abs/1409.0473
- Alammar, J. — The Illustrated Transformer — the standard visual walkthrough; read alongside §5: https://jalammar.github.io/illustrated-transformer/
- Su et al. (2021) — RoFormer: Enhanced Transformer with Rotary Position Embedding, arXiv:2104.09864 — the positional scheme most current models use: https://arxiv.org/abs/2104.09864
- Dao et al. (2022) — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, arXiv:2205.14135 — same maths, much smaller constant: https://arxiv.org/abs/2205.14135
- Elhage et al. (2021) — A Mathematical Framework for Transformer Circuits — what individual heads actually do, and why they are messier than §4 suggests: https://transformer-circuits.pub/2021/framework/index.html
- Karpathy, A. — Let's build GPT: from scratch, in code, spelled out — builds §5 up into a working model: https://www.youtube.com/watch?v=kCc8FmEb1nY
