Attention Is All You Need — Notes from a Slow Read

TL;DR — In 2017, a Google Brain team replaced the recurrent backbone of sequence models with pure attention. The result — the Transformer — trained faster, learned longer dependencies, and quietly became the substrate for almost every interesting model that followed: BERT, GPT, ViT, AlphaFold's Evoformer. This post walks through what the paper actually says, why it worked, and what surprised me on a careful read.

The problem in 2017

If you wanted to model a sequence in 2016, you reached for an RNN. LSTMs and GRUs had patched the worst of the vanishing-gradient problem, and Bahdanau-style attention had partly fixed the bottleneck where an entire source sentence got compressed into a single thought vector. But two stubborn issues remained.

The first was sequential computation. RNNs process tokens one at a time, because the hidden state at position t depends on the hidden state at t-1. You cannot use the GPU you paid for. Training was slow, and it got slower the longer your sequences became.

The second was path length for long-range dependencies. For an RNN, the signal from token 1 has to travel through every intervening hidden state to reach token 100. Each hop is a chance for the gradient to die or the information to get overwritten. Attention helped, but attention was still bolted on top of recurrence — the recurrence itself was load-bearing.

ConvS2S (Facebook, 2017) showed you could swap recurrence for convolutions and get parallelism. But convolutional receptive fields grow only with depth, so the path length between distant tokens still scales with the network depth. The recurrence was gone; the fundamental issue was not.

Then came a paper with a title that read like a manifesto: Attention Is All You Need.

The central claim

The paper's argument is almost cheeky in how directly it states things. You do not need recurrence. You do not need convolutions. Self-attention, stacked deep enough, with a few engineering touches, is sufficient — and it is also dramatically more parallel and has constant path length between any two positions.

Read Table 1 of the paper, and the argument becomes mechanical. For a sequence of length n and representation dimension d:

Self-attention is more expensive per layer when n > d, but it is fully parallel and any token can attend to any other token in one hop. For typical sentence-length sequences and the d the paper uses, the tradeoff is wildly favourable.

The core idea: scaled dot-product attention

Forget the architecture diagram for a moment. The single idea that makes the Transformer work is this:

For each position, produce a query. For every position (including the same one), produce a key and a value. The output at the query position is a weighted sum of values, where the weight on each value is the similarity between the query and that position's key.

In math, with Q, K, V as matrices stacking queries, keys, and values across all positions:

Attention(Q, K, V) = softmax(Q Kᵀ / √dₖ) V

A few things to notice:

1. The dot product Q Kᵀ is the similarity matrix. Row i, column j is how much position i wants to listen to position j.
2. The softmax turns similarities into a distribution. Every row sums to one, so the output at each position is a convex combination of the value vectors.
3. The √dₖ scaling is not cosmetic. In high dimensions, random dot products have large variance. Without the scaling, the softmax saturates — one value gets ~1 and the rest get ~0, and the gradient through softmax vanishes. Divide by √dₖ and you keep the pre-softmax logits in a sane range.

That is the whole mechanism. Everything else in the paper is plumbing around this idea.

Multi-head attention

One attention computation gives you a single "lens" — one way of relating positions to each other. But a sentence has many simultaneous relationships: syntactic ("who is the subject of this verb"), referential ("which noun does this pronoun point to"), positional ("what is the previous word"), semantic ("what is the topic"). One soft alignment cannot capture all of these.

Multi-head attention runs h parallel attention operations on different learned projections of Q, K, V, then concatenates the outputs and projects them back to the model dimension. The paper uses h = 8. Each head can specialise. When you visualise the attention matrices of a trained Transformer, you actually see different heads doing different things — one head attends to the previous token, another to syntactically related words, another to the start-of-sequence token (a useful "null" pointer).

The architecture, briefly

The full Transformer is an encoder-decoder. The encoder is a stack of N=6 identical layers, each containing one multi-head self-attention sublayer followed by a position-wise feed-forward sublayer, with residual connections and layer normalisation around each.

The decoder mirrors the encoder, but with two changes:

• The self-attention is masked. When predicting token t, the decoder must not see tokens t+1, t+2, … — that would be cheating. The mask sets the relevant entries in the pre-softmax similarity matrix to -∞, so the softmax assigns them zero weight.
• Each decoder layer has an extra cross-attention sublayer between self-attention and the feed-forward. Here the queries come from the decoder, but the keys and values come from the encoder's output. This is how the decoder reads the source sentence.

Two more details that are easy to gloss over:

• Positional encodings: self-attention is permutation-invariant by construction — it does not know the order of the tokens. The paper adds sinusoidal positional encodings to the input embeddings, of the form sin(pos / 10000^(2i/d)) and cos(...). The frequencies form a geometric progression, so any positional offset can be expressed as a linear function of the encoding at the original position. The paper notes this should help the model generalise to sequence lengths it did not see during training.
• Position-wise FFN: between attention sublayers, each position passes independently through a two-layer MLP with ReLU. The hidden size is 4× the model size (d_ff = 2048 vs d_model = 512). This is where most of the parameters live, and it is also where the model does most of its "thinking" per position once attention has gathered the relevant context.

Why it won

The empirical results are almost a footnote — but they are striking. On WMT14 English-to-German, the Transformer beats every prior result on BLEU, at a fraction of the training cost. On English-to-French it sets a new state of the art. The "base" model trains in 12 hours on 8 GPUs.

But the long-run story is not BLEU on WMT. The long-run story is that this architecture turned out to be a substrate. The same encoder, slightly modified, became BERT. The same decoder, slightly modified, became GPT. Strip the encoder-decoder distinction and feed in image patches instead of tokens, and you get a Vision Transformer. The Transformer is general in a way that RNNs and CNNs are not, and once people realised that, the rest was scaling.

What surprised me on a careful read

• How small the original Transformer is by 2026 standards. Base model: 65M parameters. Big model: 213M. By contemporary standards these are tiny. The architecture clearly had headroom no one in 2017 fully appreciated.
• How much of the paper is engineering, not theory. Label smoothing, warmup learning-rate schedule, dropout placement, weight tying between the embedding and the pre-softmax linear. None of these are the "headline" idea, but the model probably does not train without them.
• The bet on parallelism. The authors are very explicit that they are trading per-layer FLOPs for parallel hardware utilisation. In 2017 this was a wager. In 2026 it looks like prophecy.

What I want to dig into next

• Efficient attention variants. O(n²) is fine for sentences and brutal for documents. FlashAttention (a memory-aware exact attention), Linformer, Performer, Longformer — all attempts to push beyond the quadratic wall.
• Positional encoding alternatives. RoPE (rotary positional embeddings) and ALiBi have largely replaced sinusoidal encodings in modern LLMs. Worth understanding why.
• Implementing self-attention from scratch. Karpathy's nanoGPT is the canonical starting point, and there is no substitute for writing the einsums yourself.

References

• Vaswani et al., Attention Is All You Need, NeurIPS 2017. arxiv.org/abs/1706.03762
• Bahdanau et al., Neural Machine Translation by Jointly Learning to Align and Translate, ICLR 2015. arxiv.org/abs/1409.0473
• Jay Alammar, The Illustrated Transformer. jalammar.github.io/illustrated-transformer
• Andrej Karpathy, nanoGPT. github.com/karpathy/nanoGPT