How a transformer works

Every term below that isn't standard mathematics gets defined where it first appears, and no mechanism shows up before the problem it exists to solve. There's a glossary at the end.

Before any internals, be precise about input and output, because it's narrower than it looks from outside.

You hand the model a sequence of text. It returns one probability distribution over its entire vocabulary — a number for each of roughly 100,000 possible chunks of text, giving the likelihood that each one comes next. That is the whole output. A single list of probabilities.

input:   "The probe reached "

output:  Jupiter   0.31
         Saturn    0.22
         Earth     0.14
         the       0.09
         ...  100,000 entries in total, summing to 1.0

Something outside the model then picks one entry, appends it to the text, and calls the function again on the longer string. Predict, append, repeat. That loop is where sentences come from. The model itself never produces a sentence — it produces one distribution, with no notion that it is about to be called again.

The model can only do arithmetic, so text has to become numbers. First it's cut into pieces called tokens.

Tokens are not characters and not quite words. They're chunks chosen by a compression procedure run over a large body of text before training starts: begin with individual characters, repeatedly find the most frequent adjacent pair and merge it into a new unit, stop after about 100,000 units. Common words survive whole; rare ones get split. A leading space is usually glued to the front, which is why ·the and the are two distinct entries.

The output is a fixed list — the vocabulary — and each token is identified by its position in that list. Nothing more than that.

The model holds a large matrix called the embedding table: one row per vocabulary entry, each row a few thousand numbers wide. GPT-2 small used 50,257 × 768. A frontier model is closer to 100,000 × 5,000 — tens of millions of numbers, every one of them learned during training.

Turning a token ID into a vector is then the simplest operation in the entire model: read that row. Token 84049 means "fetch row 84049 of the table". It's a lookup, not a computation. The vector you get back is the token's embedding.

Why go to the trouble? Two reasons:

• Vectors support arithmetic. You can add them, scale them, project them, take dot products. None of that means anything on an arbitrary index.
• Distance can encode similarity. Training pushes tokens that get used in similar contexts toward similar directions. Nobody programs this — it falls out of the optimisation, because the model does better when it can generalise across related words. Once it has happened, ·Jupiter and ·Saturn end up near each other, and the angle between two vectors becomes a usable measure of relatedness.

The embedding table begins as random noise. Everything meaningful in it is a residue of training.

There's a problem with what we have, and it's clearest if you look one step ahead: the mixing operation coming in step 5 treats its inputs as an unordered set. Give it "Jupiter orbits the probe" or "The probe orbits Jupiter" and, with embeddings alone, it computes identical results. Word order would be invisible to it.

So position has to be pushed into the vectors themselves. The original scheme — still the easiest to look at — builds a positional encoding vector for each slot out of sines and cosines at geometrically spaced frequencies, and simply adds it:

vector at position p  =  embedding[token_id]  +  position_vector[p]

Adding sounds destructive — surely the two signals scramble together? In a space of several thousand dimensions there's room for both to coexist and for later operations to pull them apart again. Different frequencies occupy different dimensions, so "which token" and "which slot" don't compete for the same coordinates.

Every token now has one vector encoding what it is and where it is. That's the input to the model proper.

Here's the limitation of everything so far, and it motivates the entire rest of the architecture.

An embedding is context-free. The row for ·bank is one fixed set of numbers. It is the same vector in "the river bank was steep" as in "the bank approved the loan". Those are different meanings, and any useful model has to end up representing them differently.

The same goes for anything involving reference. In "the probe reached it in 2016", the vector for ·it starts as the generic ·it row, carrying nothing about what it points to.

So each token's vector needs to be updated using information from the other tokens present. Which raises exactly two questions:

1. Which other tokens should it draw from? For ·bank, the answer had better be ·river or ·loan, not ·the.
2. What should it take from them? Having decided ·river is relevant, what exactly gets copied across?

The mechanism that answers both is called attention. That is all the word means: a procedure letting each token gather information from the others, with learned control over which ones and what.

Start somewhere familiar. In a dictionary or hash table, a lookup works like this:

d = {"river": 0.9, "loan": 0.4, "the": 0.1}
d["river"]   →   0.9

There are three roles in play. You supply a search term. It gets compared against the stored labels. Exactly one matches, and you get back the contents filed under it. Matching is all-or-nothing: one hit, everything else contributes zero.

Attention keeps the three roles and abandons the all-or-nothing part. Every token produces all three:

• a search term, called its query — "what am I looking for?"
• a label, called its key — "what do I advertise about myself?"
• contents, called its value — "what do I hand over if chosen?"

Then, instead of finding the single matching key: compare the query against every key, get a match score for each, convert those scores into weights summing to 1, and return the weighted average of all the values. A key that matched well dominates; one that matched poorly contributes nearly nothing; nothing is excluded outright.

That's what makes it a soft lookup — retrieval by blending rather than selection. It also matters that it's a blend: a smooth weighted sum has a useful derivative everywhere, so the whole thing can be trained by gradient descent. A hard lookup, being a branch, does not.

Where the three vectors come from

The layer holds three learned matrices, W_Q, W_K and W_V. Each token's input vector gets multiplied by each of them:

query_i = W_Q · x_i     key_i = W_K · x_i     value_i = W_V · x_i

The same three matrices apply to every token — they're properties of the layer, not of any particular word. Three separate matrices are needed because the three roles really are different things. What ·bank wants to ask ("is there a water word or a money word near me?") is not what ·river wants to advertise ("I am a water word"), and neither is the content ·river passes along. A single matrix couldn't produce all three.

Scoring a match

The match score between a query and a key is their dot product — large and positive when they point the same way, near zero when orthogonal, negative when opposed. Training's job is to shape W_Q and W_K so that queries and keys which ought to match come out aligned.

Two adjustments before scores become weights:

• Divide by √d, where d is the vector length. Dot products of random vectors grow like √d, and the next step misbehaves on large inputs. This holds the variance roughly constant regardless of model size.
• Apply softmax. Softmax converts any list of real numbers into positive numbers summing to 1: exponentiate each, divide by the total. Exponentiating forces positivity and exaggerates gaps, so a modest lead in raw score becomes a decisive lead in weight.

Not looking ahead

One constraint remains. During training the model predicts the next token at every position of a passage at once — that's what makes training tractable. But a position predicting its own successor must not be allowed to see it. So before softmax, every score pointing at a later position is set to −∞, which softmax maps to exactly zero weight. This is the causal mask, and a model built with it is autoregressive: each position sees only itself and what came before.

Here is the whole operation with numbers you can step through:

Several attentions at once

One set of W_Q/W_K/W_V can express only one notion of relevance. Real layers run dozens of independent copies side by side, each with its own three matrices. Each copy is a head. They operate in parallel on the same input, and their outputs are concatenated and mixed by one further learned matrix.

The point is specialisation: one head can track grammatical subjects while another tracks the immediately preceding token and a third tracks topic.

Attention moves information between positions. The second half of each layer does processing within a position: a small two-layer neural network applied to each token's vector separately and identically. It's called the feed-forward network, or MLP, and mechanically it's two matrix multiplications with a nonlinear function applied elementwise between them:

out = W_2 · nonlinear(W_1 · x)

The intermediate result is typically 4× wider than the vector itself, which is why this block holds most of the model's numbers despite being simpler than attention.

The nonlinearity is not decoration, and the reason is worth stating plainly: a stack of linear maps collapses into a single linear map. Without something nonlinear between them, a 100-layer model would be algebraically identical to one matrix and could represent nothing that one matrix couldn't. The nonlinearity is the only thing that makes depth mean anything.

Interpretability work suggests much of what we'd loosely call stored knowledge lives in these blocks, as patterns of which intermediate units fire. Attention decides what to look at; the feed-forward network decides what to make of it.

Both halves write their results back by addition rather than replacement:

x = x + attention(x)
x = x + feedforward(x)

The running vector being added into is the residual stream — "residual" because each block computes a correction to what's already there rather than a replacement value. It runs unbroken from embedding to output, and every block both reads from it and writes into it.

One piece of housekeeping appears at each block: layer normalisation, which rescales a vector to zero mean and unit variance before use. It doesn't change what's represented, it just keeps magnitudes in a range where the arithmetic stays well-conditioned. Numerical hygiene.

Rough intuition for what depth buys, with the caveat that it's a simplification: early layers resolve tokens into words and local grammar, middle layers build meaning and relationships, late layers narrow toward a specific prediction.

At the top of the stack, take the vector at the last position — the one that has been accumulating context about what should come next — and multiply it by a final matrix with one row per vocabulary entry. That yields one raw score per possible token. These unnormalised scores are called logits, and softmax turns them into the probability distribution promised at the start.

Something then has to choose one. That choice is not part of the model:

• Temperature divides the logits before softmax. Below 1 it sharpens the distribution toward the leader; above 1 it flattens toward uniform. At 0 it becomes "always take the maximum" and the model is fully deterministic.
• Top-p (nucleus sampling) discards the tail: keep the smallest set of candidates whose probabilities sum to p, renormalise, sample from those. It suppresses rare nonsense without fixing the number of options in advance.

The chosen token is appended and the entire stack runs again from the beginning. The model has no plan, no draft, and no sense that a sentence is in progress. Longer structure is an emergent consequence of repeatedly making a good local choice.

• No memory between calls. No state survives a request. Every conversational turn re-sends the whole transcript and reprocesses it from scratch.
• No variable effort. A hard question and a trivial one get exactly the same number of layers. "Thinking" tokens are the workaround — generating intermediate text buys extra passes to condition on.
• No stored lookup table. Nothing is kept verbatim and retrieved. Facts live smeared across billions of weights, which is why they can be subtly wrong in ways a database never is.
• No confidence signal. The output distribution is all there is. High probability means "text like this usually continues this way", not "this is true".