Attention Is All You Need
A new sequence transduction architecture based entirely on attention mechanisms — dispensing with recurrence and convolutions entirely.
"The Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolutions."
Encoder–Decoder Stack
The Transformer stacks N=6 identical layers in both encoder and decoder. Click any block below to inspect its role and parameters.
&
Values
from
encoder
Attention Mechanisms
Two variants work together: Scaled Dot-Product for efficiency, Multi-Head for expressiveness.
For large values of d_k, the dot products grow large in magnitude, pushing the softmax into regions with extremely small gradients. The scaling factor 1/√d_k counteracts this. With d_k=64, the scale factor is 0.125. The authors suspect additive attention outperforms dot product at large dimensions precisely due to this gradient issue.
Instead of one attention function on d_model=512 dimensions, run h=8 parallel heads each on d_k=d_v=64 dimensions. Each head can attend to different positional and semantic aspects simultaneously.
The sinusoidal form allows the model to attend to relative positions: for any fixed offset k, PE(pos+k) can be represented as a linear function of PE(pos). The model can learn to use this property.
The sinusoidal encoding can generalize to sequence lengths longer than those seen during training — an advantage over purely learned positional embeddings.
The authors found nearly identical results using learned positional embeddings. Sinusoidal was chosen as it requires no additional parameters and may extrapolate better to unseen lengths. Both embed position as additive signal on top of the token embedding.
Why Self-Attention?
Three metrics motivate replacing recurrence with self-attention: computational complexity, parallelizability, and path length for learning long-range dependencies.
| Layer Type | Complexity / Layer | Sequential Ops | Max Path Length |
|---|---|---|---|
| Self-Attention | O(n² · d) | O(1) | O(1) |
| Recurrent | O(n · d²) | O(n) | O(n) |
| Convolutional | O(k · n · d²) | O(1) | O(log_k(n)) |
| Restricted Self-Attn | O(r · n · d) | O(1) | O(n/r) |
Self-attention's O(n²·d) is cheaper than recurrence's O(n·d²) when sequence length n is smaller than representation dimension d. This is often true in practice for machine translation.
Unlike RNNs which process tokens one by one (O(n) sequential steps), self-attention processes all positions in parallel. This dramatically speeds up training.
Any two positions in the sequence can interact in O(1) operations. In RNNs, distant positions must propagate through O(n) steps, making long-range learning harder.
Self-attention layers yield interpretable behavior: different attention heads specialize in different roles. The paper visualizes heads learning to attend to anaphora resolution (e.g., "its" → its referent), syntactic structure, and other linguistic phenomena — giving researchers insight into what representations the model has learned.
Benchmarks & Ablations
State-of-the-art on WMT 2014 EN→DE and EN→FR translation, and strong performance on English constituency parsing.