Summary
The paper introduces the Transformer architecture, replacing recurrence and convolutions entirely with multi-head self-attention. This enables significantly more parallelization and achieves state-of-the-art results on translation tasks.
Key Contributions
- Multi-head self-attention — allows the model to attend to information from different representation subspaces at different positions.
- Positional encoding — sinusoidal functions inject sequence order without recurrence.
- Encoder-decoder architecture — stacked self-attention and point-wise FC layers.
Impact
This paper is the foundation of GPT, BERT, and virtually all modern LLMs. The parallelizable nature of attention over RNNs unlocked training on far larger datasets.