Multi-Head Attention
Multi-head attention is a core computational mechanism introduced in the Transformer architecture (Vaswani et al., 2017) that enables neural networks to dynamically weigh the importance of different input elements relative to each other. Unlike fixed-position sequence models, multi-head attention computes contextual representations in parallel across multiple representation subspaces, allowing the model to capture diverse relational patterns simultaneously.
Mathematical Foundation
Standard scaled dot-product attention computes a weighted sum of values, where the weights are derived from the compatibility of queries and keys:
Here, Q (queries), K (keys), and V (values) are matrices projected from the input embeddings. The scaling factor √dₖ prevents dot products from growing too large in magnitude, which would push the softmax function into regions with vanishingly small gradients.
Architecture & Mechanism
Multi-head attention extends this by applying attention h times in parallel with different learned linear projections:
Each head projects inputs to dimension dₖ = d_model / h. The concatenated outputs are linearly transformed by Wᴼ ∈ ℝ^{hdₒ × d_model}. This design maintains computational efficiency while exponentially increasing representational capacity.
Why Multiple Heads?
Single-head attention often collapses into tracking a single type of relationship (e.g., only positional or only semantic). Multiple heads act as independent expert modules:
- Head 1 may track grammatical agreement
- Head 2 may track coreference chains
- Head 3 may attend to long-range dependencies
Empirical visualization studies (e.g., Attention Flow) confirm that different heads specialize in distinct linguistic or structural features, enabling richer contextual modeling than any single attention matrix could achieve.
Computational Complexity
| Metric | Standard Attention | Multi-Head (h heads) |
|---|---|---|
| Time Complexity | O(L²·d) | O(h·L²·(d/h)) = O(L²·d) |
| Space Complexity | O(L²·d) | O(L²·d) + O(h·d²) |
| Parallelizability | Low | High (independent heads) |
While per-token complexity remains quadratic in sequence length L, multi-head attention is highly parallelizable across heads, making it exceptionally efficient on modern GPU/TPU architectures. Memory bandwidth, rather than raw FLOPs, typically becomes the bottleneck.
Applications
Originally designed for natural language processing, multi-head attention has become a foundational primitive across AI domains:
- Language Models: GPT, BERT, T5, and LLaMA families rely entirely on stacked multi-head attention layers.
- Computer Vision: Vision Transformers (ViT) replace CNNs with patch-level attention mechanisms.
- Multi-Modal Learning: Cross-attention aligns text, image, and audio embeddings in models like CLIP and Flamingo.
- Time Series & Graphs: Adapted for sequential forecasting and message passing in graph neural networks.
Variants & Extensions
Research has rapidly evolved the original formulation to address efficiency and expressivity limits:
- Grouped-Query Attention (GQA): Shares keys/values across heads to reduce KV-cache memory.
- Multi-Query Attention (MQA): Extremes of GQA (single K/V pair), used heavily in inference optimization.
- FlashAttention: I/O-aware tiling algorithm that reduces memory bandwidth constraints.
- Linear Attention: Approximates softmax with kernels to achieve O(L) complexity.
References
- Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 30.
- Kaushik, K., et al. (2020). Transformer Interpretability Beyond Attention Visualization. ACL.
- Joshi, M., et al. (2020). SpanBERT: Improving Pre-training by Representing and Predicting Spans. TACL.
- Liu, H., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention. NeurIPS.