Quick Navigation

Introduction Self-Attention Cross-Attention Key Differences Math Formulation Applications Summary
Home › Machine Learning › Transformers › Self-Attention vs. Cross-Attention

Self-Attention vs. Cross-Attention

📖 Machine Learning ⏱️ 12 min read 📅 Updated: Nov 14, 2025 Transformers NLP Computer Vision

Attention mechanisms have fundamentally reshaped modern artificial intelligence, enabling models to dynamically weigh the importance of different input elements. Within the Transformer architecture, two primary variants dominate: self-attention and cross-attention. While they share the same mathematical foundation, they serve distinct computational purposes and are deployed in different architectural contexts.

Core Distinction: Self-attention computes relationships within a single sequence, whereas cross-attention computes relationships between two different sequences.

This entry dissects both mechanisms, their mathematical formulations, architectural roles, and real-world applications across language modeling, vision, and multimodal AI.

Self-Attention Mechanism

Self-attention (also called intra-attention) allows a model to focus on different parts of the same input sequence when computing representations for each element. It was the foundational innovation introduced in the 2017 paper "Attention Is All You Need".

In self-attention, the query (Q), key (K), and value (V) matrices are all derived from the same input tensor. For a sequence of token embeddings \(X \in \mathbb{R}^{N \times d_{model}}\), linear projections produce:

Q = XW_Q, \quad K = XW_K, \quad V = XW_V

The attention scores are computed as the scaled dot-product of queries and keys, followed by a softmax normalization. These scores determine how much each token should attend to every other token in the sequence, including itself.

Self-Attention FlowPyTorch-style pseudocode
# Input: sequence X of shape (batch, seq_len, dim) Q, K, V = linear_Q(X), linear_K(X), linear_V(X) attention_weights = softmax((Q @ K.transpose(-2, -1)) / sqrt(d_k)) output = attention_weights @ V

Self-attention excels at capturing long-range dependencies, parallelizing sequence processing, and forming contextualized representations. It is the backbone of encoder-only models (e.g., BERT) and decoder-only models (e.g., GPT).

Cross-Attention Mechanism

Cross-attention computes attention between two distinct sequences. Typically, the queries originate from one sequence (e.g., the decoder's current hidden state), while the keys and values originate from another (e.g., the encoder's output).

Cross-Attention Signature: Q \(\leftarrow\) Sequence A, \quad K, V \(\leftarrow\) Sequence B

This mechanism enables conditional generation and information transfer. The decoder "queries" the encoder's representations to decide which source tokens should influence the current prediction.

Cross-Attention FlowDecoder attending to Encoder
# Decoder hidden states: h_dec, Encoder output: h_enc Q = linear_Q(h_dec) # Queries from decoder K = linear_K(h_enc) # Keys from encoder V = linear_V(h_enc) # Values from encoder attention_weights = softmax((Q @ K.transpose(-2, -1)) / sqrt(d_k)) output = attention_weights @ V # Context-aware decoder representation

Cross-attention is essential for sequence-to-sequence tasks, multimodal alignment, and retrieval-augmented generation (RAG), where the model must dynamically ground its outputs in external or auxiliary information.

Key Differences

Aspect Self-Attention Cross-Attention
Source of Q, K, V All from the same sequence Q from sequence A; K, V from sequence B
Primary Function Intra-sequence contextualization Inter-sequence information transfer
Typical Location Encoder layers, Decoder layers Decoder layers (in seq2seq models)
Masking Optional (causal mask in decoders) None required (full access to source)
Computational Complexity O(N² × d) for sequence length N O(N × M × d) for lengths N and M

Mathematical Formulation

Both mechanisms use the scaled dot-product attention function. The general form is:

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

The difference lies entirely in the provenance of the inputs:

  • Self-Attention: \(Q = K = V = \text{projections of } X\)
  • Cross-Attention: \(Q = \text{projection of } X_{source1}, \quad K = V = \text{projections of } X_{source2}\)

Multi-head attention extends both variants by projecting Q, K, V into multiple subspaces, computing attention in parallel, and concatenating the results. The head mechanism operates identically regardless of attention type.

Architectural Placement

In the original Transformer (Vaswani et al., 2017), the encoder stacks contain only self-attention layers, while the decoder stacks alternate between masked self-attention and cross-attention layers. This design allows the encoder to build rich contextual representations of the input, while the decoder generates output tokens conditioned on both previously generated tokens (via self-attention) and the full input sequence (via cross-attention).

Modern architectures have expanded these patterns:

  • Encoder-only (BERT, RoBERTa): Pure self-attention for bidirectional understanding.
  • Decoder-only (GPT, LLaMA): Masked self-attention for autoregressive generation.
  • Encoder-Decoder (T5, BART, NLLB): Self-attention in encoder, cross-attention in decoder for translation/summarization.
  • Diffusion Models (Stable Diffusion): Cross-attention injects text conditioning into latent image generation.

Applications

Self-Attention

  • Contextual word embeddings and language modeling
  • Document classification and sentiment analysis
  • Protein structure prediction (AlphaFold uses attention variants)
  • Time-series forecasting and anomaly detection

Cross-Attention

  • Multilingual machine translation
  • Image captioning and visual question answering
  • Retrieval-Augmented Generation (RAG) systems
  • Text-to-image and text-to-video diffusion models
  • Code generation with documentation/context conditioning

Key Takeaways

Self-attention and cross-attention are mathematically identical but architecturally distinct. Self-attention enables models to understand internal structure and dependencies within a single sequence, making it ideal for representation learning. Cross-attention bridges separate modalities or sequences, enabling conditional generation and multimodal alignment. Together, they form the relational backbone of modern AI systems.

References & Further Reading

  1. Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017.
  2. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015.
  3. Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019.
  4. Ramesh, A., et al. (2022). Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint.
  5. Press, O., et al. (2022). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR 2022.