Which transformer component lets each token weigh every other token in the sequence when it builds its representation?
Show answer and explanation
Answer: D
Self-attention computes relevance scores between each token and every other token in the sequence, then mixes information from those tokens into each token's representation using the weights. That is how a transformer resolves references, such as which noun a pronoun points to, and captures long-range context. Because all positions are processed together, the computation also parallelizes well. Layer normalization stabilizes activations, dropout reduces overfitting during training, and positional encodings supply word order. Each matters, but none lets tokens weigh one another.
Why the other options are wrong
- A. Layer normalization rescales activations to stabilize training; it does not compute relationships between tokens.
- B. Dropout randomly zeros activations during training to reduce overfitting; it does not relate tokens to each other.
- C. Positional encodings tell the model where each token sits in the sequence, but they do not let tokens weigh one another.
Exam tip. Attention provides context between tokens; positional encodings only supply order.