An engineer fine-tuning a transformer language model on a single accelerator keeps hitting out-of-memory errors, and the batch size they need is out of reach. Which two changes most directly lower peak memory while keeping training quality?
Show answer and explanation
Answer: B
Activation checkpointing discards intermediate activations during the forward pass and recomputes them when gradients are needed, trading extra compute for memory. Mixed precision stores and computes most tensors in FP16 or BF16, roughly halving the footprint of activations and weights while usually preserving accuracy. A larger step size leaves memory unchanged, CPU execution is much slower, and shrinking the vocabulary harms the model.
Why the other options are wrong
- A. A larger step size changes how far each update moves but leaves the memory footprint of weights, gradients, and activations unchanged, so the out-of-memory failures would persist.
- C. CPU execution avoids accelerator memory limits but is far slower for transformer math, and host RAM is finite too, so this trades a memory problem for a throughput problem.
- D. Vocabulary reduction hurts tokenization coverage and language quality, and the embedding table is only one part of memory use; it does not address activations, which dominate at large batch sizes.