A developer wants generated text to draw only from the smallest group of next-token candidates whose combined probability reaches a chosen threshold, so rare words are excluded while the output still varies. Which inference parameter does this?
Show answer and explanation
Answer: A
Top-p, also called nucleus sampling, keeps the smallest set of most likely next tokens whose cumulative probability meets the threshold, then samples only from that set. A low value narrows choices toward safer wording, while a high value widens them. Temperature reshapes the whole probability distribution rather than cutting it off, maximum output tokens caps response length, and stop sequences end generation when a chosen string appears.
Why the other options are wrong
- B. Temperature rescales the probabilities to make the output more or less random. It does not set a cumulative probability cutoff on candidate tokens.
- C. Maximum output tokens limits how long the response can be. It does not decide which candidate words are eligible at each step.
- D. A stop sequence tells the model where to end generation when that text appears. It does not control how candidate tokens are chosen.