EE508 — Hardware Foundations for ML · NLP & Language Models

Text Generation Decoding Strategies

When a language model generates text it must choose the next token at each step. This visualization shows how four strategies navigate the probability distribution — and why each leads to different outputs.

↳ Autoregressive example from ML_LectureSp26_transformers2.pdf, Slide 6 — "The cat sat → on → the → mat"
Prompt "The cat sat" ← from Lecture slides
Greedy Search
Decision Explanation
Press Run to begin the step-by-step animation.
Cumulative log-score
–
Generated:
Strategy
Greedy Search

At each step, pick the single token with the highest probability. Fast and deterministic — but may miss better overall sequences.

x̂ₜ = argmax P(xₜ | x₁…xₜ₋₁)
Parameters

No parameters — greedy always picks the single highest-probability token.

Next-token distribution
Strategy Comparison
GREEDY
Deterministic: Yes
Search width: 1
Complexity: O(V)
Best for: speed
BEAM
Deterministic: Yes
Search width: k paths
Complexity: O(k·V·L)
Best for: translation
TOP-K
Deterministic: No
Search width: k tokens
Complexity: O(V+k)
Best for: creative text
TOP-P
Deterministic: No
Search width: dynamic
Complexity: O(V log V)
Best for: open-ended