What is Perplexity?

Perplexity measures how well a probability model predicts a sequence. It is a measure of uncertainty and confidence.

Toy Vocabulary

"I" "like" "cats" "dogs"

Scenarios

Key Benchmarks

  • ✔ Perfect model: \( PP \to 1 \)
  • 🎲 Random guessing: \( PP \to \text{vocab size} \)
Total Perplexity
2.07
Perplexity ≈ 2.07 → Choosing between ~2 options on average.

Sequence: "I like cats"

How it's calculated

1. Average Negative Log Likelihood
Loss = -\frac{1}{N} \sum \log P(w_i)
2. Final Perplexity
PP = \exp(Loss)
Lower Perplexity

Better model. It predicted the sequence with high probabilities. Lower uncertainty = better fit.

Higher Perplexity

Worse model. It was "confused" or "surprised" by the actual words in the text.