How Wild the Improv Gets

Opening Scene

Some improv performers play it safe, choosing the most obvious, expected next line every time — reliable, coherent, a little predictable. Others take real creative risks, occasionally reaching for a genuinely surprising, less obvious choice that can pay off spectacularly or fall flat. Neither approach is universally correct; the right choice depends entirely on what kind of scene is actually being performed. A language model has a direct, adjustable setting that controls exactly this same tradeoff: temperature.

In Plain English

At each step, a model doesn’t just predict one single “correct” next token — it computes a probability distribution across many possible next tokens. Temperature is a setting that controls how that distribution gets sampled: low temperature makes the model strongly favor the highest-probability token (more predictable, more repetitive, more “safe”), while higher temperature flattens the distribution, giving lower-probability tokens a genuinely better chance of being selected (more varied, more creative, but also more prone to incoherence at extremes).

The Old Way

Before temperature was a standard, tunable setting, earlier text generation approaches handled this tradeoff much more crudely:

Rule-based generation systems, covered in Article 3, had no genuine notion of “creative variation” at all — output was deterministic, following whatever rule fired.
Early statistical language models sometimes used simpler, less controllable randomness, without a clean, adjustable parameter for tuning the creativity-versus-coherence tradeoff.
Earlier generation approaches often defaulted to always picking the single most likely next word, producing notoriously repetitive, mechanical-sounding output.

Temperature, alongside related sampling techniques like top-k and nucleus (top-p) sampling, emerged specifically to give practitioners deliberate, fine-grained control over this tradeoff.

What’s Changing (and Why AI Is the Reason)

Temperature has become a standard, widely available parameter across nearly every LLM API, letting practitioners tune output behavior deliberately for their specific use case rather than accepting a fixed, one-size-fits-all generation style.
Complementary sampling techniques like top-p (nucleus) sampling have refined this control further, restricting sampling to a dynamically sized set of plausible candidates rather than the full vocabulary, improving output quality at a given creativity level.
As LLMs get applied to genuinely different tasks — precise code generation versus open-ended creative writing — deliberately tuning temperature and sampling strategy to the specific task has become recognized as a genuine, practical skill.

The Metaphor, Fully Extended

The Improv Scene	Temperature Concept
A performer consistently choosing the safest, most obvious next line	Low temperature, strongly favoring the highest-probability token
A performer taking genuine creative risks with less obvious choices	Higher temperature, giving lower-probability tokens a real chance
A scene needing reliability and clarity	A task needing precise, consistent output, like code generation
A scene needing genuine surprise and creative range	A task benefiting from varied, creative output, like brainstorming

For Beginners: What to Actually Do

Experiment directly with different temperature settings on the same prompt, and observe how output style changes concretely.
Learn the practical rule of thumb: lower temperature for factual, precise, or code-related tasks; higher temperature for creative, exploratory tasks.
Get familiar with top-p (nucleus) sampling as a complementary control, often used alongside temperature.

For Practitioners and Leaders: The Deeper Layer

Set temperature deliberately per use case rather than relying on a single default setting across every application.
Recognize that very high temperature settings risk incoherent, low-quality output, and very low settings risk repetitive, mechanical output — the right setting is task-dependent, not universally “better.”
Build temperature tuning into standard prompt engineering practice, covered in more depth in this content library’s dedicated prompt engineering series.

Quick Recap

Temperature controls how a model samples from its predicted probability distribution over possible next tokens.
Low temperature favors the most probable token; higher temperature gives less probable tokens a genuine chance of selection.
This parameter lets practitioners deliberately tune the tradeoff between predictable coherence and creative variation.
The right temperature setting depends on the specific task, not a universal default.

Where This Fits in the Series

Article 9 covered tuning how predictable or creative a model’s output is. Article 10 covers a technique for shaping that output further, before generation even begins: giving the performer a quick warm-up.