Controlling Creativity: Temperature & Stop Sequences

TL;DR: Large Language Models are probabilistic next-token predictors. By configuring temperature and stop_sequences inside GenerateContentConfig, you control whether the model behaves like a strict rule-bound deterministic engine (temp=0.0) or a creative generator (temp=1.0+). Learn how to set exact boundaries and inspect candidate finish reasons in production.

📦 Interactive Notebook: Test and run this code live on GitHub: 03_sampling_parameters.ipynb in the ai-architect-foundation-gemini repository.

What You'll Learn

How token sampling works under the hood, how temperature shapes the probability distribution over tokens, how to enforce hard cutoffs with stop sequences, and how to verify response integrity using finish_reason.

The Core Idea: Probabilistic Token Sampling

When an LLM generates text, it does not pick words out of thin air. For every token position, it computes a score (a logit) for every token in its vocabulary.

Temperature is a mathematical scaling factor applied to these logits before they are converted to probabilities via the softmax function:

Probability(Token_i) = exp(Logit_i / Temperature) / Sum(exp(Logit_j / Temperature))

• When Temperature approaches 0 (e.g. 0.0): The highest logit dominates completely. The model becomes deterministic and selects the most probable token every time.
• When Temperature is around 0.7: The distribution retains natural linguistic variety while staying coherent.
• When Temperature > 1.2: The probability curve flattens, giving low-probability tokens a higher chance of being selected (creative, but risks hallucination).

The Practical Temperature Spectrum

← Deterministic / Strict                          Creative / Varied →
    0.0               0.3             0.7             1.0             1.5
    |__________________|_______________|_______________|_______________|
   JSON schemas     Support bots    Summarisation    Brainstorming   Creative
   Classification   Policy checks   Conversations    Copywriting     Fiction

Production Rule of Thumb

Rule: Whenever your application parses output (JSON, SQL, boolean decisions, enum classifications), always set temperature=0.0. For customer-facing chat responses, use 0.2 to 0.4 for consistent, professional tone.

from google import genai
from google.genai import types

client = genai.Client()
model = "gemini-2.5-flash"

# Deterministic classification call
response = client.models.generate_content(
    model=model,
    contents="Classify this ticket intent into RETURN, BILLING, or TECHNICAL: 'I got charged twice.'",
    config=types.GenerateContentConfig(
        temperature=0.0,
        max_output_tokens=50,
    )
)

print(response.text.strip())

# --- Execution Output ---
# BILLING

Stop Sequences: Hard Termination Boundaries

A stop sequence is a string that immediately halts generation when the model produces it. The stop sequence itself is excluded from the returned text.

Why Stop Sequences Are Essential

1. Enforce Structural Fences: Prevent the model from continuing past a desired XML or markdown block.
2. Prevent Verbose Over-Generation: Stop the model before it generates unsolicited conversational filler.
3. Save Token Costs & Latency: Avoid wasting tokens on unwanted text.

config = types.GenerateContentConfig(
    temperature=0.0,
    stop_sequences=["<END>", "\n\nHuman:"],
    max_output_tokens=250,
    system_instruction="You are EcommAssist AI. Provide a concise answer and conclude with <END>."
)

response = client.models.generate_content(
    model=model,
    contents="Customer: How do I track my package?",
    config=config
)

print(response.text)

Inspecting Why Generation Stopped

In enterprise systems, never assume a response completed normally without checking its finish_reason.

candidate = response.candidates[0]
finish_reason = candidate.finish_reason

print(f"Finish Reason: {finish_reason}")

Production Config Blueprint

Here is a recommended production configuration for customer-facing AI agents:

production_config = types.GenerateContentConfig(
    temperature=0.0,                    # Deterministic, repeatable logic
    stop_sequences=["<END>"],           # Explicit termination marker
    max_output_tokens=400,              # Strict cost & latency ceiling
    system_instruction="""
    You are EcommAssist AI. Assist customers efficiently.
    Always conclude your final response with <END>.
    """
)

Key Takeaways

1. Temperature Controls Variance: Set 0.0 for structured extraction and logic; set 0.3–0.7 for natural conversational flows.
2. Stop Sequences Guarantee Bounds: Use them to seal outputs and eliminate excess rambling.
3. Always Check Finish Reason: Catch MAX_TOKENS truncation and safety triggers before passing output downstream.

What's Next

In Part 4: Testing & Evaluating LLM Outputs with LLM-as-a-Judge, we build automated test harnesses, exact-match evaluations, and LLM-based grading pipelines.