Temperature 0 is not a test strategy: how we test features built on LLMs
Ask a large language model the same question 1,000 times at temperature 0 and you would expect one answer. In a 2025 experiment by Thinking Machines Lab, Qwen3-235B gave 80 different ones. Classic unit tests assume that the same input gives the same output. LLM features break that assumption, and most teams ship them anyway with a handful of manual checks.
This is how we test features built on LLMs: what runs on every commit, what runs nightly, and what runs in production, where the most dangerous input comes from your users.
Why temperature 0 does not save you
Temperature 0 means the model picks the most likely next token every time. In theory that is deterministic. In practice, an inference server batches your request together with other people’s. The batch size changes with load, the order of floating point operations changes with the batch, and a tiny difference in one logit flips one token. After that, the two answers drift apart. In the experiment above, the first difference appeared at token 103.
You do not control the provider’s load, and you will switch models during the life of the product. So we stopped trying to make LLM output identical. Instead we test three different things, each in its own place:
Contract tests check our own code around the model: parsing, tools, retries, fallbacks. They run on every commit against recorded responses, never a live model.
An eval set checks the quality of answers on 100 to 300 real cases. It runs nightly and before any model or prompt change, with real calls.
Production checks look for injection, leaked data, broken output and cost on every request.
Layer one: test your code, not the model
Most bugs in an LLM feature are not in the model. They are in the code around it: a JSON parser that fails on a trailing comment, a tool call with a missing argument, a retry that charges the user twice. None of that needs a live model to test.
We record real model responses once and replay them in CI, the way you would record HTTP calls to any third party. The fixtures include the awkward cases on purpose: invalid JSON, a refusal, an empty answer, a timeout, a response cut off at the token limit. Tests run in seconds, cost nothing and never flake.
Layer two: an eval set that grows with every bug
Quality needs real calls, so it gets its own suite. An eval set is a list of inputs with a way to grade each answer. We start with 50 cases taken from real usage and add a case every time a user reports a bad answer. After a few months it is the most valuable test asset in the project.
Grading depends on the case. Exact checks come first: valid schema, required fields present, numbers that match the source. When an answer needs judgement, a second model grades it against a short rubric. Strong judges agree with human raters more than 80% of the time, about as often as humans agree with each other (Zheng et al., 2023). We still read a sample of the judge’s verdicts every week, because a judge can drift too.
Before a prompt or model change ships, the eval set runs, exact checks grade first and the judge grades the rest. The score is compared with the version currently in production, on the same cases. If it drops, we read the failed cases, fix the prompt or the code, and run it again. Only then does it ship.
Two details make this affordable. Eval runs are not urgent, so they go through the provider’s batch API, which costs half the price of normal calls at OpenAI. And because we compare against production rather than a target number, a model switch is judged on what it actually changes.
The industry is not there yet. In LangChain’s late 2025 survey of 1,340 people building agents, 89% had observability in production, but only 52% ran offline evals on test sets and 37% evaluated live traffic.
Layer three: production, where users write the prompts
Prompt injection is number one in the OWASP Top 10 for LLM applications, and no prompt wording fully prevents it. If the model reads text it did not write (an email, a web page, a support ticket), assume that text will try to give it orders. So we design for the moment it succeeds.
The model never holds credentials. When it asks to call a tool, ordinary code checks the request against an allow list, checks the arguments, and runs it with the permissions of the user who made the request, not of the service. Actions that cannot be undone, such as payments, deletions and emails, need a confirmation from the user. Output is validated against a schema before anything renders it, and we log the full trace of every request with its cost, so a bad answer in production becomes a new case in the eval set the same day.
What to take from it
Stop expecting identical output. Test your own code with recorded responses and test quality separately.
Start an eval set this week. Fifty real cases is enough to begin, and every bug report adds one.
Grade with code first, a judge second. Read a sample of the judge’s verdicts every week.
Compare against production, not a target number. A model switch is a regression test, not an upgrade.
Assume injection will succeed. Limit what the model can do, not only what it is told.
You cannot make an LLM deterministic. You can make everything around it deterministic, and measure the rest.
Building a feature on top of an LLM and not sure how to test it? Talk to us: https://www.commonsaltx.com/contact
