Reasoning and Test-Time Compute
For most of this course, a model has had exactly one speed. You send tokens, it produces tokens, and the only lever on quality was what you put in the window.
That changed. A reasoning model spends tokens thinking before it answers you. Not because you asked it to, but because it was trained to. The thinking is generated the same way as any other output, it costs the same as any other output, and in most products you never see it.
The engineering question is not whether this is impressive. It is when spending three times the tokens and ten times the wait buys you anything. That is a routine decision now, made per task, and getting it wrong in either direction is expensive.
In short: Test-time compute means paying at inference for accuracy, in generated thinking tokens. It wins on problems with steps, where one early mistake ruins the answer. It is waste on recognition, extraction and formatting. And "be concise" does not control it, because the thinking is not the reply.
Thinking Is Just More Tokens
Nothing new happens inside the model. Go back to the loop from How a Language Model Works: score the next token, pick one, append, repeat.
A reasoning model runs that same loop for a while on its own working-out before it starts on your answer. Those are thinking tokens or reasoning tokens. They are real generated output. They fill context, they take time, and they appear on your bill, usually as their own line.
This is why the trade is honest and easy to reason about. You are buying compute at the moment of the question, which is what test-time compute means, instead of getting it for free from training. More thinking, better odds on hard problems, higher bill, longer wait.
This Is Not Chain of Thought With a New Name
The obvious objection: we already had this. Prompt Engineering Is Software Engineering covered chain of thought, where you tell a model to work through it step by step and accuracy goes up.
The mechanism looks similar and the difference matters in practice. Chain of thought is something you apply, in a prompt, to a model that would otherwise answer immediately. A reasoning model was trained to do it, does it whether you ask or not, and was optimised specifically for that behaviour, so its working-out is generally better than what you get by asking a standard model nicely.
Two consequences follow.
Telling a reasoning model to think step by step is redundant, and can hurt. You are prompting for behaviour it already has, and elaborate reasoning instructions can push it into re-explaining rather than solving. Ask for the outcome and the constraints. Let it do the middle.
You cannot prompt your way out of paying. With a standard model, dropping the chain-of-thought instruction makes it cheap again. With a reasoning model, the thinking is the product. If you do not want to pay for it, use a different model.
When It Pays and When It Burns Money
Split the work by whether it has steps that can go wrong.
Test-time compute pays on problems where an early mistake poisons everything after it. Multi-step maths. Debugging, where the cause is three layers from the symptom. Planning a sequence of tool calls where step two depends on getting step one right. Anything you would want a colleague to think about before answering, rather than answer instantly.
It burns money on recognition and transformation. Extraction. Classification. Reformatting. Summarising. Routing. These have no intermediate steps to get wrong, so the thinking has nothing to fix and you pay for it anyway. A reasoning model on a field-extraction endpoint is the most expensive way to be slightly slower.
| Task shape | Reach for | Why |
|---|---|---|
| Multi-step derivation, planning, debugging | Reasoning model | An early error ruins the result, so thinking has something to catch |
| Extraction, classification, reformatting | Standard model | No intermediate steps, so no accuracy to buy |
| High volume, latency sensitive, simple per call | Standard model | The wait is the product problem, not the accuracy |
| Rare, expensive to get wrong, a human waits anyway | Reasoning model | Seconds do not matter when the answer being wrong does |
The pattern behind the table: buy thinking where being wrong is expensive and being slow is cheap. In an agent that is often a small number of decisions, not every call. Routing a hard planning step to a reasoning model and leaving the rest on a fast one is normal architecture now, not a clever trick.
The Two Traps
Latency is a product decision, not a technical one. A reasoning model can take tens of seconds. That is fine in a batch pipeline, in a code review, in something a person kicked off and walked away from. It is unusable in an interactive chat where a user is watching a cursor. Deciding this after you have built the interface is how a feature dies.
Effort is the real dial. Reasoning models expose how hard to think, under names like effort or a thinking budget. That is the control that changes your bill. Prompt wording does not, and asking for a shorter answer trims the visible part while the expensive part is untouched. Turn the dial down first and measure whether accuracy actually drops on your task, because it often does not.
Buying Accuracy Is Now a Line Item
The old shape of this field was simple: quality came from the model you picked and the context you fed it, and both were fixed at the moment you made the call.
Now there is a third dial, and it is priced. You can spend more at the moment of the question and get better odds on the hard ones. That is a real capability, and treating it as a default rather than a choice is how teams triple a bill for no measurable gain on work that never had a hard step in it.
So decide it per task, the same way you would decide between a cache and a database. Thinking is a resource you buy. Buy it where being wrong actually costs something.