A Language Model Is a Probability Machine

A language model is a probability machine. Not a brain, not a database, not a little person trapped in a server. Understanding how a language model works starts and ends with one fact: it takes text and it guesses what comes next. Everything you will ever build on top of it is steering that guess.

Hold that fact. Every knob you touch later, temperature, context length, the cost on your bill, the moment it invents a fact, traces straight back to it. This post is the companion to What an LLM Actually Is: that one names the engine, this one opens the casing and shows you the parts.

It Predicts One Token, Then Does It Again

The core move is next-token predictionnext-token predictionThe single thing a language model does: score every token in its vocabulary for how well it fits next.. You hand the model some text. It scores every possible continuation and tells you how likely each one is. A runtime picks one, sticks it on the end, and feeds the whole thing back in. That single pass through the model is called inferenceinferenceRunning a trained model to get an answer, as opposed to training it. What you pay for per call., and a full reply is just inference run a few hundred times in a loop.

There is no second mechanism. Chat, code, reasoning, agents: all of it is this loop, dressed differently. Once you accept that, the model stops being magic and starts being a thing you can reason about.

Text Becomes Tokens, And Tokens Are What You Pay For

The model does not read letters or words. It reads tokenstokensThe chunks of text a model reads and writes, roughly a word or a piece of one., chunks of text that are usually a short word or a fragment of a longer one. Turning your text into these chunks is tokenizationtokenizationCutting text into the chunks a model actually reads. It decides your cost and your length limits., and the most common method, BPEBPEByte Pair Encoding, the common way a vocabulary is built by merging frequent character pairs into single tokens. (byte-pair encoding), builds its fixed vocabularyvocabularyThe fixed set of tokens a model knows, often around a hundred thousand of them. by merging the most frequent character pairs until it has around a hundred thousand pieces.

This is not a detail for later. Token count is the meter on everything. You pay per token, in and out. The context limit is counted in tokens. Latency scales with them. A word like the is one token. A rare technical term or a long German compound can be five. That is why the same idea can cost twice as much in one phrasing as another, and why trimming a prompt is real money, not tidiness.

Meaning Is Geometry

A token ID is just a number, and numbers carry no meaning. So the first thing the model does is turn each token into an embeddingembeddingA vector that stands for a piece of text, positioned so that similar meanings sit close together., a long list of numbers that places the token as a point in a high-dimensional vector spacevector spaceThe coordinate space embeddings live in, where distance stands in for similarity of meaning..

The trick is what that space encodes. Tokens with related meanings land near each other. king sits close to queen, Berlin close to Paris, and directions in the space line up with concepts. Meaning becomes distance and direction, something you can measure. Hold onto this picture. It is the exact idea that later powers retrieval and memory, where you find relevant text by finding nearby points.

Attention Is How It Reads Relationships

A pile of word meanings is not understanding. In the bank of the river and money in the bank, the token bank means opposite things, and the difference lives entirely in the words around it. AttentionAttentionThe mechanism that lets a model weigh how much each token matters to every other token. is the mechanism that lets each token look at the others and pull in the context it needs.

No math here, just the intuition. For every token, the model asks: which other tokens matter for what this one means right now? It weighs them and mixes them in. It does this with three vectors it derives for each token, together called QKVQKVQuery, Key and Value, the three projections attention computes to decide what each token should look at.: the query is what a token is looking for, the key is what every other token advertises, and the value is what gets mixed in when a query and key match. When tokens in one sentence attend to each other, that is self-attentionself-attentionAttention within one sequence, where each token looks at the other tokens around it.. When one stream of tokens attends to a separate one, the way a translation looks back at its source sentence, that is cross-attentioncross-attentionAttention from one sequence to another, used where a model conditions on a separate input.. And the model does not do this once. It runs many attention passes side by side, each free to track a different kind of relationship, one following grammar, another following who-did-what. That is multi-head attentionmulti-head attentionRunning attention several times in parallel so the model can track several kinds of relationship at once.. Attention is why a Transformer reads a whole passage as a web of relationships instead of a flat string of words.

The Context Window Is Its Whole World

Everything the model can see at once lives in the context windowcontext windowThe maximum tokens a model can hold at once, and the whole of what it knows for that call., measured, of course, in tokens. Your system prompt, the conversation so far, retrieved documents, all of it competes for the same fixed budget. Nothing outside the window exists to the model. It has no memory between calls beyond what you put back in.

Two mechanics fall out of this. Attention treats tokens as a set, so order has to be added deliberately through positional encodingpositional encodingHow a model is told the order of tokens, since attention alone sees a bag of them., which stamps each token with its place in the sequence. The method most current models use is RoPERoPERotary Position Embedding, the positional scheme most current models use, which extends to longer contexts more gracefully., rotary positional encoding, which encodes position by rotating each token's vector by an amount that grows with its place, so the model reads distances between tokens consistently as the sequence gets long. And to avoid re-computing the whole history on every step, the model caches the attention work it already did in a KV cacheKV cacheStored intermediate state from tokens already processed, so generating the next token does not redo the whole sequence.. That cache is why generation speeds up after the first tokens, and why a long prompt is slow and expensive twice over: more tokens to process, and a bigger cache to carry. Context is not free storage. It is the most contested resource you manage.

Every Knob Is a Way to Pick the Next Token

Here is where the "probability machine" claim earns its keep. The model's raw output is a vector of logitslogitsThe raw unnormalised scores a model produces for every possible next token, before they become probabilities., one unbounded score per token in the vocabulary. Scores are not probabilities, so softmaxsoftmaxThe step that turns raw logits into probabilities that add up to one. squeezes them into numbers between zero and one that sum to one. Now you have a real distribution over what comes next.

Every sampling control you have ever set just reshapes or reads this one distribution.

TemperatureTemperatureThe dial that flattens or sharpens the probability distribution. Higher is more varied, lower is more predictable. scales the logits before softmax. Low temperature sharpens the distribution toward the top choice, giving safe and repetitive output. High temperature flattens it, giving variety and risk. Top-kTop-kSampling restricted to the k most likely tokens, so the long tail of unlikely ones is never chosen. keeps only the k highest-scoring tokens and ignores the rest. Top-pTop-pSampling from the smallest set of tokens whose probabilities add up to p. Also called nucleus sampling., or nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p, so the pool grows when the model is unsure and shrinks when it is confident. Greedy decodingGreedy decodingAlways taking the single most likely next token. Deterministic, and often flat and repetitive. skips the dice entirely and always takes the single most likely token. Beam searchBeam searchKeeping several candidate continuations alive at once and choosing the best complete one. goes further, tracking several high-probability continuations at once and extending the best few before it commits, spending compute to find a sequence more likely as a whole rather than token by token.

None of these change what the model knows. They only change how you draw from the same probabilities. That is the whole game.

Prefill Reads, Decode Writes

Generation runs in two phases, and knowing them explains your latency. PrefillPrefillThe first phase of a call, where the model reads your whole input at once. is the model reading your entire prompt in one parallel pass to fill that KV cache. DecodeDecodeThe second phase, where the model writes the answer one token at a time. is the loop that follows, producing one token at a time, each depending on the last. Prefill is fast per token because it is parallel. Decode is slower because it is sequential. This is why time-to-first-token and time-per-token are different numbers, and why streaming tokensstreaming tokensSending tokens to the user as they are generated instead of waiting for the whole answer. back to the user feels responsive: you are watching the decode loop, live. Speculative decodingSpeculative decodingA small fast model drafts several tokens and the big model checks them in one pass, which cuts latency. speeds that loop up: a small, fast model drafts several tokens ahead and the large model verifies them in one pass, so you get the big model's quality at closer to the small model's speed.

Hallucination Is Not a Bug

Now the payoff. When a model states something false with total confidence, that is a hallucinationhallucinationA confident, fluent, wrong answer. It is the mechanism working normally, not a bug to be patched., and it is structural, not a defect you can patch out.

Look at what the machine actually does. It samples a plausible next token from a probability distribution. It has no ground truthground truthThe known correct answer you measure against. Without it you are judging vibes. inside it, no fact table to check against, no notion of true versus made-up. A real citation and a fabricated one look equally fluent to a machine whose only target is plausibility. Fluent and correct are different objectives, and the model was only ever trained on the first. So it will always be able to produce confident nonsense. This is not a reason to distrust the tool. It is the reason the later levels exist: retrieval to give it ground truth, guardrails to catch it, evaluation to measure it.

The One Fact That Explains the Rest

Come back to where we started. A language model is a probability machine that guesses the next token, again and again, from a distribution it builds out of geometry and attention over a fixed window of text. Tokenization sets your cost. The context window sets its world. Sampling sets its personality. Hallucination is the shadow the whole design casts.

You do not need the math to be dangerous with this. You need the one fact, held tightly enough that every knob has a home. Once it does, you stop guessing at the model and start steering it.

Check what stuck

1You set temperature to 0. What have you actually bought?

Correct answersThe same answer every time, which is consistency and not correctnessFaster generationA model that stops hallucinating

Check answer

2Why does a long prompt cost more and feel slower even before the answer starts?

Long prompts confuse the modelThe model reads the whole input in the prefill phase before writing a single tokenLonger prompts are sent more slowly over the networkThe context window shrinks as it fills

Check answer

3A model states a wrong fact with total confidence. What is happening?

A bug in the model that a patch will fixThe mechanism working normally: it samples a plausible next token and has no notion of truthThe context window overflowedTemperature was set too low

Check answer