A Language Model Is a Probability Machine
A language model is a probability machine. Not a brain, not a database, not a little person trapped in a server. Understanding how a language model works starts and ends with one fact: it takes text and it guesses what comes next. Everything you will ever build on top of it is steering that guess.
Hold that fact. Every knob you touch later, temperature, context length, the cost on your bill, the moment it invents a fact, traces straight back to it. This post is the companion to What an LLM Actually Is: that one names the engine, this one opens the casing and shows you the parts.
It Predicts One Token, Then Does It Again
The core move is next-token prediction. You hand the model some text. It scores every possible continuation and tells you how likely each one is. A runtime picks one, sticks it on the end, and feeds the whole thing back in. That single pass through the model is called inference, and a full reply is just inference run a few hundred times in a loop.
There is no second mechanism. Chat, code, reasoning, agents: all of it is this loop, dressed differently. Once you accept that, the model stops being magic and starts being a thing you can reason about.
Text Becomes Tokens, And Tokens Are What You Pay For
The model does not read letters or words. It reads tokens, chunks of text that are usually a short word or a fragment of a longer one. Turning your text into these chunks is tokenization, and the most common method, BPE (byte-pair encoding), builds its fixed vocabulary by merging the most frequent character pairs until it has around a hundred thousand pieces.
This is not a detail for later. Token count is the meter on everything. You pay per token, in and out. The context limit is counted in tokens. Latency scales with them. A word like the is one token; a rare technical term or a long German compound can be five. That is why the same idea can cost twice as much in one phrasing as another, and why trimming a prompt is real money, not tidiness.
Meaning Is Geometry
A token ID is just a number, and numbers carry no meaning. So the first thing the model does is turn each token into an embedding, a long list of numbers that places the token as a point in a high-dimensional vector space.
The trick is what that space encodes. Tokens with related meanings land near each other. king sits close to queen, Berlin close to Paris, and directions in the space line up with concepts. Meaning becomes distance and direction, something you can measure. Hold onto this picture. It is the exact idea that later powers retrieval and memory, where you find relevant text by finding nearby points.
Attention Is How It Reads Relationships
A pile of word meanings is not understanding. In the bank of the river and money in the bank, the token bank means opposite things, and the difference lives entirely in the words around it. Attention is the mechanism that lets each token look at the others and pull in the context it needs.
No math here, just the intuition. For every token, the model asks: which other tokens matter for what this one means right now? It weighs them and mixes them in. It does this with three vectors it derives for each token, together called QKV: the query is what a token is looking for, the key is what every other token advertises, and the value is what gets mixed in when a query and key match. When tokens in one sentence attend to each other, that is self-attention. When one stream of tokens attends to a separate one, the way a translation looks back at its source sentence, that is cross-attention. And the model does not do this once. It runs many attention passes side by side, each free to track a different kind of relationship, one following grammar, another following who-did-what. That is multi-head attention. Attention is why a Transformer reads a whole passage as a web of relationships instead of a flat string of words.
The Context Window Is Its Whole World
Everything the model can see at once lives in the context window, measured, of course, in tokens. Your system prompt, the conversation so far, retrieved documents, all of it competes for the same fixed budget. Nothing outside the window exists to the model. It has no memory between calls beyond what you put back in.
Two mechanics fall out of this. Attention treats tokens as a set, so order has to be added deliberately through positional encoding, which stamps each token with its place in the sequence. The method most current models use is RoPE, rotary positional encoding, which encodes position by rotating each token's vector by an amount that grows with its place, so the model reads distances between tokens consistently as the sequence gets long. And to avoid re-computing the whole history on every step, the model caches the attention work it already did in a KV cache. That cache is why generation speeds up after the first tokens, and why a long prompt is slow and expensive twice over: more tokens to process, and a bigger cache to carry. Context is not free storage. It is the most contested resource you manage.
Every Knob Is a Way to Pick the Next Token
Here is where the "probability machine" claim earns its keep. The model's raw output is a vector of logits, one unbounded score per token in the vocabulary. Scores are not probabilities, so softmax squeezes them into numbers between zero and one that sum to one. Now you have a real distribution over what comes next.
Every sampling control you have ever set just reshapes or reads this one distribution.
Temperature scales the logits before softmax. Low temperature sharpens the distribution toward the top choice, giving safe and repetitive output. High temperature flattens it, giving variety and risk. Top-k keeps only the k highest-scoring tokens and ignores the rest. Top-p, or nucleus sampling, keeps the smallest set of tokens whose probabilities add up to p, so the pool grows when the model is unsure and shrinks when it is confident. Greedy decoding skips the dice entirely and always takes the single most likely token. Beam search goes further, tracking several high-probability continuations at once and extending the best few before it commits, spending compute to find a sequence more likely as a whole rather than token by token.
None of these change what the model knows. They only change how you draw from the same probabilities. That is the whole game.
Prefill Reads, Decode Writes
Generation runs in two phases, and knowing them explains your latency. Prefill is the model reading your entire prompt in one parallel pass to fill that KV cache. Decode is the loop that follows, producing one token at a time, each depending on the last. Prefill is fast per token because it is parallel; decode is slower because it is sequential. This is why time-to-first-token and time-per-token are different numbers, and why streaming tokens back to the user feels responsive: you are watching the decode loop, live. Speculative decoding speeds that loop up: a small, fast model drafts several tokens ahead and the large model verifies them in one pass, so you get the big model's quality at closer to the small model's speed.
Hallucination Is Not a Bug
Now the payoff. When a model states something false with total confidence, that is a hallucination, and it is structural, not a defect you can patch out.
Look at what the machine actually does. It samples a plausible next token from a probability distribution. It has no ground truth inside it, no fact table to check against, no notion of true versus made-up. A real citation and a fabricated one look equally fluent to a machine whose only target is plausibility. Fluent and correct are different objectives, and the model was only ever trained on the first. So it will always be able to produce confident nonsense. This is not a reason to distrust the tool. It is the reason the later levels exist: retrieval to give it ground truth, guardrails to catch it, evaluation to measure it.
The One Fact That Explains the Rest
Come back to where we started. A language model is a probability machine that guesses the next token, again and again, from a distribution it builds out of geometry and attention over a fixed window of text. Tokenization sets your cost. The context window sets its world. Sampling sets its personality. Hallucination is the shadow the whole design casts.
You do not need the math to be dangerous with this. You need the one fact, held tightly enough that every knob has a home. Once it does, you stop guessing at the model and start steering it.
Part of the Applied AI Engineering series. Start at The Applied AI Engineering Ladder. Previous: The Applied AI Engineering Ladder. Next: Talking to Models.