Prompt Engineering Is Software Engineering
You open a chat window and type a request. It feels like conversation. It is not. A prompt is an interface to a machine, and like every interface it has layers, rules, and failure modes. The moment code on the other end has to read the reply, prompt engineering stops being writing and becomes software engineering. This post is about that line, and about crossing it on purpose.
In the last post the model was a probability machine that predicts the next token. Now we drive it.
The System Prompt Is Standing Law
Every call to a model carries messages, and messages have roles. The system prompt sets the rules: who the model is, what it must and must not do, the format it must answer in. The user prompt is the request the model answers under those rules. Same model, two channels, and they do not carry equal weight.
The system prompt dominates. It is read as standing law, applied to every user turn in the conversation, and the model is trained to treat it as the higher authority when the two conflict. Put "answer only in JSON" in the user turn and a clever question can talk the model out of it. Put it in the system prompt and it holds across the whole session. This is role prompting: you assign the model a role and a boundary up front, and everything after inherits it.
Rule of thumb: stable rules go in the system prompt, the variable request goes in the user prompt. Mixing them is the first bug.
Examples Are Not Free
The cheapest prompt gives no examples. You state the task and trust the model to know the shape of a good answer. That is zero-shot, and for common tasks it is often enough.
When the task is unusual, or the format is fussy, you show it. One worked example is one-shot. A handful is few-shot. Examples are the most reliable way to pin down a format or a tone, because the model copies the pattern it sees.
But examples ride in the context window, and every token you spend on them is a token you pay for on every call. So examples have to earn their keep. Add them when zero-shot fails on the format, not as a reflex. If two examples fix the problem, do not paste six.
Make It Show Its Work
Ask a model for the answer to a multi-step problem and it will often blurt a wrong one, because it is drawing each token from a fresh guess with no room to reason. Give it that room. Chain of thought is a single instruction, "work through it step by step," and it visibly lifts accuracy on anything with intermediate steps. The model reasons in the open, then lands the answer.
Structure buys you control too. Wrapping parts of a prompt in tags, <context>, <task>, <format>, is XML prompting, and models parse those boundaries cleanly because their training is full of them. It is the simplest form of structured prompting: you stop writing a paragraph and start writing fields.
One Big Prompt Is a Bad Program
The instinct is to cram the whole job into one giant prompt. Resist it. A long prompt that classifies, extracts, summarises, and formats in one pass fails in ways you cannot debug, because you cannot see which stage broke.
Split it. Task decomposition breaks the job into ordered steps. Prompt chaining runs them in sequence, each prompt's output feeding the next, so classify then extract then format become three small calls you can test in isolation. And the parts that repeat become prompt templates: a fixed skeleton with slots you fill per call, the same way you would never inline a SQL string you could parameterise.
This is where it starts to look like code, because it is code. A prompt with slots is a function. A chain of prompts is a pipeline.
The Real Product Is Output Your Code Can Parse
Here is the line. As long as a human reads the answer, prose is fine. The instant a program reads it, prose is a liability. Your code cannot act on "Sure, the total comes to about 42 dollars, roughly." It needs {"total": 42}.
Structured outputs are the answer: you constrain the model to return machine-readable data, not free text. JSON mode is the blunt version, "reply with valid JSON," and it gets you parseable output but not necessarily the right shape.
For the right shape you supply a JSON Schema: an explicit contract naming every field, its type, and which are required. Modern APIs take the schema and constrain the reply to conform to it, which turns "please format nicely" into an enforced structure. Then you run validation on your side anyway, checking the parsed object against the schema before you trust it, because a contract you do not check is only a wish.
Function Calling Is the Seed of Every Agent
Structured output pointed inward: shape the answer. Turn it outward and you get the most important mechanism in this whole series. Function calling lets you hand the model a menu of functions your code exposes, each with a name and a JSON Schema for its arguments. Instead of answering in prose, the model replies with a structured request: call get_weather with {"city": "Berlin"}. Your code runs the real function and hands the result back.
The model never runs anything. It only asks, in a format you can parse, and your code decides. Tool calling is the same idea under a different name. This one mechanism, structured output wired to real functions, is the seed from which every agent in the later posts grows. An agent is this loop, run until the job is done.
Assume It Will Break
The model is still a probability machine, so sometimes the JSON comes back malformed: a trailing comma, a hallucinated field, prose wrapped around the braces. Parsing the output is not a formality, it is a trust boundary, and you treat model output exactly like untrusted user input.
So you build the loop the schema demands. Parse. Validate. If it fails, do not crash, feed the error back and ask again. That is retry logic, and it is ordinary defensive engineering: the same posture you would take toward any input you did not generate yourself.
Prompt Engineering Ends Where Code Begins
Prompt engineering has a reputation as a soft skill, phrasing tricks and magic words. The reputation is half true and getting less true every month. The system prompt, the examples, the chain of thought: those are the interface's ergonomics, and they matter. But the product is the contract, a schema your code can enforce, a function the model can request, a retry loop for when it does not comply.
A prompt is an interface. Design the loose half well. Engineer the half that code has to read, because that half is not writing at all. It is software.
But a perfectly engineered prompt still hits one wall: the model can only answer from what it already knows. The next rung tears that wall down, handing the model your own data so it can answer from facts it was never trained on.
Part of the Applied AI Engineering ladder. Previous: How a Language Model Works. Next: Giving Models Your Knowledge.