Shipping Agents Is a Level of Its Own

The demo works. That is the trap. A working demo is the most dangerous artifact in AI engineering, because it looks like the finish line and it is barely the start. Shipping agents to production is a separate level of skill, and most projects die in the gap between the two.

You measured it last rung, but a passing harness is not the same as production. You can feel the objection already: the model is hosted, the API is one HTTP call, what is left to ship? Everything that separates a thing that impressed you once from a thing that serves ten thousand strangers on a Tuesday without you watching. This is the last rung of the ladder, and it holds up all the others.

Serving Is Not Calling

When you hit a hosted API, someone else already solved the hard part. Pull that curtain back and you meet the inference engineinference engineSoftware built to serve a model efficiently under load, doing far more than forwarding a request.: the software that runs the model, takes your tokens in, and streams tokens out. InferenceInferenceRunning a trained model to get an answer, as opposed to training it. What you pay for per call. is just that forward pass, the model predicting the next token, over and over. The engine people name is vLLMvLLMA widely used open-source inference engine, known for high throughput through smart batching and memory reuse., and it exists because naive serving wastes almost all of your expensive GPU.

Here is the waste. Call the model one request at a time and the GPU sits idle between requests. ThroughputThroughputHow much work the system completes per unit of time, usually tokens or requests per second., how many requests you finish per second, collapses. LatencyLatencyHow long one request takes. The number your user actually feels., how long one user waits, stays fine for that one user and terrible for everyone in line. The number that ties them together is tokens/sectokens/secThe common measure of generation speed, and the one users perceive during streaming., and a serving engine exists to push it up.

The main trick is continuous batchingcontinuous batchingA serving engine adding and removing requests from a running batch instead of waiting for a full one., sometimes called dynamic batching. Instead of running requests one after another, the engine packs many into the GPU at once and slots new ones in the moment a slot frees, rather than waiting for a whole batch to finish. The GPU stops idling.

The second trick is prefix cachingprefix cachingReusing the computed state of an identical opening, so repeated context is not processed again.. Every request in your app probably starts with the same long system prompt, and without caching the engine reprocesses those identical tokens every time. Prefix caching computes them once and reuses the result, so a shared preamble is nearly free after the first call. When the model is too big for one GPU, you split it: tensor parallelismtensor parallelismSplitting one model across several GPUs so a model too big for one card can run at all. cuts each layer across cards, pipeline parallelismpipeline parallelismSplitting a model by layer across devices, each handling one stage of the forward pass. puts different layers on different cards. You will rarely wire this yourself, but know the words when the platform team says them.

Cost Is a Feature You Build

An agent that works and costs too much is a failed agent. Cost optimizationCost optimizationReducing spend without giving up quality, most often through caching, model choice and shorter context. is not an afterthought you bolt on later, it is a design constraint you carry from the first line.

Start with token budgetingtoken budgetingDeciding in advance how many tokens a task may spend, so no single run can run away.: know how many tokens each step spends, and cap it. A retrieval loop that stuffs forty documents into context when three would do is not smarter, just expensive and slower. Then cachingcachingStoring work already done so it does not have to be repeated. Usually the cheapest cost lever available., the same idea as prefix caching but at your level: identical questions should not pay full price twice. Then rate limitingrate limitingCapping how often a caller may hit something, to protect both cost and availability., which guards against both a runaway loop and a bad actor, because an agent that can call a tool a thousand times a second is a billing incident waiting to happen.

Two efficiency words worth recognising. QuantizationQuantizationUsing lower-precision numbers so a model is smaller and faster, at a small cost in accuracy. shrinks a model by storing its numbers at lower precision, trading a sliver of accuracy for large wins in speed and memory, which is how capable models run on hardware you can afford. FlashAttentionFlashAttentionA faster, memory-efficient implementation of attention that made longer contexts practical. is a faster, leaner way to compute attention under the hood. You will not implement either, you will choose models and settings that use them.

You Cannot Fix What You Cannot See

Ship an agent blind and you will learn about every failure from angry users, days late. The cure is observabilityobservabilityBeing able to tell what a running system is doing, and why, without guessing.: the ability to see inside a run after it happened. Its core is tracingtracingRecording every step of one run so you can see the path a decision actually took., recording every step the agent took, every prompt, every tool call, every result, as one connected timeline you can replay. TelemetryTelemetryThe measurements a running system emits: latency, cost, errors, token counts. is the raw stream of measurements, latency, token counts, error rates, that feeds the dashboards.

Tracing tells you what happened once. MonitoringMonitoringWatching those measurements over time and being told when something moves. watches quality over time, and the thing it watches for is drift detectiondrift detectionNoticing that behaviour or inputs have changed since the thing was last known to be good.: the slow rot where inputs shift away from what your agent was built for, or a model update quietly changes behaviour, and yesterday's good answers become today's wrong ones. Nothing crashed. The numbers just bent.

Rolling out changes safely is its own craft. Canary deploymentCanary deploymentSending a small slice of traffic to a new version first, so a bad release is caught small. sends the new version to a small slice of traffic first, watches it, and widens only if it holds, so a bad release burns one percent of users instead of all of them. A/B testingA/B testingRunning two versions against real traffic and comparing outcomes rather than guessing. runs two versions side by side to measure which is actually better, not which feels better. Model versioningModel versioningPinning and tracking which model version served which traffic, so a change is attributable. ties every answer to the exact model and prompt that produced it, so when something breaks you know precisely what to roll back. For the deploy pattern itself in its smallest form, I wrote about zero-downtime deploys on a Raspberry Pi: the same moved pointer the big platforms sell you.

Long Jobs Need to Survive Themselves

A chat reply takes two seconds. A real agent job, research across fifty sources, a migration, an overnight pipeline, can take an hour, and in that hour the network will blip, a tool will time out, a machine will restart. If one failure at minute fifty throws away all the work, you do not have a product. You have a lottery.

The answer is durable executiondurable executionRunning long jobs so they survive crashes and restarts, resuming instead of starting over.: designing long runs to survive their own failures. It leans on async executionasync executionStarting work without waiting for it, so slow steps do not block everything behind them. and queue-based processingqueue-based processingPutting work on a queue so producers and workers scale independently and nothing is lost., where work is dropped onto a queue and picked up by workers, so a crash means the job waits and retries instead of vanishing. It requires idempotencyidempotencyMaking an operation safe to repeat, so a retry cannot charge twice or send twice., meaning a step run twice has the same effect as run once, so a retry never double-charges a card. And it requires checkpointingcheckpointingRecording progress at points along a long run so failure resumes rather than restarts., saving state at each step so a resumed job restarts from minute fifty, not minute zero. This state persistencestate persistenceSaving agent state so a run can survive a restart or be resumed later. is the difference between an agent that finishes and one that merely usually finishes.

Frameworks Are the Last Choice, Not the First

You can build all of this from a plain loop and an HTTP client, and for a first agent you should, because a framework you do not understand hides the very mechanics this ladder taught you. But at scale, glue you wrote at 2am becomes glue you debug at 2am. So know the toolkits.

Under most of them sits FastAPIFastAPIA Python web framework commonly used to put an HTTP interface in front of a model or an agent. to expose the agent as a service and Hugging FaceHugging FaceThe main hub for open models, datasets and the libraries used to run them. and LangChainLangChainAn early and widely used toolkit of components for chaining models, tools and retrieval together. for models and building blocks. The rule is simple. Pick the framework that removes work you understand, never the one that hides work you do not.

The Capstone Is the Whole Ladder, Shipped

Every level of this series was a rung. The capstone is climbing all of them at once, in one build, and getting it to production.

You start with the engine, a model predicting tokens. You talk to it with prompts and structured output. You give it your knowledge with retrieval, tools so it can act, and MCP so the tools plug in cleanly. You make it a single agent that plans and remembers, and only if you truly must, you coordinate several. You wrap it in guardrails and least privilege. You measure it with evals instead of trusting a vibe. And then you serve it, watch it, budget it, and make it durable. Follow the ladder to here and that whole list is yours to build and ship. You now hold the full skill set of an applied AI engineer.

That is the honest shape of the work. The intelligence was never the hard part, the labs handed you that. The hard part is everything around it: the serving that keeps it fast, the observability that keeps it honest, the durability that keeps it alive. Shipping is not the step after building. It is the level where building finally counts, and the one that quietly decides which agents get to exist at all.

Check what stuck

1What does a serving engine do that simply calling an API does not?

Nothing, it is the same thingIt manages batching, memory and scheduling across many concurrent requests to raise throughputIt makes the model more accurateIt writes the prompts for you

Check answer

2Your agent gave a wrong answer in production. What lets you find out why?

The error logA trace of that run: every step, tool call and result, in orderThe model version numberThe total token count

Check answer

3When should you reach for an agent framework?

First, before writing anythingLast, once you understand the loop well enough to know what the framework is doing for youNever, frameworks are always wrongWhenever the team is unfamiliar with agents

Check answer