Shipping Agents Is a Level of Its Own

The demo works. That is the trap. A working demo is the most dangerous artifact in AI engineering, because it looks like the finish line and it is barely the start. Shipping agents to production is a separate level of skill, and most projects die in the gap between the two.

You measured it last rung, but a passing harness is not the same as production. You can feel the objection already: the model is hosted, the API is one HTTP call, what is left to ship? Everything that separates a thing that impressed you once from a thing that serves ten thousand strangers on a Tuesday without you watching. This is the last rung of the ladder, and it holds up all the others.

Serving Is Not Calling

When you hit a hosted API, someone else already solved the hard part. Pull that curtain back and you meet the inference engine: the software that runs the model, takes your tokens in, and streams tokens out. Inference is just that forward pass, the model predicting the next token, over and over. The engine people name is vLLM, and it exists because naive serving wastes almost all of your expensive GPU.

Here is the waste. Call the model one request at a time and the GPU sits idle between requests. Throughput, how many requests you finish per second, collapses. Latency, how long one user waits, stays fine for that one user and terrible for everyone in line. The number that ties them together is tokens/sec, and a serving engine exists to push it up.

The main trick is continuous batching, sometimes called dynamic batching. Instead of running requests one after another, the engine packs many into the GPU at once and slots new ones in the moment a slot frees, rather than waiting for a whole batch to finish. The GPU stops idling.

The second trick is prefix caching. Every request in your app probably starts with the same long system prompt, and without caching the engine reprocesses those identical tokens every time. Prefix caching computes them once and reuses the result, so a shared preamble is nearly free after the first call. When the model is too big for one GPU, you split it: tensor parallelism cuts each layer across cards, pipeline parallelism puts different layers on different cards. You will rarely wire this yourself, but know the words when the platform team says them.

Cost Is a Feature You Build

An agent that works and costs too much is a failed agent. Cost optimization is not an afterthought you bolt on later, it is a design constraint you carry from the first line.

Start with token budgeting: know how many tokens each step spends, and cap it. A retrieval loop that stuffs forty documents into context when three would do is not smarter, just expensive and slower. Then caching, the same idea as prefix caching but at your level: identical questions should not pay full price twice. Then rate limiting, which guards against both a runaway loop and a bad actor, because an agent that can call a tool a thousand times a second is a billing incident waiting to happen.

Two efficiency words worth recognising. Quantization shrinks a model by storing its numbers at lower precision, trading a sliver of accuracy for large wins in speed and memory, which is how capable models run on hardware you can afford. FlashAttention is a faster, leaner way to compute attention under the hood. You will not implement either, you will choose models and settings that use them.

You Cannot Fix What You Cannot See

Ship an agent blind and you will learn about every failure from angry users, days late. The cure is observability: the ability to see inside a run after it happened. Its core is tracing, recording every step the agent took, every prompt, every tool call, every result, as one connected timeline you can replay. Telemetry is the raw stream of measurements, latency, token counts, error rates, that feeds the dashboards.

Tracing tells you what happened once. Monitoring watches quality over time, and the thing it watches for is drift detection: the slow rot where inputs shift away from what your agent was built for, or a model update quietly changes behaviour, and yesterday's good answers become today's wrong ones. Nothing crashed. The numbers just bent.

Rolling out changes safely is its own craft. Canary deployment sends the new version to a small slice of traffic first, watches it, and widens only if it holds, so a bad release burns one percent of users instead of all of them. A/B testing runs two versions side by side to measure which is actually better, not which feels better. Model versioning ties every answer to the exact model and prompt that produced it, so when something breaks you know precisely what to roll back. For the deploy pattern itself in its smallest form, I wrote about zero-downtime deploys on a Raspberry Pi: the same moved pointer the big platforms sell you.

Long Jobs Need to Survive Themselves

A chat reply takes two seconds. A real agent job, research across fifty sources, a migration, an overnight pipeline, can take an hour, and in that hour the network will blip, a tool will time out, a machine will restart. If one failure at minute fifty throws away all the work, you do not have a product. You have a lottery.

The answer is durable execution: designing long runs to survive their own failures. It leans on async execution and queue-based processing, where work is dropped onto a queue and picked up by workers, so a crash means the job waits and retries instead of vanishing. It requires idempotency, meaning a step run twice has the same effect as run once, so a retry never double-charges a card. And it requires checkpointing, saving state at each step so a resumed job restarts from minute fifty, not minute zero. This state persistence is the difference between an agent that finishes and one that merely usually finishes.

Frameworks Are the Last Choice, Not the First

You can build all of this from a plain loop and an HTTP client, and for a first agent you should, because a framework you do not understand hides the very mechanics this ladder taught you. But at scale, glue you wrote at 2am becomes glue you debug at 2am. So know the toolkits.

Under most of them sits FastAPI to expose the agent as a service and Hugging Face and LangChain for models and building blocks. The rule is simple. Pick the framework that removes work you understand, never the one that hides work you do not.

The Capstone Is the Whole Ladder, Shipped

Every level of this series was a rung. The capstone is climbing all of them at once, in one build, and getting it to production.

You start with the engine, a model predicting tokens. You talk to it with prompts and structured output. You give it your knowledge with retrieval, tools so it can act, and MCP so the tools plug in cleanly. You make it a single agent that plans and remembers, and only if you truly must, you coordinate several. You wrap it in guardrails and least privilege. You measure it with evals instead of trusting a vibe. And then you serve it, watch it, budget it, and make it durable. Follow the ladder to here and that whole list is yours to build and ship. You now hold the full skill set of an applied AI engineer.

That is the honest shape of the work. The intelligence was never the hard part, the labs handed you that. The hard part is everything around it: the serving that keeps it fast, the observability that keeps it honest, the durability that keeps it alive. Shipping is not the step after building. It is the level where building finally counts, and the one that quietly decides which agents get to exist at all.

---

This is the final post in the Applied AI Engineering series. Start from the top: The Applied AI Engineering Ladder. Previous: Measuring Agents.