Running Your Own Model
Everything in this course so far assumed a model you rent. You hold an API key, someone else holds the hardware, and the bill arrives monthly.
There is another option, and it is now genuinely practical. Open-weight models can be downloaded and run on machines you control, on a laptop for experiments or on a GPU server in production. The capability gap against the best hosted models is real, and it is much smaller than it was, and for many jobs it stopped mattering.
The question is not whether you can. You can, this afternoon. The question is whether you should, and that has three parts: what it actually costs, what problem it actually solves, and what the licence actually permits. Most people answer the first, skip the third, and are surprised later.
In short: Local models are easy to start and are an operations commitment to run. Self-host for data control, which no contract fully solves, or for sustained high volume where a reserved GPU beats per-token pricing. Read the licence before you build, because open weights is not open season.
Open Weights Is Not Open Source
Start with the words, because they are used loosely and the difference has teeth.
Open weights means the trained parameters are downloadable. You can run the model, inspect it, fine-tune it, and deploy it on your own hardware. What you usually do not get is the training data or the training code, so you cannot reproduce it. That is why open weights is not the same as open source, whatever the announcement post says.
What you may do with them is decided by the licence, and licences vary far more than people assume. Some are genuinely permissive, close to Apache or MIT, and you can do essentially anything including selling a product built on them. Others are custom, written by the lab, and carry real conditions: restrictions above a user threshold, naming and attribution requirements, prohibited use categories, or terms that limit certain commercial deployments.
Read it before you build, not before you launch. The licence question is cheap on day one and structural on day two hundred, when the model choice is welded into a product you are selling. This is the single most skipped step in self-hosting, and it is the only one with a legal consequence.
Local First, Because It Is Nearly Free to Try
Before any infrastructure decision, run one on your own machine. It takes minutes and it corrects most of your intuitions at once.
Ollama is the easiest path: install it, pull a model, and you have a local endpoint that speaks a familiar API shape. llama.cpp is the layer underneath much of this world, focused on running models efficiently on ordinary hardware including CPUs and Apple silicon, and it is what makes a laptop a viable place to run a small model at all.
Two things become obvious within an hour of doing this, and neither is obvious from reading about it.
Quantization is the deciding factor. From the vocabulary chapter, it means using lower-precision numbers so the model takes less memory. In practice it is what decides whether a model fits on your machine, and the trade is real: smaller and faster, slightly worse. On a laptop it is not optional, it is the whole reason the thing runs.
Small models are better than you expect at narrow jobs and worse than you expect at open ones. Classification, extraction, routing, reformatting: often fine. Multi-step reasoning across a long context: not close. That distinction is the whole self-hosting strategy in one sentence.
What It Actually Costs in Production
A laptop demo is not a deployment. Here is the shape of the real bill, and the money is the smaller half of it.
A GPU costs the same whether you use it or not. That is the whole economic difference. An API bills per token, so a quiet week costs nothing. A reserved GPU bills every hour, awake or idle. Self-hosting wins above a sustained volume threshold and loses badly below it, and most products are far below it for far longer than they expect.
You are buying operations, not just compute. Someone owns capacity planning, model updates, the serving engine, and the pager. The serving layer itself is covered in Shipping Agents, and running it is a job rather than a setup step.
Quality work moves to you. With an API, the provider improves the model and you inherit it. Self-hosted, your model is exactly as good as the day you deployed it until you do the work to change that.
The Three Reasons That Hold Up
Strip out the enthusiasm and three reasons survive.
| Reason | Does it hold? |
|---|---|
| Data cannot leave our infrastructure | Yes, and it is the strongest. A contract limits what a provider does with your data. Self-hosting removes the transfer. If a regulator or a customer requires that, this is the answer, and it connects directly to What You Must Not Send |
| Sustained high volume on a narrow task | Yes, with arithmetic. A small model doing one classification job at constant volume can beat per-token pricing clearly. Work out the crossover before, not after |
| Latency and availability under our control | Sometimes. No third-party rate limit, no provider incident, no deprecation notice. Real, and you have replaced their outage with yours |
| It is cheaper | Rarely, on its own. True at scale, false at the volume most teams actually have |
| Independence from a vendor | Weak as a reason to start. Keep the model boundary clean in your code and you can move later, which costs nothing today |
The pattern: self-host for a constraint, not for a preference. Constraints do not change with the next model release. Price and capability do, constantly, and building infrastructure around a number that moves every quarter is how you end up maintaining a cluster to save money you are no longer saving.
The Realistic Answer Is Both
Framing this as hosted against self-hosted is the mistake. Most mature systems run a mix, and the split follows the shape of the work.
Small local models take the high-volume narrow jobs: classification, routing, extraction, the calls that happen thousands of times and need to be fast and cheap. The hosted frontier model takes the hard reasoning, the small number of calls where being right matters more than being cheap, which is the same argument as test-time compute made from a different direction.
So keep the boundary clean. One interface, more than one model behind it, and the freedom to move a task from one to the other when the numbers change. That is worth more than either answer, because the numbers will change.
Run one locally this week. Not to migrate, but so the trade stops being theoretical, and so you can tell whether the answer for your workload is a laptop, a cluster, or the API you already have.