There is a reflex in every AI conversation to reach for the largest model available. It is understandable. The frontier models are astonishing, and if you are trying to convince a room that AI can do the job, the biggest model is the safest demo.
It is usually the wrong choice for production. Not because large models are bad, but because the job you are automating is almost never "be brilliant at everything". It is "do this one thing, correctly, ten thousand times a day, for a price that makes sense". That is a different problem with a different answer.
One job, one model
A general model carries the weight of everything it can do, every time you call it. When the job is narrow, most of that weight is wasted. A model a fraction of the size, trained on the job, will often match or beat the general model on that job, and only that job.
That last clause is the whole trade. The small model will be worse at everything else. In a bounded production workflow, everything else is not on the menu.
What you gain
Three things, and they compound.
- Latency. Smaller models answer faster on the same hardware. For a voice agent or anything a person is waiting on, this is the difference between natural and awkward.
- Cost. Serving cost scales with model size. At ten thousand calls a day, the gap between a frontier API and a small model on your own GPUs is a budget line, not a rounding error.
- Ownership. The weights are yours. They sit on your servers, they do not change under you when a provider ships an update, and they do not phone home.
What distillation means
Distillation is how the small model gets good. In plain words: you use a large model to produce a great many worked examples of the job, done well. Then you train the small model to reproduce those examples. The large model is the teacher. It is used once, offline, under your control. The small model is the student, and the student is what goes to production.
It is not magic and it is not free. It takes real examples from your own data, a way to check the teacher's answers, and an evaluation set that tells you when the student is ready. Our engineering team has written about the post-training side of this, where the checks come from code rather than from people.
When a small model is the wrong answer
We would rather say this now than after a pilot.
- Open-ended reasoning. If the job genuinely needs the model to think through a novel problem, use the big one.
- A long tail you cannot enumerate. If the inputs are wildly varied and you cannot collect representative examples, a small model will be brittle exactly where it matters.
- Low volume. If the job runs fifty times a day, the serving savings will not pay for the training. Use an API and move on.
The routing pattern
In practice most of our deployments are neither small nor large. They are both. The small model handles the common cases. When it is unsure, and it is trained to know when it is unsure, the request goes to a larger model, or to a person. The big model becomes an exception path rather than the default, which is where its cost and latency stop mattering.
The question is not "which model is best". It is "which model is best for the ninety per cent, and what happens to the ten".
Side by side
| Frontier model over an API | Fine-tuned small model on your hardware | |
|---|---|---|
| Latency | Depends on the provider and the queue | Determined by your hardware, predictable |
| Cost per call | Per token, for ever | Fixed hardware, near-zero marginal |
| Data residency | Provider's region | Your building, your rules |
| Drift | Provider updates change behaviour | Nothing changes unless you change it |
| Ceiling | Highest available | High on the job, low elsewhere |
| Time to first result | Hours | Weeks, including the data work |
What it takes
Honestly: examples, evals and patience. A few hundred to a few thousand good examples of the job, depending on how varied it is. An evaluation set that the business agrees represents "correct". And a few weeks in which the first versions are not good enough and the team keeps going.
At the end you own a model that does your job, on your servers, at a cost you can forecast. That is a very different asset from a subscription, and it is why we push most clients toward it once the pilot has proved the job is worth doing.