The author of AI Engineering shared her inference optimization strategies a year ago. Here’s how it all aged.
P99 CONF 2025 featured a cracking keynote from Chip Huyen, author of the bestselling AI Engineering book from O’Reilly. She opened with some simple maths: training a frontier model is a one-off cost, but inference is the same cost paid over and over. That’s great for the frontier model providers (here’s looking at you Anthropic/OpenAI) and I guess bad for us token burners… So, over the life of a model, she reckons the compute split ratio lands somewhere between 1:10 and 1:100 in terms of training to inference, and reasoning models (which burn even more tokens) push that out even further. We all know how that feels when we hit our weekly session quotas 🙂
Anyway, her point was that if inference is too expensive, then nobody ever recovers the training bill (which I guess is why there are tons of memes out there of the ‘profitability’ of frontier models). So, how do we optimize it?
That’s a topic that Chip spent months researching for her book. And in the spirit of optimization, she distilled it down into < 30 minutes for the conference. Watch the full talk – or read the recap below – then let’s talk about how it all aged.
Note: Chip is returning for P99 CONF 2026 – a free + virtual conference on all things performance. Grab a complimentary pass and join us October 21 + 22 to chat with Chip.
Grab your conference pass — it’s free + virtual
What to measure
Chip recommends focusing on a few key latency metrics:
- Time to first token (TTFT): How much time elapses before the user sees anything
- Time per output token (TPOT): The average time between consecutive tokens (aka inter-token latency)
- End-to-end latency: Time to first token, plus time per output token, multiplied by the number of output tokens minus one
With reasoning models, some of those tokens never reach the user. “The first generated token might not be the same as the first visible token,” she explained. “The model might think for a while, and it will only show the first token of the final output to the user.” Chip noted that some people also measure Time to Publish for that (i.e., how long until the user sees the first token). The best metric to prioritize depends on what matters most for your users.
Also, consider “goodput” as well as throughput. Throughput measures requests processed in a given window. Goodput measures the requests that actually met your targets. Chip’s example: an app targets 200 ms time to first token and 100 ms time per output token, and processes 10 requests per minute, but only three hit both.
3 ways to optimize LLM inference
With inference servers, you can optimize from 3 different angles: the hardware, the model, and the service that manages the requests and responses.
Chip, who previously worked at NVIDIA, opted out of the hardware discussion. “Even though I find it to be an intellectually interesting topic, it’s not relevant to a lot of people because we don’t have the power to change the hardware itself,” she explained. She also didn’t want to spend much time on the obvious solution: replica parallelism, or just adding more machines. It’s costly, and it gets complicated fast – especially if you end up with a mix of 80GB, 48GB and 24GB machines and models of varying sizes to distribute across them.
That leaves the model and the service. Chip’s tips on how to decide: “If you want to host the models yourself, or if you have access to the model weights, or if you train a model yourself, or you want to fine-tune or distill a model, then model optimizations might be for you. However, if you want to take a model as is and try to make it more efficient on your own inference service, then you might want to go into service optimizations.”
Model optimization
The following techniques change the actual weights, so they can change the model outputs.
Quantization lowers the precision used to store weights and activations (e.g., from four bytes per parameter at 32-bit to one byte at 8-bit). Chip explained, “Reducing the precision not only reduces the memory requirement to run the model, making it cheaper. It can also make the model a lot faster. If you do additions bit by bit and each weight is 32 bits, you have to do it 32 times. If it’s 8 bits, you only have to do it eight times.”
The tradeoff is a small quality hit. She continued, “It’s possible to reduce a lot of the model’s memory footprint with minimal quality degradation, and quantization has been shown to be pretty generalizable to a wide variety of model architectures and model sizes. That’s why it’s very popular. I rarely see any companies running a model at full precision anymore.”
Distillation involves using a large model to generate training data for a smaller model. For example, say you have a truly large model (the example Chip used was o1) and want a model that performs like it, but is much smaller. Basically, you collect a large set of prompts, run them through the larger model, then train the smaller model on its responses.
Proceed with caution, though. Chip warned, “A lot of model providers have the condition that they do not allow their models to be used to train competitive models. So even though it’s a very common technique, you need to check licensing.”
Service optimization
This set of techniques targets how requests are scheduled, routed and reused. The actual weights aren’t impacted.
Batching groups multiple requests so they’re processed together in a single pass through the model – which is much more efficient than dealing with them one at a time. Chip presented a few batching options:
- Static batching waits for the batch to fill. This maximizes compute utilization, but it might increase the latency for the first requests.
- Dynamic batching runs on a timer instead (e.g., batching every 15 ms). This is less compute-efficient, but it’s better for latency.
- Continuous batching handles the case where requests finish at wildly different times, which is common with LLMs. Say one request asks for the capital of Vietnam, another kicks off deep research. With static or dynamic batching, the finished request’s slot sits idle until the slowest one completes – and new requests queue up behind it. Continuous batching returns each request as it finishes and fills the spot with another request. That can improve compute resource utilization as well as latency.
Decoupling prefill and decode separates the two phases of a request onto different machines. (Prefill processes the input, while decode generates the output.) Chip said, “Input tokens can be processed in parallel, whereas output tokens need to be generated sequentially. With parallel processing, it’s bounded by the compute, the processing power of the chip. With decoding, it’s bounded by memory, because you have to move model weights.” Since both phases stress different resources, most services now separate them. If you want to improve time to first token, shift machines toward prefill. If you care more about improving time per output token, shift them to decode.
Parallelism splits work across machines. Replica parallelism copies the whole model onto more machines. Tensor parallelism divides a very large matrix so different machines compute different parts of it. Pipeline parallelism divides the model by layer so requests move through as a pipeline.
Prompt caching processes shared text just once, which saves on cost and latency. There’s a lot of repetition across requests to the same application: the system prompt, the examples, the same code base, the same document behind different questions. You might as well process that shared segment once, cache it, and reuse it.
The technique was relatively rare when Chip was writing AI Engineering. “There was one paper about it, and it was not really known, but it made a lot of sense. So I included prompt caching in the book, and I’m very happy to see that nowadays it’s pretty much everywhere.”
The savings scale depending on how much of your prompt gets cached. In Claude Code logs, Chip’s open source tool Sniffly found cache hit rates at 90% to 97%. Some providers rewrite prompts internally to improve hit rates, but you might as well structure them yourself. Her tip: since caching works on shared prefixes, make sure the stable parts of your prompt come first and the variable parts come later. “It’s pretty easy to do, and it can improve your application performance significantly,” she noted.
Evaluating inference providers
Chip closed with a warning for anyone evaluating inference providers: “There are many inference companies that provide inference optimizations for models you want to use, and a lot of them advertise just cost and latency. But pay attention to how many inference optimization techniques also change the model behavior, or reduce the model quality. So when evaluating an inference service, it’s important to look not just at cost and latency, but also at model quality. Does this model, provided on this service, also perform similarly on standard benchmarks?”
What’s changed one year later?
So where do we stand today, one year on from this keynote? Most of it actually aged quite well.
On the economics, I reckon she was bang on… I think for the majority of us as users, we don’t have all the cost levers to pull that Chip has outlined. But it’s great to understand what is happening. And as a novice local LLM user myself, I found I could relate to her points on parallelism (I don’t have it) and prompt caching / quantization (within my grasp of control). Prompt caching (which Chip said was new when she wrote the AI Engineering book) is now priced into every bundle purchase of API tokens. And her Claude Code observation (90% cache hit rates) is probably the reason us mere mortals can still afford agentic coding agents at all.
Some of it did age in ways that would have been difficult to predict at the time. Chip mentioned how reasoning models make inference even more significant. One year on, I think agents running multi-step loops with tool calls have really turned that idea from a footnote to turning the rate limits on Claude (and their infamous 99.x% availability) behavior on its head. All the metrics she described – time to first token, time to publish, goodput under a latency SLO etc, – are all now part of the lingo and probably need to be reasoned about differently.
That’s one thing I hope she’s talking about this year! Grab a free conference pass and join us online on October 21 and 22.
