We tuned vLLM by hand for 64% more throughput. An agent should have done it.
In our last post we fine-tuned a 235B model on four GPUs, then never shipped it, because a newer open model was accurate enough out of the box. So this post is about the next step: serving that model in production. We thought this part was easy. Start vLLM, point traffic at it, done. It wasn't. Fine-tuning is a cost you pay once; serving is a cost you pay every week, and getting it right had its own set of problems. This is what we hit, what we learned, and how we ended up handing the work to an agent.
The task is the same kind as before: read a long, structured document from a customer, find the relevant entities, label each one. Prompts average about 6,500 tokens, and the model reasons before it answers, so outputs are long too. We had one machine with 8× H100 80 GB and served everything with vLLM.
We ended up on Qwen3.6 27B in FP8. It takes about 15× less memory than the 397B we started with1 and is just as accurate on this task. Changing how we deployed it got us 64% more output tokens per second on the same eight GPUs, and for the tasks we moved, running them on these GPUs costs 44% less per week than the API bill they replaced.
Most of our time went into finding that setup, one run at a time: change a setting, restart vLLM, run the load test, read the metrics, keep it or revert it. Some of that time was needed. Most was not. Near the end we noticed we had been running by hand the exact loop that Karpathy's autoresearch hands to an agent. The rest of this post is what the hand runs taught us, and why we'd still do the first few by hand.
Start big, then shrink
We started with the largest model that scored well and planned to step down only while accuracy held. We expected accuracy to tell us where to stop.
It never did. Average F1 (the balance of precision and recall, higher is better) stayed within a point of the 397B all the way down to the 27B. What stopped the large models was throughput under load.
| Model | Weights | P99 at 100 concurrent | Average F1 |
|---|---|---|---|
| Qwen3.5 397B (MoE, FP8) | ~397 GB | Not usable | 95.1 |
| Qwen3.5 122B (MoE, bf16) | ~244 GB | 8–10 min | 95.6 |
| Qwen3.6 27B (bf16) | ~54 GB | 2–3 min | 94.7 |
| Qwen3.6 27B (FP8) | ~27 GB | 2–3 min | 95.3 |
These runs are at 100 concurrent requests. At 1,000, which is closer to our production peak, even the 27B needed the changes in the next sections.
These F1 scores are not comparable to the last post's. It is a similar task on a different evaluation set.2
The 9B failed, but not on accuracy. It ignored the thinking budget and kept reasoning until it ran into max_tokens. About 15% of its responses leaked reasoning into the answer and more hit the token limit before finishing. Roughly a quarter of its outputs couldn't be parsed, against 0.2% for the 27B. For this task, 27B was the floor.
The KV cache sets the ceiling
Every in-flight request holds a KV cache: the attention keys and values for every token it has seen so far, kept so they aren't recomputed for each new token. With a 6.5k-token prompt plus a few thousand reasoning tokens, each request's cache is large.
The weights claim GPU memory first. The KV cache gets what's left. That leftover decides how many requests can run at once. The rest wait in a queue, and queue time is most of P99.
The 397B in FP8 needs about 397 GB for weights, which leaves about 240 GB of the 640 GB before vLLM's own overhead. The 122B in bf16 left more room, but P99 under load was still 8 to 10 minutes.
We did not go back and give the 122B the serving changes we later made to the 27B. Some of them might have helped it. It would still have to hold ~244 GB of weights where the 27B holds 27, and the difference is KV cache.
What we watched
vLLM exports dozens of metrics. After a few runs we only looked at four, on a Grafana dashboard:
- KV cache usage. When it hits 100%, vLLM can't start new requests and they pile up in the queue.
- Running vs waiting requests. If the waiting count keeps going up, the server is falling behind.
- P99 latency. The median was often fine while the slowest 1% took many minutes, and those are the ones that hit timeouts in the services calling us.
- Prefix cache hit rate. Our prompts all start with the same long instructions, so about 61% of prompt tokens were reused from earlier requests instead of being computed again.
Every bad run looked the same: KV cache at 100%, waiting queue growing. So most of what we tried next was about giving the KV cache more memory.
Getting the 27B to handle the load
Our load test is a hand-written script that sends 1,400 requests and keeps up to 1,000 in flight at once. Every number in this section is from that test.
Baseline: DP8
The model fits on one GPU, so the obvious start is data parallelism (DP8): eight independent copies of the model, one per GPU, each taking its own share of requests. In bf16 the weights take 54 GB of each 80 GB GPU. KV cache sat at 100%, requests queued, and P99 was 16 minutes.
FP8 weights
FP8 stores each weight in one byte instead of two. We switched to the FP8 checkpoint (Qwen/Qwen3.6-27B-FP8). Weights go from ~54 GB to ~27 GB per copy, and the freed memory goes to KV cache. Accuracy didn't drop (last row of the table above).
TP2 DP4
Each data-parallel copy holds the full weights, so eight FP8 copies is 216 GB of identical weights. Tensor parallelism of 2 (--tensor-parallel-size 2) splits each copy's weight matrices across a pair of GPUs. Four such copies (--data-parallel-size 4) cover all eight GPUs. The weights take 108 GB in total, and the rest, roughly 530 GB before vLLM's overhead, is available for KV cache.
FP8 KV cache and FlashInfer
Storing the KV cache itself in FP8 (--kv-cache-dtype fp8) halves its size, so roughly twice as many requests fit in the same memory. In the same step we switched the attention backend from vLLM's default on H100, FlashAttention 3, to FlashInfer (VLLM_ATTENTION_BACKEND=FLASHINFER).
We measured the baseline and the final config. We did not measure each step on its own, so we can't say how much each one contributed. What we can show is where the memory went:
| Config | Weights per GPU | Left for KV per GPU (before overhead) |
|---|---|---|
| bf16, DP8 | 54 GB | ~26 GB |
| FP8, DP8 | 27 GB | ~53 GB |
| FP8, TP2 DP4 | 13.5 GB | ~66 GB |
| + FP8 KV cache | 13.5 GB | ~66 GB, holding ~2× the requests |
And what it bought:
| bf16, DP8 | FP8 weights + FP8 KV + FlashInfer, TP2 DP4 | Change | |
|---|---|---|---|
| Output tokens/s | 9,642 | 15,834 | +64% |
| P99 latency | 16.0 min | 8.0 min | −50% |
| Time for 1,400 requests | 10.4 min | 6.4 min | −39% |
| KV cache usage | 100% | 80% | 20% headroom |
What didn't help
Before the 1,000-concurrent runs, we tested settings one at a time on a smaller setup: four GPUs (TP2 DP2, FP8 weights), 400 requests at 100 concurrent. Baseline P99 there was 2.03 min.
Two terms for the table. Prefill is the pass over the prompt. Decode is generating the answer one token at a time.
| Setting | Why we tried it | P99 | Result |
|---|---|---|---|
--enable-chunked-prefill | Long prompts might block decode for other requests | 2.01 min | Within noise |
--gpu-memory-utilization 0.95 | More memory for KV cache | 2.05 min | Within noise |
VLLM_USE_FASTOKENS=1 | Faster tokenization | 2.07 min | Within noise |
--no-scheduler-reserve-full-isl | Don't reserve KV cache for the full prompt up front, so more requests get in | 2.02 min | Within noise |
None of these change how much memory the KV cache gets, and at 100 concurrent we weren't short of it yet. Chunked prefill splits a long prompt so it doesn't stall other requests' decoding, but with 61% of prompt tokens served from the prefix cache, the uncached part was short and stalled nothing. Extra memory didn't help because there was spare cache. The faster tokenizer didn't matter because the bottleneck was decoding on the GPU, not work on the CPU.
We also tried different values of --max-num-seqs, the cap on how many requests vLLM runs in one batch. It made no difference on the 27B, because KV memory ran out well before the cap. On the bigger models, setting it too high made things worse: more requests got in than the cache could hold, and vLLM had to evict some and recompute them later. Size it to the memory you have instead of setting it as high as it goes.
The caveat: at 100 concurrent there was spare cache, so memory changes did nothing. At 1,000 concurrent the baseline was out of cache, and most of what helped was about freeing memory for it. We didn't repeat these four tests at 1,000, and they might behave differently there.
Handing the loop to an agent
Every row in every table above came from the same loop. One run at high load takes 6 to 10 minutes plus a restart, and we sat through each one. The knobs we had to choose between:
- model size
- weight precision
- KV cache precision
- how the eight GPUs split between tensor and data parallelism
- attention backend
- scheduler flags
- the load you test at, which changes the answer to all of the above
The combinations multiply, and by hand we covered a small corner of them. This is the problem Karpathy's autoresearch solves for training: give an agent a change to make, a run to wait on, numbers to read, and a rule for keeping or reverting. Serving has the same shape. So late in the project we built one for vLLM. The agent got:
- the load test script, fixed at the load we expect in production
- read access to our Grafana dashboards, so it could read the four metrics after each run
- the list of experiments we still wanted to run
- a results file it had to write after every run: what it changed, what the metrics said, keep or revert
Then it ran the loop on its own: pick an experiment, change the config, restart vLLM, run the load, read the dashboards, record, next. Runs that would have taken our attention for days ran without us, and the results file is the log we wish we had kept from day one.
We could only set this up because of the hand runs. They told us which four metrics to put on the dashboard, that KV cache memory was the thing to free, and what to optimize for: tokens per second, with P99 under a limit and accuracy held. Without that we would have had nothing to tell the agent, and it would have spent GPU hours on flags that didn't matter.
Most of what we learned isn't specific to this task. How vLLM uses memory, what the metrics mean, how a thinking model behaves when it runs out of budget: that will be the same next time. What changes with each task is the data, the load, and which config holds up under it. That part is a search, and the agent is better at it than we are.
Next task, we'll do a handful of runs by hand to learn what's new about it, then hand the search to the agent from the start, with the model as one more knob. We'd also give it two things it didn't have this time: an accuracy and parse check in the loop, so it can't win by trading away correctness, and one written objective, so it isn't guessing what we want. That is the part we should have done first. The 64% was always going to come from the search. We just didn't have to be the ones running it.