In our last post we fine-tuned a 235B model on four GPUs, then never shipped it, because a newer open model was accurate enough out of the box. So this post is about the next step: serving that model in production. We thought this part was easy. Start vLLM, point traffic at it, done. It wasn't. Fine-tuning is a cost you pay once; serving is a cost you pay every week, and getting it right had its own set of problems. This is what we hit, what we learned, and how we ended up handing the work to an agent.

The task is the same kind as before: read a long, structured document from a customer, find the relevant entities, label each one. Prompts average about 6,500 tokens, and the model reasons before it answers, so outputs are long too. We had one machine with 8× H100 80 GB and served everything with vLLM.

We ended up on Qwen3.6 27B in FP8. It takes about 15× less memory than the 397B we started with[^1] and is just as accurate on this task. Changing how we deployed it got us 64% more output tokens per second on the same eight GPUs, and for the tasks we moved, running them on these GPUs costs 44% less per week than the API bill they replaced.

Most of our time went into finding that setup, one run at a time: change a setting, restart vLLM, run the load test, read the metrics, keep it or revert it. Some of that time was needed. Most was not. Near the end we noticed we had been running by hand the exact loop that Karpathy's autoresearch hands to an agent. The rest of this post is what the hand runs taught us, and why we'd still do the first few by hand.

We had been using fine-tuned OpenAI and Gemini models to classify entities in our customers' documents. Accuracy was fine. The monthly bill kept growing, so we wanted to know whether we could fine-tune an open model ourselves, on hardware we control, and do at least as well.

We could. One epoch of fine-tuning on about 3,600 examples improved accuracy for every customer we measured, by 7.6 F1 points on average, at about a third of the per-token cost of the API models. The run took 26 hours on four H100s.

We never shipped it. That is the second half of this story. The first half is the five training setups we went through, four of which failed before a single training step ran, because the reason surprised us and it applies to anyone trying to fine-tune a model this size on one machine.

If you ship a multi-gigabyte ML container — Hugging Face weights baked in, PyTorch + transformers, a FastAPI server — and you've watched the deploy take five-plus minutes from docker run to first inference, this post is the playbook we wish we had.

The AI media generation space moves fast. Every few weeks there's a new model worth trying: better motion, better prompt adherence, better pricing. So we tried them all.

Every switch meant the same work: different SDK, different parameter names, different polling logic, different error format. duration_seconds on one provider, duration on another, an enum string on a third. Each integration took a day. Each one was slightly different from the last in ways that didn't matter but had to be handled anyway.

And when a provider went down, the pipeline went with it. No fallback, no graceful degradation. Just errors until someone manually switched.

We weren't solving a new problem each time. So we built Tarash Gateway. We're Vertexcover, and we build tooling for teams working with AI-generated media.

If you've ever worked with AI-generated videos, you know the problem. The content is great, but the endings are... weird. The avatar freezes, stares blankly into the soul of the viewer, or cuts off mid-breath.

To make these videos production-ready, you have to manually trim that awkward tail. For one video, it's fine. For 1,000 personalized outreach videos? It's a nightmare.

We wanted to automate this. But simple tools like ffmpeg or basic silence detectors aren't enough. They don't "see" the video. They don't know if an avatar's face looks unnatural.

So we built SceneFlow.

Full source code: github.com/vertexcover-io/linkedin-spider