I posted this as a thread on X and someone asked for a longer version, so here it is. I tried to sort out for myself how I’d do post-training inside a company, more or less in order: why bother at all, how to measure, which data, and which model. It’s long, and I’m not apologizing.
First, what is it? Post-training is basically everything you do to a model after pretraining to make it behave the way it should, or in our case to adapt it to your problem: fine-tuning, distillation, RL, etc. There are tons of ways to do it. I’ll say a bit about them at the end, but the methods are not the main point here. If you want more on the methods, I talked about them with Maxime Labonne on the podcast.
Why not just send everything to OpenAI?
I think this is the first question, and you should have an actual answer to it. There are a few reasonable ones: it’s cheaper, it’s faster, the data has to stay with you, or you think your data will get better performance on your task.
The reason also tells you what counts as a win. For cost or speed you only need to match the big model, you don’t have to beat it. When the data can’t leave you might even live with a slightly worse one. The hard case is when you think your data will do better, because then you really do have to beat the API.
If you don’t have one of these reasons, keep using the API. That’s fine.
Most companies aren’t there yet. In LangChain’s agent survey most organizations don’t fine-tune at all. a16z’s CIO survey says the big enterprises mostly do prompt engineering and routing between models, and it’s the startups that ship fine-tuned open models.
The numbers are also going down. In Theory Ventures’ 2026 survey, the share of teams that ever fine-tuned fell from 81% to 51% in a year, and OpenAI’s fine-tuning product dropped even more. Homegrown tooling went the other way, from 7% to 20%.
The obvious explanation is that the base models got good enough and fine-tuning stopped paying off. Maybe, for the hosted products. But homegrown tooling almost tripled in the same year, so I don’t think that’s the whole story. As I see it, the decline is because companies are not doing it right. They train before they have an eval, and then they can’t tell if it helped. The same survey says teams are checking quality with humans and vibes more than with real evals.
It’s also much easier to train models these days. You can run LoRA on a 30B model with a single GPU, and a few hundred examples is enough to start. LoRA Without Regret from Thinking Machines shows that LoRA matches full fine-tuning in most post-training settings if you set it up correctly. So I still think we’re early.
Evaluation
Once we have the reason, we need to decide what to measure and how. Repeat after me: the single most important thing is evaluation. That’s where you should spend most of your time. If you have a good understanding of what you care about and a good way to measure it, everything else gets much simpler.
A few hundred real tasks from production is a good start. Start looking at your data manually and get a feel for it. What would you want to happen as a result, which cases are good and which aren’t, and just get to know it. After that it’s much easier to define the metrics you want.
If you define the metric before looking at the data, you usually find out later that it rewards something you don’t care about.
Also, run the eval on what you have today before you train anything. If you don’t know what the API model scores on your tasks, you don’t know what you’re trying to match. And keep it cheap to run, you’ll be running it a lot.
Data
The data I find most useful is traces of real activity: user sessions, agent runs, tool calls, edits people made to the output. They don’t have to be labeled, or the labels can be very sparse (did the user accept the answer, edit it, or reopen the ticket), with no final answer for each step.
Someone asked in the replies how much data you need, assuming that updating billions of weights takes a lot. Not much. The big models are pretty good at generalizing. It depends on what you want to do, but a few hundred examples is a good start. The cool part is that you don’t need a very strong signal.
It’s also worth keeping a small set of general capabilities to check, because sometimes training improves the task and breaks something else.
Check there’s a signal before you train
Before training, I’d want to see a signal that the thing I’m measuring can improve a lot. One way to check that without training is distillation into skills: extract a set of skills from the traces (manually, automatically, or just from looking at the data), give them to an existing model in context, and rerun the evaluation.
By skills I mean the things the traces show that the model doesn’t do by default, like which tool to call first for a certain kind of request. You write them down as instructions and put them in the prompt.
If that improves things, it’s a sign there’s something in the data the model doesn’t know, and then it makes sense to train on it. If not, it doesn’t mean there’s nothing there, but I’d be less optimistic.
You don’t need GPUs for this. And sometimes the skills in context are good enough and you can stop there. There’s research in this direction too, like GEPA, which optimizes prompts from trajectories and beat GRPO with up to 35x fewer rollouts.
Iterate
From there it’s mostly iterations. Train, look at where the model fails on the eval, try to figure out which kinds of traces are missing or noisy, fix, and repeat. Cleaning the data usually helps more than playing with hyperparameters or the model type, but it depends on the problem.
And read the failures one by one, like you did with the data at the start. The total score won’t tell you what’s wrong.
Which model
Which model to pick varies a lot by problem, and it doesn’t have to be a general-purpose LLM. If the problem is classification, you can use a classification model like TabPFN. Something very hot right now is combining such models with a text encoder, so they also understand text and you can use them on text classification problems. TabSTAR and ConTextTab are two examples.
There are also small models that specialize in agentic tasks, and a lot is being invested in that direction right now. Gemma 4 E4B and the small Qwen3.5 models do tool calling and are small enough to train cheaply. If your agent has a limited set of tasks, you can try handing them to a model like that. And there are similar things for vision, like Qwen3-VL.
The idea is similar in all these cases. There’s a set of models that are smaller and more efficient than the giant ones, and you can train them on your data. If you need a general LLM, I’d probably start around 30B. Right now that means something like Qwen3.8-27B, Gemma 4 31B or Muse Glimmer 30B. That still runs on a single GPU, and you can run LoRA experiments on it without it costing much. There are public results at this size too. SWE-smith took a 32B model up 33 points on SWE-bench Verified with about 5,000 trajectories. If 30B gets you there, try going smaller afterwards.
How to train
How exactly to train is also a great question you could talk about for hours. Basically, skill distillation is considered the simplest and most convenient, since nothing is trained and you only change the context. Then SFT, where you train on examples of inputs and the outputs you want. Your traces are mostly that already, once you filter for the good ones. And finally model distillation and RL. Distillation makes sense when a bigger model already does the task well and you want it cheaper or faster. For RL you need a reward you trust, because the model will optimize whatever the reward actually measures.
You can of course add lots of other things like quantization, routing, pruning, caching, and more.
That’s it, roughly
In the end most of the time goes to evaluation and data, and the training itself is the easy part. I’m sure I missed things. Good luck, and don’t let the model fool you!


