Everyone talks about post-training. Few people explain what it actually looks like.
From instruction tuning to reinforcement learning, a practical look at the systems, datasets, and tradeoffs that turn a base model into a capable coding assistant.
The two visuals in this post (the post training card and the flow diagram) is designed to print on a single A3 sheet each. If you'd like the print-ready PDFs, just DM me and I'll send them over. If you found this useful, subscribe to get future deep dives, diagrams, and printable resources delivered straight to your inbox.
A pretrained language model knows a lot but helps with nothing. It can continue text, but it can’t follow instructions, write clean code, or use tools. Post-training is how you fix that and for code generation, it’s where the real engineering happens.
What is Post-Training?
When a language model finishes pretraining, it has read a meaningful fraction of the internet and learned the structure of language and code. But all it has learned to do is predict the next token. Ask it a question and it might respond with another question, because that’s a likely continuation. Show it a bug and it might write more buggy code, because that’s what most code on the internet looks like.
Post-training takes that raw capability and shapes it into something useful. It happens in stages, each solving a different problem, each building on the one before it.
Stage 1 - Supervised Fine-Tuning (SFT) shows the model examples of good behavior: here’s a prompt, here’s how a helpful assistant responds. It teaches the model to imitate.
Stage 2 - Preference Learning (RLHF, DPO) teaches it to distinguish better from worse when multiple reasonable answers exist. Less about memorizing patterns, more about developing judgment.
Stage 3 - Reinforcement Learning with Verifiable Rewards (GRPO) lets the model improve past its demonstrations by being scored on outcomes. Does the code compile? Do the tests pass?
Each stage lifts a different ceiling, and the order matters, you imitate first, then refine, then discover. Run reinforcement learning on a model that hasn’t been through SFT, and it will find degenerate strategies that maximize the reward signal but produce gibberish.
I spent the last few months building these systems end to end. This post walks through the key ideas with enough depth that you understand the mechanics.
Stage 1: Supervised Fine-Tuning (SFT)
SFT teaches format and basic competence. You show the model prompt–response pairs and train it to produce the response given the prompt.
The Loss Function
The training objective is standard cross-entropy but computed only on the assistant’s tokens. The prompt tokens are masked, we don’t want the model to learn to generate prompts, we want it to learn to answer them.
Given a prompt x and target response y = (y₁, y₂, …, y_T):
where π_θ is the model’s token-level probability.
Examples
Code generation:
Prompt: Write a Python function that takes a list of integers and returns the second largest unique value. If there are fewer than two unique values, return None.
Response:
def second_largest(nums: list[int]) -> int | None: unique = set(nums) if len(unique) < 2: return None unique.discard(max(unique)) return max(unique)
Natural language:
Prompt: Explain why the sky is blue in two sentences, suitable for a ten-year-old.
Response: Sunlight is made up of all the colors of the rainbow, and when it hits our atmosphere, the blue light bounces around in every direction much more than the other colors. That scattered blue light is what you see when you look up.
Tool use:
Prompt: What’s the weather like in Berlin right now?
Response:
[calls weather_api(location="Berlin")]→ The current temperature in Berlin is 14°C with overcast skies and a chance of rain this afternoon.
You collect thousands of these pairs covering the range of behaviors you want — code generation, explanation, refusal, multi-turn conversation, tool invocation. The ceiling of SFT is the quality of these demonstrations. If your best sorting example uses bubble sort, the model will never discover quicksort on its own.
Stage 2a: RLHF (Reinforcement Learning from Human Feedback)
RLHF breaks through SFT’s ceiling. Instead of showing the model what to produce, you show it two outputs and tell it which one humans preferred. A reward model learns to score any output the way a human would, and then the policy is updated to produce outputs that score higher.
Step 1 - Train the Reward Model
Human annotators compare pairs of model outputs and label a preference. The reward model r_ϕ is trained on these pairs using the Bradley-Terry loss:
where y_w is the preferred (winning) response, y_l is the rejected (losing) response, and σ is the sigmoid function. The loss pushes the reward model to assign higher scores to preferred outputs.
Step 2 - Optimize the Policy with PPO
The language model (the “policy” π_θ) is then updated to maximize the reward while staying close to the SFT checkpoint π_ref via a KL penalty:
The KL term is critical, without it, the model drifts into outputs that game the reward model but are useless to humans.
Example
Prompt: Write a Python function that takes a list of integers and returns the second largest unique value. If there are fewer than two unique values, return None.
The model generates two candidate responses:
Response A (preferred):
def second_largest(nums: list[int]) -> int | None: unique = sorted(set(nums)) if len(unique) < 2: return None return unique[-2]Response B (rejected):
def second_largest(nums): return sorted(nums)[-2]Annotators prefer Response A — it handles duplicates, has type hints, and covers the edge case. Response B silently returns wrong results on inputs like
[5, 5, 3]. The reward model learns from thousands of such comparisons:Reward(A) = 8.7 Reward(B) = 2.1The policy model then generates new responses, receives reward scores, and is updated via PPO to produce outputs that score higher.
The Cost
RLHF is expensive. You’re running four models in memory simultaneously: the policy, the reference policy, the reward model, and the value network. And the failure modes are real, reward hacking (gaming the rubric instead of learning the material), KL collapse (drifting so far that the model loses existing capabilities), and reward model accuracy ceilings around 68–70% (meaning nearly a third of the signal is noise).
