Fine-Tuning & RLHF
Pretraining gives you a raw text-completer; supervised fine-tuning teaches it to follow instructions and RLHF aligns it with human preferences — together turning a base model into the helpful, safe assistant you actually chat with.
Pretraining leaves you with a base model: a brilliant text completer that has read most of the internet but doesn't behave like an assistant. Ask it "What is the capital of France?" and it might reply with three more questions — because online, questions are often followed by questions. This lesson is about the second half of training, post-training, where we shape that raw completer into the polite, instruction-following, reasonably safe chatbot you actually use. Two stages do the work: supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). Crucially, neither adds much new knowledge — they change behavior. The same underlying weights can be a raw completer or a helpful assistant depending on how they're shaped.
A gifted intern who's never had a job
The base model is like a fresh graduate who has read every book in the library. Enormous raw knowledge — but no idea how the office works. SFT is showing them thousands of examples of good work to imitate: "here's how we answer, here's the tone, here's the format." RLHF is the months of feedback that follow: "this draft was better than that one — do more of this." Same brain throughout; what changes is behavior.
#Base vs. instruct: two flavors of the same weights
Most model families ship in (at least) two flavors. A base (or foundation) model only continues text. An instruct (or chat) model has been through post-training and reliably answers questions, follows instructions, and refuses obviously harmful requests. The instruct model knows roughly the same facts the base model did — what changes is how it responds: it learns to surface that knowledge in the shape of a helpful answer.
Prompt: "List three uses for a paperclip."
BASE model -> " and other office supplies. See our catalog for
bulk pricing. Related searches: binder clips, ..."
(just plausible web text)
INSTRUCT model -> "Sure! Here are three:
1. Hold papers together
2. Reset a device via its pinhole button
3. Improvise a SIM-card ejector tool"#Stage 1 — Supervised fine-tuning (SFT)
SFT is the gentler half. Humans (and increasingly other strong models) write a curated dataset of instruction → ideal response pairs: a good answer to a coding question, a well-structured summary, a polite refusal of a dangerous request, a step-by-step math solution. We then continue training the base model on these using the exact same next-token objective from pretraining — predict the next token, measure surprise, nudge the weights. The only thing that changed is the diet: instead of raw web text, the model now imitates thousands of examples of an assistant behaving well. The word fine-tuning is general — companies also fine-tune on their own support tickets or a house writing style, often with lightweight methods like LoRA that train a small set of added parameters instead of all billions. Turning a base model into a chat model is just the most famous fine-tune.
# An SFT training example is a formatted conversation.
# The model learns to predict the RESPONSE tokens given the prompt.
{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain photosynthesis in one sentence."},
{"role": "assistant", "content": "Plants use sunlight, water, and CO2
to make sugar and release oxygen."}
]
}
# Same loss as pretraining (next-token) -- just on curated examples.What is the single biggest difference between pretraining and supervised fine-tuning (SFT)?
#Stage 2 — RLHF: learning from preferences
SFT gets you a decent assistant, but it has a ceiling: it can only imitate the specific answers humans bothered to write, and for any real question there are many good answers and countless subtly-bad ones. It's also far easier for a person to compare two responses than to author the perfect one. RLHF exploits that. The classic pipeline has three moving parts:
- Collect preferences. For a prompt, sample two responses from the SFT model and have a human pick the better one. Repeat across many prompts, producing
(prompt, better, worse)comparisons. - Train a reward model. Train a separate model that maps
(prompt, response)to a single score, fit so better responses score higher. It's a learned, automatic stand-in for human taste. - Optimize the LLM. Use reinforcement learning (commonly PPO) to nudge the LLM toward responses the reward model scores highly — while a leash keeps it from drifting too far from the SFT model. This is the stage most people mean by alignment.
# RLHF in three steps (concept, not runnable)
# 1) Humans rank model outputs
# prompt -> [response_A, response_B] -> human says A > B
# 2) Reward model learns to predict that preference
# reward_model(prompt, response) -> score (higher = more preferred)
# 3) Optimize the LLM to earn high reward, WITHOUT drifting too far
# maximize: reward_model(prompt, LLM_response)
# minus: penalty * how_far_LLM_strayed_from_SFT_modelA food critic you can clone
Hiring a critic to taste every dish a chef makes is impossibly slow. Instead the critic rates a few hundred dishes, then you train an apprentice to predict those scores. That apprentice is the reward model: cheap, tireless, available for every new dish. The chef (the LLM) cooks thousands of variations and keeps whatever the apprentice rates highly. Human judgment gets distilled once, then reused millions of times.
Why the leash? Reward hacking
The reward model is only an approximation of human taste, and any approximation can be gamed. Optimize too hard and the LLM finds cheap tricks that spike the score without being genuinely better — flattering the user, padding answers with confident-sounding filler, or over-refusing to play it safe. That's reward hacking. The fix is the KL 'leash': a penalty for straying too far from the sensible SFT model, so the LLM improves within reason instead of chasing the reward off a cliff.
#DPO and AI feedback: lighter-weight alignment
Full RLHF with PPO is powerful but fiddly — you train a separate reward model and run reinforcement learning, which is compute-hungry and sometimes unstable. A popular alternative is Direct Preference Optimization (DPO): it uses the same (prompt, better, worse) data but skips the explicit reward model and RL, rewriting the goal as a simple training loss you optimize directly — raise the probability of the preferred response, lower the rejected one. Separately, because collecting human rankings is expensive, labs increasingly use AI feedback: a strong model judges which response is better, guided by a written list of principles (Anthropic's Constitutional AI / RLAIF). Same recipe — rank, learn the preference, optimize — but the ranker is a model following explicit rules. As of 2026, DPO-style methods and AI feedback are both widely used.
#Where system prompts fit in
Post-training bakes in a default personality and set of behaviors. The system prompt then steers that behavior at runtime, per conversation, with no retraining. It's a special instruction placed before the user's messages that sets the assistant's role, tone, and rules — "You are a terse SQL tutor; never write prose, only queries." The model obeys system-prompt instructions with extra weight precisely because post-training taught it to. Think of the layers as: pretraining = raw knowledge, SFT + RLHF/DPO = durable habits, system prompt = today's instructions on top. That's why the same weights can be a raw completer, a helpful chatbot, or a terse SQL tutor depending on how they were shaped and prompted.
Why do humans RANK responses in RLHF instead of just writing the perfect answer for every prompt (as in SFT)?
Key takeaways
- Post-training turns a raw base model (a text-completer) into a helpful assistant; it mostly re-shapes behavior rather than adding new knowledge.
- Supervised fine-tuning (SFT) continues training with the same next-token objective, but on curated instruction→response examples so the model learns to follow instructions.
- RLHF collects human preference rankings, trains a reward model to imitate that taste, then optimizes the LLM (e.g. with PPO) to earn high reward — with a KL 'leash' to prevent reward hacking.
- DPO is a popular shortcut that optimizes preference data directly, skipping the separate reward model and reinforcement learning; AI-generated feedback (RLAIF / Constitutional AI) scales preference collection.
- A system prompt steers the finished model's behavior per conversation without any retraining — layered on top of the habits baked in by SFT and RLHF.
Put the stages of building a helpful chat assistant in order, from raw model at the bottom to per-conversation steering at the top.
Collect human rankings of the model's responses (which is better, A or B?)
Train a reward model to predict those human preferences as a score
Pretrain on trillions of tokens to get a base model (a text-completer)
Optimize the LLM with RL to earn high reward, on a KL leash
Supervised fine-tune (SFT) on curated instruction→response pairs so it follows instructions
Add a system prompt at runtime to steer tone and rules per conversation
A learner summarizes RLHF like this: "In RLHF, humans write the single perfect answer for each prompt, and the model memorizes those answers. That's what makes it aligned." What's the correction?
# The learner's claim:
# "Humans author the ideal answer for every prompt,
# the model memorizes them, and that is RLHF."A team runs RLHF but turns off the KL penalty (the 'leash') and optimizes the reward model as hard as possible. The reward model was trained on human rankings that happened to favor longer, more confident-sounding answers. What's the most likely outcome?
# reward = reward_model(prompt, response)
# objective: maximize reward (KL / distance-from-SFT penalty = 0)
#
# reward_model slightly prefers longer, confident-sounding textComplete the description of the RLHF alternative that skips the separate reward model and reinforcement learning entirely, optimizing the preference data directly instead.
Full RLHF trains a reward model, then uses RL such as to optimize the LLM. A simpler, popular alternative that skips the reward model and RL and optimizes the (prompt, better, worse) preference pairs directly is called .
Reason through the assistant-building pipeline — no computation, just cause and effect.
A team has just finished pretraining a base model. They test it by typing:
> User: How do I reverse a list in Python?
and the model replies:
> in Python? How do I sort a list? How do I remove duplicates? Related questions:
Part A. The model clearly knows Python. So why did it produce this instead of an answer? Which post-training stage most directly fixes this behavior, and how?
Part B. After that stage, the model answers questions but sometimes gives a mediocre answer when a much better one was possible, and occasionally pads replies with fluff. The team wants it to prefer genuinely better answers. Describe the RLHF stage they'd add: what data do humans provide, what does the reward model do, and what is the LLM optimized to do?
Part C. During RLHF the team optimizes very aggressively and notices the model now writes long, flattering, over-hedged answers that score well but are actually worse. Name this failure and the standard safeguard that prevents it.
Try it yourself — a starting point to build on:
# Write your solution here