How AI models are trained
Follow a model from raw text to helpful assistant: instruction tuning, fine-tuning for specific jobs, learning from scores and human feedback, and copying one model's skills into another.
Pretraining and post-training: from raw text to a useful assistant#
Post-training does: it turns a model that only predicts text into one that follows instructions.123 Straight out of pretraining, a base model writes fluent text but does not reliably do what the user wants.23
Pretraining trains the model to guess the next token, a word or piece of a word, across a vast body of text.24 Post-training usually begins with supervised fine-tuning, in which the model learns from example answers, often written by people, that show the desired behavior.152 Next, in reinforcement learning from human feedback, people rank the model's answers and the model learns to give the kind they prefer.56 Other methods, such as Direct Preference Optimization and rewards from automatically checked answers, are covered in reinforcement learning.7 Post-training can also include tuning on data labeled specifically for safety.4
In a 2022 paper, OpenAI described post-training its GPT-3 base model first on answers written by people and then on people's rankings of its outputs, and called the results InstructGPT.53 Its raters, on its test prompts, preferred answers from a small InstructGPT model over GPT-3, a model more than 100 times larger.53 Size alone had not made GPT-3 follow instructions well; post-training made it much better at it.53
Fine-tuning: adapting a model to a task#
Fine-tuning keeps training an already-trained model on examples of the job you want, starting from its existing weights instead of from scratch.8910 It changes the model itself, not the harness, the software wrapped around a model.11
Full fine-tuning updates all of a model's parameters (the weights), so each task gets its own full-size copy.8 LoRA is a parameter-efficient method: it leaves the original weights unchanged and trains only small add-on matrices, called adapters.1213 Each task then needs only its small adapter on top of the shared model, which cuts computing and storage costs.814 QLoRA, a variant of LoRA, also stores the unchanged weights in fewer bits, so large models can be fine-tuned on far smaller machines, even consumer GPUs.1314 One risk is catastrophic forgetting: a model fine-tuned on a new task tends to lose what it learned before.1516 Fine-tuning can also weaken a model's safety training: in October 2023, researchers found that a few hostile training examples could compromise it, and that even ordinary datasets weakened it somewhat.17
In 2023, the team behind QLoRA fine-tuned a 65-billion-parameter model on a single professional GPU.1813 The team named the resulting models Guanaco.18 The largest Guanaco model nearly matched ChatGPT on one chatbot benchmark (Vicuna), though the authors found such benchmarks untrustworthy.1318
Reinforcement learning: training by reward, including RLHF#
Reinforcement learning (RL) trains a model by rewarding attempts that score well, instead of showing it correct answers.19 In RLHF (reinforcement learning from human feedback), the score is based on people's judgments: they pick the better of paired outputs.205
In RL, the learner is called an agent (a narrower sense than the AI agents that act for users), and the task it acts in is its environment.19 The agent acts, gets a reward from a reward function, a scoring rule that researchers design, and updates its policy, its rule for choosing actions, to earn more.1921 In RLHF, the human comparisons train a reward model, a stand-in judge that scores any answer.2220 An RL algorithm, such as proximal policy optimization (PPO), then tunes the model to win that judge's approval.2223 The goal is to raise the reward model's score without drifting too far from the base model, because the reward model is an imperfect judge that a model can learn to game, a failure called reward hacking.622 Direct preference optimization (DPO) skips the reward model and trains on the comparisons directly, while Constitutional AI replaces human judgments of harmfulness with an AI applying a written list of principles.22624 For checkable tasks like math and code, the reward can be whether the final answer is correct.25
In a 2017 study, people compared pairs of segments of an agent's behavior and picked the one they preferred.26 With no researcher-designed reward function, the agent learned Atari games and simulated robot movement from those choices.26 People gave feedback on less than one percent of the agent's interactions, and the authors wrote that this cut the cost of human oversight enough to apply it to state-of-the-art RL systems.26
Distillation: teaching a small model from a big one#
Distillation trains a small student model to imitate a large teacher model, so the student gains abilities that cost the teacher far more effort to learn.2728 Labs distill their own models routinely, but treat it as a threat when a rival does it without permission and gains costly abilities cheaply.2729
In the classic version, the student learns to match the teacher's full output: a probability for every possible next token.2830 With today's language models, the student is often just fine-tuned on text the teacher wrote.31 Distillation is cheap because the teacher already did the hard learning; the student skips the costly training the teacher needed.2732 In adversarial distillation, a rival harvests a model's answers at scale without permission, prizing its written-out reasoning because it shows how answers are reached.33 Providers can respond with terms of service, account bans, and showing users only a summary of the reasoning.272933
As of October 2026, Anthropic has reported that DeepSeek, Moonshot and MiniMax generated over 16 million exchanges with Claude through tens of thousands of fraudulent accounts.27 As of October 2026, the same report says the labs collected Claude's answers to train their own models, using prompts that stood out from normal use.27 As of October 2026, the report also says models distilled this way are unlikely to keep Claude's safeguards.27
- Glossary: every term, in a sentence or two
- All guides
Sources (33)
- Understanding Reasoning from Pretraining to Post-Training arxiv.org
- Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs arxiv.org
- Aligning language models to follow instructions openai.com
- Llama 2: Open Foundation and Fine-Tuned Chat Models arxiv.org
- Training language models to follow instructions with human feedback arxiv.org
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model arxiv.org
- Tulu 3: Pushing Frontiers in Open Language Model Post-Training arxiv.org
- LoRA: Low-Rank Adaptation of Large Language Models arxiv.org
- Supervised fine-tuning | OpenAI API platform.openai.com
- Universal Language Model Fine-tuning for Text Classification arxiv.org
- Meta-Harness: End-to-End Optimization of Model Harnesses arxiv.org
- LoRA: Low-Rank Adaptation of Large Language Models arxiv.org
- QLoRA: Efficient Finetuning of Quantized LLMs ar5iv.labs.arxiv.org
- PEFT · Hugging Face huggingface.co
- Overcoming catastrophic forgetting in neural networks arxiv.org
- Document-tuning for robust alignment to animalsThanks: Website: https://www.compassionml.com/. Correspondence to <compassioninmachinelearning@gmail.com>. arxiv.org
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! arxiv.org
- QLoRA: Efficient Finetuning of Quantized LLMs arxiv.org
- Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models arxiv.org
- Learning to summarize from human feedback arxiv.org
- mjlab A Lightweight Framework for GPU-Accelerated Robot Learning github.com/mujocolab/mjlab arxiv.org
- Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards arxiv.org
- Proximal Policy Optimization Algorithms arxiv.org
- Constitutional AI: Harmlessness from AI Feedback arxiv.org
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs arxiv.org
- Deep reinforcement learning from human preferences arxiv.org
- Detecting and preventing distillation attacks anthropic.com
- Making Knowledge Distillation Cheap Enough to Run at Scale huggingface.co
- Disrupting a coordinated model-distillation campaign | OpenAI openai.com
- Distilling the Knowledge in a Neural Network arxiv.org
- Distillation in 2026 (so far): which frontier models use it and how huggingface.co
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning arxiv.org
- Strengthening our defenses against distillation attacks anthropic.com