How ChatGPT and Claude are trained: pretraining, fine-tuning and RLHF

1. 1

   The big picture: three stages

   Think of a student preparing for a competitive exam. First they read every textbook they can find. Then a teacher shows them model answers in the right format. Finally, they write practice answers and a teacher grades them, and they adjust.

   LLM training follows the same shape. Each stage builds on the one before. The first stage is by far the most expensive; the later stages are smaller but change the model’s behaviour the most.

   1. Pretraining on internet text↓→
   2. Base model↓→
   3. Instruction tuning↓→
   4. Feedback (RLHF / RLAIF)↓→
   5. Chat assistant
2. 2

   Stage 1: Pretraining builds a base model

   In pretraining, the model reads a huge amount of text and practises one task: predict the next token. Every wrong guess nudges its billions of parameters a little. For Llama 2 70B, Karpathy estimates about 10 TB of text, about 6,000 GPUs and about 12 days, costing roughly $2 million.

   The result is a base model. It has absorbed grammar, facts and patterns from its data, but it has no idea it is supposed to be an assistant. It is a document-completer: it continues whatever text you give it, in the style of the internet.

   Prompt: “What is the capital of France?” Base model might continue: “What is the capital of Germany? What is the capital of Italy? …” (it thinks it is completing a quiz page) Chat model answers: “The capital of France is Paris.”

   Key idea

   A base model knows a lot but does not know how to help. Later training teaches it to behave like an assistant.

3. 3

   Stage 2: Instruction tuning teaches the assistant format

   Next comes supervised fine-tuning (SFT), also called instruction tuning. People write example conversations: a user’s request and an ideal assistant reply. For ChatGPT, OpenAI says human trainers wrote both sides of the conversation.

   Training on these examples teaches the model the format and personality of a helpful assistant: answer the question, follow instructions, be polite, say when it does not know. This stage uses far less data than pretraining, so it is much cheaper.

4. 4

   Stage 3: Learning from feedback (RLHF)

   Writing perfect answers is slow, but ranking answers is easy. In reinforcement learning from human feedback (RLHF), the model writes several answers to the same prompt and people rank them from best to worst. Those rankings train a second model, a reward model, that predicts which answers people prefer. Then the main model is trained to produce answers the reward model scores highly.

   It works remarkably well. In OpenAI’s 2022 InstructGPT paper, people preferred answers from a 1.3 billion parameter model trained this way over the 175 billion parameter GPT-3, a model about 100 times bigger. ChatGPT was trained with the same method.

   1. Model writes several answers↓→
   2. Humans rank them↓→
   3. Reward model learns the rankings↓→
   4. Model trained to score higher
5. 5

   Anthropic’s twist: Constitutional AI (RLAIF)

   Human ranking is slow, costly, and can mean people reading disturbing content. In December 2022 Anthropic published Constitutional AI. Instead of humans labelling harmful answers, the model is given a written list of principles, a “constitution”. It critiques and rewrites its own answers using those principles, and then an AI judge picks the better of two answers. That is reinforcement learning from AI feedback (RLAIF).

   Anthropic says its early constitution drew on sources such as the UN Universal Declaration of Human Rights and principles published by other AI labs. In January 2026 it published a new, longer constitution for Claude. The principles are written down, so anyone can read what the model is being trained toward.

   Key idea

   RLHF learns from human rankings. Constitutional AI learns from AI feedback guided by written principles.

6. 6

   The newest stage: training models to reason

   Since 2024, labs have added more reinforcement learning on problems with checkable answers, like maths and code. The model tries a problem, the answer is checked, and approaches that lead to right answers are rewarded. It is like solving past papers with the answer key: you learn what kind of working gets marks.

   OpenAI’s o1 (September 2024) was trained this way to think in a chain of steps before answering, and DeepSeek’s R1 paper (January 2025) showed that reasoning skills like checking its own work can grow from reinforcement learning alone. That is why today’s models can show “thinking” before they reply.

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

Long Ouyang and 19 co-authors at OpenAI

The OpenAI team behind InstructGPT. OpenAI trained ChatGPT later that year with the same method.

Training language models to follow instructions with human feedback (InstructGPT, 2022)

1 · Understand

They started from the point that “making language models bigger does not inherently make them better at following a user’s intent”. After training GPT-3 on human-written example answers and then on human rankings, people preferred answers from a 1.3 billion parameter model over the 175 billion parameter GPT-3.

2 · Break it down

• Step 1: people write good answers to real prompts, and the model learns from them (supervised fine-tuning).
• Step 2: the model writes several answers, people rank them, and a reward model learns those rankings.
• Step 3: reinforcement learning pushes the model toward answers the reward model scores highly.
• Size is not everything: the tuned model was about 100 times smaller, was more truthful and gave fewer toxic answers, though it still made simple mistakes.

3 · Use it

The chatbot you use was shaped by people choosing better answers. That is also why a clear example of the reply you want often works better than a vague instruction: examples and rankings are how it was taught to help.

Yuntao Bai and 50 co-authors at Anthropic

The Anthropic research team, including co-founders Jared Kaplan and Dario Amodei, who introduced the Constitutional AI method used to train Claude.

Constitutional AI: Harmlessness from AI Feedback (December 2022)

1 · Understand

They trained a harmless assistant without any human labels for harmful answers. The only human input was “a list of rules or principles”, a constitution, which the AI used to critique and improve its own answers and to judge which of two answers was better.

2 · Break it down

• Phase 1: the model answers, critiques its answer against the principles, rewrites it, and is then fine-tuned on the rewrites.
• Phase 2: an AI compares pairs of answers and picks the better one. These AI preferences replace human rankings, which is why it is called RL from AI Feedback (RLAIF).
• The goal was an assistant that is “harmless but non-evasive”: it explains why it will not help with something harmful instead of just refusing.
• Because the principles are written in plain language, people can read and debate what the model is being trained toward.

3 · Use it

You can borrow the critique-and-revise loop in any chat: give a short list of rules, ask the model to check its answer against them, then ask for a rewrite. The Try it task below does exactly this on a small scale.

6 terms

Words to know

Pretraining

The first and biggest training stage: learning to predict the next token on huge amounts of text.

Base model

The model after pretraining. It continues text but does not reliably answer or follow instructions.

Supervised fine-tuning (SFT)

Extra training on example conversations written by people, to teach the assistant format. Also called instruction tuning.

RLHF

Reinforcement learning from human feedback: people rank answers, a reward model learns their preferences, and the model is trained toward them.

Reward model

A helper model that predicts how good an answer is, trained on human or AI rankings.

Constitutional AI (RLAIF)

Anthropic’s method that uses written principles and AI feedback, instead of human labels, to make a model safer and more helpful.

2 videos · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Be the human feedback: run mini RLHF and Constitutional AI yourself

1. 1Open Claude or ChatGPT. Ask: “My friend failed an exam and feels low. Write a short WhatsApp message to cheer them up.”
2. 2Use the regenerate (retry) button to get a second answer. You now have two answers to the same prompt, just like in RLHF.
3. 3Rank them. Write one line on why the winner is better: kinder, shorter, more honest, more natural?
4. 4Now write your own 3-rule “constitution”, for example: 1) Be honest, no fake promises. 2) Sound like a real friend, not a greeting card. 3) Keep it under 40 words.
5. 5Paste the losing answer back with: “Critique this message against these 3 rules, then rewrite it to follow them.” You just did the critique-and-revise step of Constitutional AI by hand.

Open Claude (free)

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

You give a base model (straight out of pretraining) the prompt “Write a leave application.” What is most likely?

AIt writes a perfect leave applicationBIt continues the text in some internet-document way, maybe listing more requests, because it was only trained to continue textCIt refuses because it has no internet accessDIt asks you a clarifying question like an assistant

Question 2

Question 2 of 3

In RLHF, what do the humans mainly provide?

AThe internet text used for pretrainingBNew parameters for the modelCRankings of which model answers are better, used to train a reward modelDThe GPUs for training

Question 3

Question 3 of 3

How does Anthropic’s Constitutional AI differ from standard RLHF?

AIt skips pretrainingBIt uses written principles and AI feedback instead of human labels to judge harmful answersCIt only works for coding tasksDIt lets users vote on the model’s answers live

Lesson 7 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 8 · 12 minReasoning models and thinkingA reasoning model is an AI model that writes out hidden rough work, called thinking, before it gives you an answer. It was trained with reinforcement learning to solve problems step by step, so it is much better at maths, code and planning, but slower and more expensive. In ChatGPT and Claude you control this with thinking or effort settings: turn it up for hard multi-step problems and down for quick facts and rewriting.← Previous: The papers that built modern AI

People also ask

Questions people ask

What is RLHF in simple terms?+

Reinforcement learning from human feedback. The model writes several answers, people rank them, a reward model learns what people prefer, and the model is trained to give answers that score higher. It is like a teacher grading practice answers until the student learns what a good answer looks like.

What is the difference between a base model and an instruct (chat) model?+

A base model has only been pretrained to continue text, so it may ramble or continue your question instead of answering it. An instruct or chat model has had extra training on example conversations and feedback, so it follows instructions and answers like an assistant.

How much does it cost to train an LLM?+

It varies hugely with size. Karpathy estimates Meta’s Llama 2 70B took about 6,000 GPUs for about 12 days, roughly $2 million, just for pretraining. Labs rarely publish exact costs for their newest frontier models. Fine-tuning a small open model for your own task can cost far less.

13 sources · checked Nov 2026

Sources and further reading

• OpenAI (Jun 11, 2018): Improving language understanding with unsupervised learning
• Ouyang et al. (2022): Training language models to follow instructions with human feedback (InstructGPT)
• OpenAI (Nov 30, 2022): Introducing ChatGPT
• Bai et al. (2022): Constitutional AI: Harmlessness from AI Feedback
• Anthropic (May 9, 2023, updated Jan 2026): Claude’s Constitution
• OpenAI (Sep 12, 2024): Learning to reason with LLMs
• DeepSeek-AI (2025): DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Touvron et al. (2023): Llama 2: Open Foundation and Fine-Tuned Chat Models
• Andrej Karpathy (YouTube): [1hr Talk] Intro to Large Language Models
• Notes on Karpathy’s Intro to LLMs talk (gist)
• Chapter list for Karpathy’s Intro to LLMs (rosetta.to)
• Chapter list for Karpathy’s Deep Dive (rosetta.to)
• Wikipedia: Anthropic (founders)

Related on this site

Previous: Transformers and attentionNext: Your first promptPrompting vs RAG vs fine-tuningWhy AI makes things upAI product manager guide