The AI papers that built ChatGPT and Claude, explained simply

1. 1

   2017: Attention Is All You Need (the engine)

   The problem: older language models read a sentence one word at a time, in order, like reading with your finger under each word. That made them slow to train, because the computer could not work on many words at once.

   The key idea: a Google team of eight authors, with Ashish Vaswani as first author, proposed the Transformer. It drops word-by-word reading and uses only “attention”: every word looks at every other word at the same time to work out which ones matter to it. In “the bank of the river”, attention helps “bank” look at “river” and pick the right meaning.

   Why it mattered: it translated English to German better than any earlier system (28.4 on the BLEU translation score, more than 2 points above the previous best) and trained in 3.5 days on eight GPUs for English to French, a small fraction of earlier costs. Because it could run in parallel, people could now train much bigger models.

   Key idea

   You see it today in every chatbot: the GPT models that followed were all built on the transformer. Lesson 5 explains attention in depth.

2. 2

   2018 to 2020: GPT-1, GPT-2 and GPT-3 (predict the next word, learn everything)

   The problem: AI models were trained separately for each task, one for translation, one for answering questions, each needing lots of labelled examples that people had to prepare by hand.

   The key idea, in three steps. GPT-1 (June 2018): first train a transformer on a few thousand books just to predict text, then fine-tune it a little for each task. It beat the previous best on most tasks it was tested on. GPT-2 (February 2019): a 1.5 billion parameter model trained only to predict the next word on about 40 GB of text from 8 million web pages. With no task-specific training it set records on several tests. GPT-3 (May 2020): 175 billion parameters, ten times bigger than any earlier model of its kind. You could teach it a new task just by showing a few examples in the prompt.

   1. Lots of text↓→
   2. Predict the next word↓→
   3. Model picks up grammar, facts, skills↓→
   4. Do new tasks from a prompt

   Translate to Hindi: Good morning → सुप्रभात Thank you → धन्यवाद See you tomorrow → The model continues the pattern. No retraining, just examples in the prompt.

   Key idea

   What you see today: you can ask ChatGPT or Claude to do almost any text task in plain words, and giving examples in your prompt still helps.

3. 3

   2020 and 2022: Scaling laws and Chinchilla (how big to build it)

   The problem: training a large model costs millions of dollars. Before you spend that, you want to know: will a bigger model actually be better, and should you spend the money on a bigger model or on more data?

   Scaling laws (Kaplan and colleagues at OpenAI, January 2020): the model’s error falls in a smooth, predictable way as you add parameters, data and computing power, across more than seven orders of magnitude (a range of ten million times). Think of it like knowing that each extra hour of study reliably raises your marks by a known amount. Their advice was to train very large models on a modest amount of data.

   Chinchilla (Hoffmann and colleagues at DeepMind, March 2022) trained over 400 models and corrected that advice: most large models were undertrained. Model size and training data should grow together. Their 70 billion parameter Chinchilla used the same computing budget as the 280 billion parameter Gopher, but with 4 times more data, and it beat Gopher and GPT-3 on a wide range of tasks while being cheaper to run.

   “For every doubling of model size the number of training tokens should also be doubled.” Chinchilla, 70B parameters → 67.5% on the MMLU knowledge test Gopher, 280B parameters → more than 7 points lower

   Key idea

   What you see today: labs plan model size and data together, which is why smaller, cheaper models can still be very capable.

4. 4

   2022: InstructGPT (from autocomplete to assistant)

   The problem: a bigger model is not automatically a more helpful one. A model trained only to predict internet text can be untruthful, toxic or simply ignore what you asked, because it is continuing text, not following instructions.

   The key idea: Long Ouyang and colleagues at OpenAI fine-tuned GPT-3 with human feedback in three stages. People wrote good example answers, then ranked several model answers from best to worst, a reward model learned those rankings, and the model was trained to score higher. This is RLHF (reinforcement learning from human feedback).

   Why it mattered: people preferred answers from the 1.3 billion parameter InstructGPT over the 175 billion parameter GPT-3, a model about 100 times bigger. In November 2022, OpenAI trained ChatGPT using the same approach.

   1. Humans write example answers↓→
   2. Humans rank model answers↓→
   3. Reward model learns rankings↓→
   4. Model trained to score higher

   Key idea

   What you see today: ChatGPT and Claude answer like helpful assistants instead of random web pages. Lesson 7 covers this training in detail.

5. 5

   2022: Chain-of-thought prompting (show your working)

   The problem: large models were good at language but often wrong on maths word problems and multi-step logic. They jumped straight to an answer.

   The key idea: Jason Wei and colleagues at Google showed that if your prompt includes a few worked examples that write out the steps, the model also writes out its steps, and gets far more answers right. With just eight such examples, Google’s 540 billion parameter PaLM model reached 58% on GSM8K, a set of school maths word problems, a new best. The benefit only showed up in big models, around 100 billion parameters and above.

   A few months later, another team (Kojima and colleagues) found you do not even need examples. Adding “Let’s think step by step” raised one model’s score on GSM8K from 10.4% to 40.7%.

   Key idea

   What you see today: “thinking” or reasoning models that work through a problem before answering grew out of this idea. That is the next lesson after training.

6. 6

   2022: Constitutional AI (training with written principles)

   The problem: RLHF needs lots of people to rate answers, including harmful ones, which is slow and unpleasant work. And the values being taught stay hidden inside thousands of individual ratings.

   The key idea: Yuntao Bai and colleagues at Anthropic gave the model a short written list of principles, a “constitution”. The model critiques and rewrites its own answers using those principles, then an AI judge compares answers to train it further. They called this RL from AI feedback (RLAIF). The result was an assistant that was harmless but not evasive: when it declined a harmful request, it explained why instead of just refusing.

   Key idea

   What you see today: Claude is trained with a public constitution. Anthropic published a new, longer one in January 2026 that anyone can read and reuse.

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

Ashish Vaswani, Noam Shazeer, Niki Parmar and colleagues

The Google team of eight authors who introduced the transformer

Attention Is All You Need (paper, 2017)

1 · Understand

They showed that a model built only from attention, with no step-by-step reading of words, translates better and trains much faster. Their architecture was “more parallelizable and requiring significantly less time to train.”

2 · Break it down

• Older models processed words in order, so they could not use many processors at once.
• Attention lets every word look at every other word in one go, which suits GPUs.
• Faster training meant labs could afford to train far bigger models on far more text.
• The GPT series, and the chatbots built on it, came out of this design.

3 · Use it

When an AI tool handles your whole document at once and connects a pronoun on page 3 to a name on page 1, that is attention at work. It is also why the context you give matters: the model attends to everything you paste in.

Jared Kaplan, Sam McCandlish, Dario Amodei and colleagues

OpenAI researchers in 2020; Kaplan, McCandlish and Amodei were later among Anthropic’s founders

Scaling Laws for Neural Language Models (paper, 2020)

1 · Understand

They found that a language model’s error falls in a smooth, predictable way as you scale it up: “The loss scales as a power-law with model size, dataset size, and the amount of compute.”

2 · Break it down

• Loss means how wrong the model’s next-word guesses are. Lower is better.
• A power law means steady, predictable gains each time you multiply size, data or compute.
• Small details of the design mattered much less than overall scale.
• This gave labs the confidence to spend huge sums on bigger models, and Chinchilla later refined how to split that budget between size and data.

3 · Use it

It explains why each new model generation is usually better and why larger models in a family (for example, a “Pro” or “Opus” tier) tend to beat smaller ones on hard tasks, while costing more per answer.

6 terms

Words to know

Research paper

A written report of a new method and its results. AI papers are usually shared free on arXiv.org before or alongside journal publication.

Transformer

The 2017 model design that uses attention to look at all words at once. Today’s chatbots are built on it.

Scaling laws

The finding that a model’s error drops in a smooth, predictable way as you add parameters, data and computing power.

Few-shot prompting

Teaching a model a task by putting a few examples in your prompt, with no retraining. Made famous by GPT-3.

RLHF

Reinforcement learning from human feedback: people rank answers, a reward model learns their preferences, and the model is trained toward them.

Chain of thought

Having the model write out intermediate reasoning steps before its final answer.

1 video · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Read a real AI paper abstract with an AI tutor, then test chain of thought

1. 1Open the Attention Is All You Need page on arXiv (link below). Read only the title and the short abstract. It is about 10 lines.
2. 2Copy the abstract into ChatGPT, Claude or Gemini (free versions are fine) and ask: “Explain this abstract to a first-year college student in 5 short bullet points. Explain every technical term.”
3. 3Check the AI: does its summary mention 28.4 BLEU and 3.5 days on eight GPUs, as the abstract does? If it adds numbers that are not in the abstract, it is making them up.
4. 4Now test chain of thought. In a new chat, ask: “A pen and a notebook cost ₹110 in total. The notebook costs ₹100 more than the pen. How much is the pen? Answer with just a number.” Then ask again in another new chat, ending with “Think step by step.” Compare. The right answer is ₹5.
5. 5Optional: repeat step 2 with the InstructGPT abstract (arxiv.org/abs/2203.02155) and ask how it connects to ChatGPT.

Open “Attention Is All You Need” on arXiv

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

What was the main change the 2017 transformer paper made?

AIt used much more training data than anyone beforeBIt dropped word-by-word reading and used attention, so all words could be processed at onceCIt added human feedback to trainingDIt was the first model to generate images

Question 2

Question 2 of 3

What did the Chinchilla paper find?

ABigger models are always better, whatever the dataBModel size and training data should grow together, so many large models were undertrainedCModels stop improving after 10 billion parametersDHuman feedback matters more than size

Question 3

Question 3 of 3

Why did people prefer a 1.3 billion parameter InstructGPT model over the 175 billion parameter GPT-3?

AIt had access to the internetBIt was trained with human feedback to follow what users actually wantedCIt used a new kind of computer chipDIt was trained on more books

Lesson 6 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 7 · 12 minHow ChatGPT and Claude are trainedLLMs like ChatGPT and Claude are trained in stages. First, pretraining on huge amounts of text creates a base model that can only continue text. Then instruction tuning on example conversations and reinforcement learning from feedback (from humans in RLHF, or from AI guided by written principles in Anthropic’s Constitutional AI) turn it into a helpful, safer assistant. Newer reasoning models add more reinforcement learning on problems with checkable answers.← Previous: Transformers and attention

People also ask

Questions people ask

What is the most important AI paper?+

Most people point to “Attention Is All You Need” (Google, 2017), which introduced the transformer. Every GPT model since was built on it. But a chatbot like ChatGPT also needed the GPT papers, scaling laws and InstructGPT, so it is better to think of a short chain of papers than one.

Do I need to read AI research papers to work in AI?+

Not to use AI or build apps with it. But reading the abstract and introduction of the key papers helps you understand why models behave as they do, and it comes up in AI engineer interviews. Paste an abstract into an AI tool and ask it to explain each term, then check its summary against the original.

Where can I read AI papers for free?+

Most AI papers are free on arXiv.org. Each paper has an abstract page (like arxiv.org/abs/1706.03762) with a short summary and a PDF link. Labs such as OpenAI, Anthropic and Google DeepMind also publish plain-language blog posts about their research.

15 sources · checked Nov 2026

Sources and further reading

• Vaswani et al. (2017): Attention Is All You Need (arXiv)
• OpenAI (June 11, 2018): Improving language understanding with unsupervised learning (GPT-1)
• OpenAI (Feb 14, 2019): Better language models and their implications (GPT-2)
• Brown et al. (2020): Language Models are Few-Shot Learners (GPT-3)
• Kaplan et al. (2020): Scaling Laws for Neural Language Models
• Hoffmann et al. (2022): Training Compute-Optimal Large Language Models (Chinchilla)
• Ouyang et al. (2022): Training language models to follow instructions with human feedback (InstructGPT)
• OpenAI (Nov 30, 2022): Introducing ChatGPT
• Wei et al. (2022): Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Google Research (May 11, 2022): Language Models Perform Reasoning via Chain of Thought
• Kojima et al. (2022): Large Language Models are Zero-Shot Reasoners
• Bai et al. (2022): Constitutional AI: Harmlessness from AI Feedback
• Anthropic (Jan 22, 2026): Claude’s new constitution
• Wikipedia: Anthropic (founders)
• Yannic Kilcher (YouTube): Attention Is All You Need

Related on this site

Previous: Transformers and attentionNext: How ChatGPT and Claude are trainedReasoning models and thinkingAI safety and alignmentBuild an LLM with Karpathy’s Zero to HeroAI engineer career guide