Transformers explained simply: how attention works in ChatGPT and Claude

1. 1

   What problem did transformers solve?

   Before 2017, most language models were recurrent neural networks (RNNs). They read a sentence one word at a time, left to right, carrying a running summary. That is like reading a long WhatsApp forward through a keyhole: by the end, you have half-forgotten the start.

   Reading in order also made training slow. The 2017 paper put it plainly: this “inherently sequential nature precludes parallelization”. GPUs are great at doing thousands of things at once, but RNNs could not use that well. The transformer dropped the word-by-word reading entirely and used only attention.

   Task: English to French translation Result: a new best score after training for 3.5 days on 8 GPUs, much less training time than earlier top models.

2. 2

   The big idea: attention

   Attention lets each word gather meaning from the other words around it, in a single step. The word “bank” means something different in “I sat on the bank of the river” and “I deposited cash at the bank”. Google’s blog post on the paper explains that the transformer can learn to look straight at “river” to decide which “bank” it is.

   Pronouns are another classic case. In “The animal didn’t cross the street because it was too tired”, “it” means the animal. Change “tired” to “wide” and “it” now means the street. Attention is how the model works out which word “it” should borrow its meaning from.

   Key idea

   Attention lets every word ask: which other words in this text matter most for what I mean here?

3. 3

   An everyday analogy: query, key and value

   Imagine every word is a member of a class WhatsApp group. The word “it” posts a question: “Who am I talking about? Probably something alive that can get tired.” This is its query.

   Every other word has a short profile line saying what it offers, called its key: “animal: a living thing”, “street: a place”. “It” checks how well each profile matches its question and gives each a score. Then it reads the actual messages, called values, mostly from the best matches. Google’s blog describes exactly this: the attention scores are used as weights for a weighted average of all the words’ representations.

   Sentence: “The animal didn’t cross the street because it was too tired” animal 0.70 · street 0.15 · tired 0.10 · everything else 0.05 New meaning of “it” = mostly “animal”, a bit of “street” and “tired”

4. 4

   What happens inside, step by step

   First, text is split into tokens, and each token becomes a long list of numbers called an embedding, which captures its meaning. Attention on its own has no idea of word order (“dog bites man” vs “man bites dog”), so the model also adds position information to each token.

   Then the tokens pass through many layers. In each layer, attention mixes in context from other tokens, and a small network refines each token further. Each layer has several attention heads working side by side, each free to look for a different kind of link, like grammar or who-did-what. The original transformer used 6 layers and 8 heads; today’s models are much bigger. At the end, the model gives a score to every possible next token.

   • In GPT-style chat models, each token can look only at earlier tokens, never ahead, because the job is to predict what comes next.
   • Google’s BERT (2018) used the transformer to read in both directions, which suits understanding tasks like search.

   1. Your text↓→
   2. Tokens↓→
   3. Embeddings + position↓→
   4. Attention layers (many)↓→
   5. Score every next token↓→
   6. Pick one
5. 5

   Why it changed everything (and its one big cost)

   Because transformers process all tokens at once, they make full use of GPUs, so labs could train much bigger models on much more text. That is what made GPT, BERT and later ChatGPT, Claude and Gemini possible. The same design now handles images and audio too: Gemini, for example, was trained on different kinds of data from the start.

   The cost: attention compares every token with every other token. Double the length of the text and the work roughly quadruples. That is a big reason why long context windows are expensive and slower, as you saw in the last lesson.

   Key idea

   Transformers won because they are parallel. Parallel means fast to train, which means they could be made very large.

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

Ashish Vaswani and seven co-authors, mostly at Google Brain and Google Research

The eight authors (Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin) of the paper that introduced the transformer, the design behind GPT, Claude and Gemini.

Attention Is All You Need (2017)

1 · Understand

They proposed a new network, the Transformer, “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”. It set new best scores for English to German and English to French translation, at a fraction of the training cost of earlier top models.

2 · Break it down

• Old way: read words one after another, so training could not make full use of a GPU’s ability to do many things at once.
• New way: every word looks at every other word at the same time through attention, so training runs in parallel.
• Proof: the English to French model reached a new best score after training for 3.5 days on eight GPUs.
• The paper was about translation. Others soon used the same design for GPT, BERT and today’s chatbots.

3 · Use it

Attention is how a model links “it” to the right noun, or a question at the end of your prompt to a detail near the top. You help it by naming things clearly and, with long documents, putting your actual question after the document.

6 terms

Words to know

Transformer

A neural network design built around attention, introduced by a Google team in 2017. GPT stands for generative pre-trained transformer.

Attention (self-attention)

A way for each token to look at all the other tokens and pull in meaning from the most relevant ones.

Embedding

A long list of numbers that represents a token’s meaning. Similar meanings get similar lists.

Query, key, value

In attention, a token’s question (query), each token’s label (key), and the information it shares (value).

Attention head

One attention unit. Models run many in parallel so each can track a different kind of relationship.

Positional encoding

Extra information added to each token so the model knows word order.

2 videos · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Play with a real transformer in your browser

1. 1Open Transformer Explainer (link below). It runs a small real model, GPT-2, inside your browser with nothing to install. A laptop works best.
2. 2Type a short sentence like “Rohit Sharma hit the ball for a” and look at the next-token predictions and their probabilities.
3. 3If you see a temperature setting, change it. Low temperature makes the top choice more likely; high temperature makes choices more random.
4. 4Click on the attention part of the diagram to see which earlier words each token looks at most.
5. 5On your phone instead? Ask a chatbot: “In ‘The trophy didn’t fit in the suitcase because it was too big’, what does ‘it’ mean? Now change ‘big’ to ‘small’.” Notice how one word flips the meaning, which is exactly what attention handles.

Open Transformer Explainer

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

What is the main advantage of transformers over older RNN language models?

AThey use hand-written grammar rulesBThey process all tokens in parallel with attention, so they train much faster on GPUs and link distant words directlyCThey need no training dataDThey read text strictly one word at a time

Question 2

Question 2 of 3

In “The animal didn’t cross the street because it was too tired”, what does attention help the model do?

ACount the words in the sentenceBTranslate the sentence word by wordCWork out that “it” refers to “the animal”DCheck the spelling of “street”

Question 3

Question 3 of 3

Why do transformers need extra position information?

AAttention by itself does not know word order, so “dog bites man” and “man bites dog” would look the sameBTo make the model run on phonesCTo count tokens for billingDBecause GPUs cannot store words

Lesson 5 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 6 · 14 minThe papers that built modern AIModern AI chatbots rest on a handful of research papers published between 2017 and 2022. The transformer (2017) gave us the engine, the GPT papers showed that predicting the next word on lots of text teaches a model many skills, scaling laws and Chinchilla showed how big to build it, InstructGPT turned it into an assistant, chain of thought taught it to show its working, and Constitutional AI trained it with written principles. Each one explains something you see in ChatGPT or Claude today.← Previous: Tokens and context windows

People also ask

Questions people ask

What is a transformer in AI, in simple terms?+

It is a type of neural network that reads all the words of a text at once and uses attention to work out how they relate. It was introduced by a Google team in 2017 and is the base of nearly every modern chatbot.

What does GPT stand for?+

Generative pre-trained transformer. Generative because it creates text, pre-trained because it first learns from huge amounts of text, and transformer because of the neural network design it uses.

What is self-attention in simple words?+

Self-attention is when each word in a text looks at all the other words in the same text and decides how much each one matters for its own meaning. It is how a model knows that “it” means the animal and not the street.

9 sources · checked Nov 2026

Sources and further reading

• Vaswani et al. (2017): Attention Is All You Need (arXiv abstract)
• Attention Is All You Need (full paper, HTML)
• Google Research blog (Aug 31, 2017): Transformer, a novel neural network architecture for language understanding
• Devlin et al. (2018): BERT
• Google (Dec 6, 2023): Introducing Gemini
• IBM: What is a context window?
• Wikipedia: Generative pre-trained transformer
• Cho et al. (2024): Transformer Explainer: Interactive Learning of Text-Generative Models
• 3Blue1Brown chapter list for “Attention in transformers” (rosetta.to)

Related on this site

Previous: Tokens and context windowsNext: How ChatGPT and Claude are trainedBuild an LLM with Karpathy’s Zero to HeroEmbeddings and RAGAI engineer career guide