Tokens and context windows: how LLMs read text, and why long chats forget

1. 1

   What is a token?

   Before an LLM sees your message, a tool called a tokenizer chops it into tokens. Common words are usually one token. Rare words, names and long words get split into pieces. Spaces and punctuation are part of tokens too.

   We ran a few examples through o200k_base, the tokenizer used by GPT-4o and newer OpenAI models. The | marks show where it cuts.

   “I love playing cricket with my friends on Sunday.” → 10 tokens I | love | playing | cricket | with | my | friends | on | Sunday | . “unbelievable” → un | bel | ievable (3 tokens) “Swiggy” → Sw | ig | gy (3 tokens) “UPI payment failed” → UP | I | payment | failed (4 tokens)

   Key idea

   Rule of thumb for English: 1 token ≈ 4 characters ≈ ¾ of a word, so 100 tokens ≈ 75 words.

2. 2

   Why tokens matter: languages and spelling

   Each model family has its own tokenizer, built from large samples of text. Text that is common there, like English, gets packed into fewer tokens. The same meaning in another language can need many more. Even within one company, tokenizers change: Anthropic says its tokenizer for Claude 4.7 and later produces about 30% more tokens for the same text.

   Tokens also explain a famous quirk. Ask a chatbot how many r’s are in “strawberry” and it may get it wrong, because it sees st | raw | berry, not single letters. Karpathy covers this in his deep dive: models think in chunks, so spelling and counting letters are surprisingly hard for them.

   English: “I love playing cricket with my friends on Sunday.” → 10 tokens Hindi: “मुझे रविवार को अपने दोस्तों के साथ क्रिकेट खेलना पसंद है।” → 30 tokens Same meaning, 3× the tokens. On an API, that is 3× the cost and 3× the space used.

3. 3

   What is a context window?

   The context window is the model’s working memory: the most tokens it can look at in one go. Anthropic’s docs say it includes everything in the request: instructions, your messages, its earlier replies, any files or images, and the answer it is writing right now.

   Sizes have grown fast. Many current Claude models handle 1 million tokens by default, and older ones 200,000. Using the 75-words-per-100-tokens rule, 1 million tokens is roughly 750,000 English words. Anything outside the window simply does not exist for the model on that turn.

   200,000 tokens × 0.75 ≈ 150,000 English words 1,000,000 tokens × 0.75 ≈ 750,000 English words

4. 4

   Why do long chats “forget”?

   An LLM has no memory between turns. Each time you send a message, the app sends the whole conversation again. As the chat grows, it eventually hits the window limit, and the app must drop or summarise older messages. Your instructions from the start can quietly fall out.

   Even before the limit, more is not always better. Anthropic’s docs call this “context rot”: as the token count grows, accuracy and recall go down. A 2023 study, “Lost in the Middle”, found models use information at the start or end of a long input better than information buried in the middle.

   • Start a new chat for a new topic.
   • Restate key facts and rules when a chat gets long.
   • Put the most important instruction at the start or the end, not the middle.

   1. Message 1↓→
   2. Messages 2 to 80↓→
   3. Window fills up↓→
   4. Old parts dropped or summarised↓→
   5. Early details “forgotten”
5. 5

   Tokens are also how AI is priced

   When developers use an AI model through an API (a way for programs to talk to it), they pay per token, usually quoted per million tokens. Output tokens (what the model writes) cost more than input tokens (what you send). On Anthropic’s current price list, output costs 5 times as much as input.

   Long inputs are also slower and more expensive to process. IBM notes that the compute needed grows quadratically with input length: double the input and the work roughly quadruples. You will see why in the next lesson on attention.

   Prices: $2 per million input tokens, $10 per million output tokens One request: 2,000 tokens in, 500 tokens out Input: 2,000 × $2 ÷ 1,000,000 = $0.004 Output: 500 × $10 ÷ 1,000,000 = $0.005 Total: $0.009 per request, or about $90 for 10,000 requests

   Key idea

   Fewer, better tokens means cheaper, faster and often more accurate answers.

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

Rico Sennrich, Barry Haddow and Alexandra Birch (University of Edinburgh)

Their 2015 translation paper brought byte pair encoding (BPE) into neural language models. Versions of BPE now build the tokenizers of GPT, Llama and many other models.

Neural Machine Translation of Rare Words with Subword Units (2015)

1 · Understand

Models that use a fixed list of whole words get stuck on new words, names and compounds. The authors split rare words into smaller, common pieces called subword units, so a model can read and write any word from a limited set of parts.

2 · Break it down

• Start with single characters. Find the pair that appears side by side most often and merge it into one new token. Repeat thousands of times.
• Common words end up as a single token. Rare words stay split into pieces, like “unbelievable” → un | bel | ievable.
• No word is ever “unknown”: in the worst case, it is built from small pieces.
• In their tests this beat the older dictionary approach for translating rare words, from English into German and Russian.

3 · Use it

This is why a rare name, a long technical word or text in a language the tokenizer saw less often takes more tokens. When you pay per token or are close to a context limit, plain, common wording packs more meaning into each token.

Andrej Karpathy

A founding member of OpenAI and former director of AI at Tesla, known for teaching how LLMs work by building them from scratch.

Let’s build the GPT Tokenizer (video lecture, 2024)

1 · Understand

Karpathy builds a GPT-style BPE tokenizer from scratch and argues that many odd LLM failures trace back to tokenization: trouble with spelling, reversing a string, simple arithmetic and non-English text.

2 · Break it down

• The model never sees letters, only token numbers. So “how many r’s in strawberry” asks about something it cannot directly see.
• Numbers are split into chunks too, which is one reason simple arithmetic can go wrong.
• A tokenizer packs the text it was trained on most efficiently. Text it saw less of, often non-English, breaks into more, smaller pieces.
• He released the code as minbpe, a small open repository, so anyone can read the whole algorithm for themselves.

3 · Use it

When a chatbot struggles with counting letters, reversing text or long sums, do not argue with it. Ask it to use its code tool, or to write the word out letter by letter first so each letter gets its own token.

5 terms

Words to know

Token

A chunk of text (a word, part of a word, or punctuation) that an LLM reads and writes.

Tokenizer

The tool that splits text into tokens and turns them into numbers the model can use. Each model family has its own.

Context window

The maximum number of tokens the model can consider at once, including both input and output.

Context rot

The drop in accuracy and recall as more and more tokens are packed into the context.

Input and output tokens

Input tokens are what you send; output tokens are what the model writes. APIs charge for both, output at a higher rate.

1 video · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Count tokens yourself

1. 1Open the free OpenAI Tokenizer page (link below). No login needed.
2. 2Paste: “I love playing cricket with my friends on Sunday.” Check the token count and the coloured chunks.
3. 3Now paste the same sentence in Hindi, Tamil, Bengali or your own language. How many more tokens does it take?
4. 4Try your full name, a 10-digit phone number, an emoji and a long word like “internationalisation”. Which ones split into many pieces?
5. 5Check the rule of thumb: divide the character count by the token count for an English paragraph. Is it close to 4?
6. 6Bonus: ask a chatbot “How many r’s are in strawberry?” and think about why tokens make this tricky.

OpenAI Tokenizer

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

Roughly how many tokens is a 750-word English essay?

AAbout 75BAbout 500CAbout 1,000DAbout 7,500

Question 2

Question 2 of 3

You gave a chatbot rules at the start of a very long chat, and now it ignores them. What is the most likely reason?

AThe model got tiredBOlder parts of the chat were dropped or summarised to fit the context window, and long contexts reduce accuracyCThe model saved your rules to a file and lost itDChatbots can only follow one rule per chat

Question 3

Question 3 of 3

The same message takes 3 times as many tokens in Hindi as in English. What is a practical effect?

AThe Hindi answer will be 3 times more accurateBNothing; tokens do not matter to usersCIt costs more on an API and fills the context window fasterDThe model cannot read Hindi at all

Lesson 4 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 5 · 11 minTransformers and attentionA transformer is the neural network design behind ChatGPT, Claude and Gemini, introduced by a Google team in the 2017 paper “Attention Is All You Need”. Its key idea, attention, lets every word look at every other word at once and decide which ones matter for its meaning. Because it processes all words in parallel instead of one by one, it trains fast on GPUs and scales to huge models.← Previous: What is an LLM?

People also ask

Questions people ask

How many tokens is 1,000 words?+

For English, about 1,300 tokens, using OpenAI’s rule of thumb that 100 tokens ≈ 75 words. Other languages, code and unusual words often need more tokens for the same length.

What happens when the context window is full?+

The model cannot see more than its window. Chat apps usually drop or summarise the oldest messages so the chat can continue, which is when it starts “forgetting”. On the API, a request that is too long is rejected with an error, or the answer stops early when the limit is hit.

Is a bigger context window always better?+

No. A bigger window lets you paste longer documents, but accuracy and recall drop as context grows (context rot), and long inputs cost more and run slower. Give the model the information it needs, not everything you have.

13 sources · checked Nov 2026

Sources and further reading

• OpenAI Help: What are tokens and how to count them?
• OpenAI tiktoken: model-to-tokenizer map (o200k_base)
• gpt-tokenizer (npm), used to run the o200k_base examples in this lesson
• Claude docs: Context windows
• Claude docs: Pricing
• IBM: What is a context window?
• Liu et al. (2023): Lost in the Middle: How Language Models Use Long Contexts
• Andrej Karpathy (YouTube): Deep Dive into LLMs like ChatGPT
• Chapter list for Karpathy’s Deep Dive (rosetta.to)
• Sennrich, Haddow, Birch (2015): Neural Machine Translation of Rare Words with Subword Units
• Andrej Karpathy (YouTube): Let’s build the GPT Tokenizer
• Andrej Karpathy (GitHub): minbpe
• fast.ai: Let’s Build the GPT Tokenizer, text version of Karpathy’s lecture

Related on this site

Previous: What is an LLM?Next: Transformers and attentionPrompting techniques that workEmbeddings and RAGWhat is an API?