What is a large language model (LLM)? Explained simply

1. 1

   So what does an LLM actually do?

   At its core, an LLM does one thing: given some text, it predicts what comes next. Your phone keyboard does a tiny version of this when it suggests the next word. An LLM does the same job, but trained on far more text and with a far bigger model.

   It does not pick one word directly. It gives a probability (a likelihood score) to every possible next piece of text, called a token. It picks one, adds it to the text, and repeats. A full answer is just this loop running hundreds of times.

   1. Your text↓→
   2. Score every possible next token↓→
   3. Pick one↓→
   4. Add it to the text↓→
   5. Repeat

   Input: “cat sat on a” The model scores possible next words. “mat” might get about 97%, and words like “chair” or “roof” get much smaller scores. Pick “mat”, add it, and predict again.

   Key idea

   An LLM is a very powerful “predict the next token” machine. Everything else is built on that.

2. 2

   What makes it “large”? Parameters

   Inside an LLM are billions of numbers called parameters. Think of them as tiny tuning knobs. During training, each knob is nudged again and again until the model gets better at predicting the next token. Nobody sets them by hand.

   GPT-3 (2020) had 175 billion parameters. Meta’s open model Llama 2 70B has 70 billion. In his talk, Karpathy points out that this model is basically two files: a 140 GB file of parameters and a small program that runs them.

   70,000,000,000 parameters × 2 bytes each = 140,000,000,000 bytes ≈ 140 GB

3. 3

   Where does it learn from? Training data

   LLMs learn from huge collections of text: web pages, books, code and more. For Llama 2 70B, Karpathy estimates about 10 TB of text, around 6,000 GPUs running for about 12 days, at a cost of roughly $2 million. Labs rarely publish these numbers for their newest models.

   Karpathy compares training to compressing the internet into that parameter file. It is a lossy compression, like a blurry photo. The model keeps the general patterns and many facts, but not a perfect copy. That is why it can be wrong about details, and why it does not know about events after its training data was collected (its “knowledge cutoff”) unless the app gives it web search.

   1. About 10 TB of text↓→
   2. Thousands of GPUs for days↓→
   3. 140 GB of parameters
4. 4

   From autocomplete to assistant

   A model fresh from this first training stage is called a base model. It just continues text. Ask it a question and it may continue with more questions, as if it were completing a quiz page from the internet.

   To turn it into a helpful assistant, companies train it further on example conversations and on human feedback about which answers are better. OpenAI built ChatGPT this way from a GPT-3.5 model. You will see exactly how in lesson 6.

5. 5

   Why does it seem to understand?

   To predict the next word well across billions of pages, a model has to pick up grammar, facts, and common patterns of reasoning. Research suggests it learns more than surface word patterns. In 2025 Anthropic looked inside Claude and found it plans a rhyming word before writing a line of poetry, and links facts in steps: for “the capital of the state where Dallas is”, it first gets to Texas, then to Austin.

   But the same research found Claude sometimes writes plausible-sounding reasoning that does not match what it actually did inside. The researchers say plainly that “we don’t understand how models do most of the things they do.” So treat an LLM like a very well-read helper who can still be confidently wrong, and check anything important.

   Key idea

   LLMs learn real patterns, not just word lists, but they produce likely text, not checked facts.

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

Andrej Karpathy

A founding member of OpenAI and former director of AI at Tesla. He now runs Eureka Labs, an AI education company.

[1hr Talk] Intro to Large Language Models (talk, 2023)

1 · Understand

Karpathy says an LLM is really just two files: a huge file of parameters and a short program that runs them. Training squeezes a large slice of the internet into those parameters, as a lossy compression, and he suggests seeing the LLM less as a chatbot and more as the core of a new kind of operating system.

2 · Break it down

• Two files: for Llama 2 70B, a 140 GB parameter file plus about 500 lines of code. Running it is cheap and works offline. Making the parameters is the expensive part.
• Compression: about 10 TB of text becomes 140 GB, roughly 100 times smaller. Like a blurry photo, the overall picture survives but fine details can be lost.
• Dreaming: when the model writes, it produces text in the style of what it read. Some of it is recalled, and some is invented in the same confident voice.
• LLM OS: he pictures the model at the centre, coordinating memory and tools such as a browser, a calculator and a code runner, the way an operating system runs a computer.

3 · Use it

Treat the model’s memory as blurry: for exact facts, paste in the source or turn on web search. And when your app lets the model use tools like search, files or code, use them, because on its own the model is only the compressed file.

6 terms

Words to know

Large language model (LLM)

An AI model trained on huge amounts of text to predict the next token. ChatGPT, Claude and Gemini run on LLMs.

Token

A small chunk of text, often a word or part of a word, that the model reads and writes. Covered in the next lesson.

Parameter

One of the billions of adjustable numbers inside the model that training tunes. More parameters means a bigger model.

Training data

The text the model learned from, such as web pages, books and code.

Base model

A model straight out of pretraining. It continues text but does not reliably follow instructions.

Knowledge cutoff

The date after which the model has no training data, so it does not know newer events unless given search.

2 videos · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Watch next-token prediction in action

1. 1Open ChatGPT, Claude or Gemini (all have free versions).
2. 2Type: “Give me 5 likely next words for this sentence and rate how likely each one is: ‘Virat hit the ball over the …’”. Note: the percentages it gives are its guess, not its real internal scores, but they show the idea.
3. 3Start a new chat and ask the same question again. Notice the answer is worded differently. The model samples from probabilities, so replies vary.
4. 4Ask: “What is your knowledge cutoff date?” Then ask about something from yesterday’s news with web search turned off, if your app lets you. See how it handles not knowing.
5. 5Ask it a fact you know well, like your city’s pin code or a cricket record. Was it right? This is why you check important answers.

Open Gemini (free)

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

When an LLM writes a reply, what is it doing at each step?

ALooking up the full answer in a databaseBPredicting a likely next token, adding it, and repeatingCSearching Google and copying a pageDRunning a hand-written rule for that question

Question 2

Question 2 of 3

What is a parameter in an LLM?

AA setting you type into the chat boxBOne of the billions of numbers inside the model that training adjustsCA rule written by an engineerDA web page in the training data

Question 3

Question 3 of 3

Why can an LLM give a confident answer that is wrong?

ABecause it is lying on purposeBBecause its database is out of dateCBecause it produces the most likely-sounding text from learned patterns, which is not the same as checking factsDBecause it only knows English

Lesson 3 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 4 · 10 minTokens and context windowsLLMs do not read words. They read tokens, small chunks of text that are often a whole word or part of one; in English, 100 tokens is about 75 words. The context window is the maximum number of tokens a model can look at at once, including your messages and its replies. When a chat gets too long, older parts get dropped or summarised, which is why AI seems to “forget”, and every token you send or receive also costs money on an API.← Previous: How AI started: 1956 to ChatGPT

People also ask

Questions people ask

Is ChatGPT an LLM?+

ChatGPT is a chat app built on top of LLMs. The LLM is the model that predicts text. ChatGPT adds the chat window, memory features, web search and safety layers around it. The same is true for Claude and Gemini.

What is the difference between AI and an LLM?+

AI is the whole field of making machines do “smart” tasks. An LLM is one specific kind of AI: a deep learning model trained on text to predict the next token. All LLMs are AI, but most AI (like a fraud detector) is not an LLM.

What are some examples of LLMs?+

OpenAI’s GPT models (used in ChatGPT), Anthropic’s Claude, Google’s Gemini, Meta’s Llama and DeepSeek’s models are well-known LLMs. Some, like Llama, are open-weight, which means anyone can download the parameter files.

8 sources · checked Nov 2026

Sources and further reading

• Google for Developers: Introduction to large language models
• Brown et al. (2020): Language Models are Few-Shot Learners (GPT-3)
• Touvron et al. (2023): Llama 2
• Andrej Karpathy (YouTube): [1hr Talk] Intro to Large Language Models
• Notes on Karpathy’s Intro to LLMs talk (gist)
• OpenAI (Nov 30, 2022): Introducing ChatGPT
• Anthropic (Mar 27, 2025): Tracing the thoughts of a large language model
• Wikipedia: Generative pre-trained transformer

Related on this site

Previous: How AI startedNext: Tokens and context windowsHow ChatGPT and Claude are trainedWhy AI makes things upAI engineer career guide