What is multimodal AI? How AI sees, hears, speaks and makes images

1. 1

   What does “multimodal” mean?

   A modality is a type of information: text, images, sound or video. Early chatbots were text-only. Multimodal AI can take in several types and often produce several too, so you can photograph a maths problem, speak a question or ask for a picture.

   The trick is the same everywhere: turn every kind of input into tokens, the small chunks a model works with (see the tokens lesson). Once a photo or a sound clip is tokens, the same transformer that handles text can handle it, and can connect what it sees with what you typed.

   1. Photo, voice or text↓→
   2. Turned into tokens↓→
   3. One model reasons over all of it↓→
   4. Text, speech or image out

   Key idea

   Multimodal AI turns pictures and sound into tokens, so one model can reason across all of them.

2. 2

   How does AI see? Images become patches

   A 2020 Google paper titled “An Image is Worth 16x16 Words” showed that you can cut an image into small square patches and feed them to a transformer like words in a sentence. Claude’s documentation describes the same idea: images are processed in 28 by 28 pixel patches, so a 1000 by 1000 photo costs about 1,296 tokens.

   To link pictures with words, OpenAI’s CLIP (2021) learned from about 400 million image and caption pairs from the internet, by guessing which caption goes with which image. That is how a model learns that a photo of a dosa and the word “dosa” mean the same thing.

   What it is good at: reading handwriting and whiteboards, explaining charts, describing screenshots, solving a photographed question. Where it slips: tiny or blurry images, exact counts of many small objects, and precise positions. Claude also refuses to name people in photos and cannot tell whether an image is AI-generated.

   Upload: a phone photo of today’s lecture whiteboard Prompt: “Turn this whiteboard into clean study notes with headings. Write [unclear] wherever you cannot read a word. Then give me 3 quiz questions on it.”

3. 3

   How does AI hear and speak?

   Speech-to-text turns your voice into words. OpenAI’s Whisper (September 2022) was trained on 680,000 hours of audio in many languages, which made it robust to accents and background noise. Text-to-speech does the reverse and reads text out in a natural voice.

   Early voice assistants chained three models: speech-to-text, then a text model, then text-to-speech. OpenAI says its old ChatGPT Voice Mode took 2.8 to 5.4 seconds on average to reply and lost tone of voice, background sounds and who was speaking. In May 2024, GPT-4o handled text, images and audio in one model and could reply to speech in about 320 milliseconds on average, close to human conversation speed. Models that take audio in and give audio out directly are called speech-to-speech or realtime models.

   1. Old way: Speech-to-text↓→
   2. Text model↓→
   3. Text-to-speech

   Claude: tap the sound wave icon next to the microphone. Available on all plans, including Free, and works best on your phone. ChatGPT: voice chats are on every plan (limited on Free), and paid plans add voice with video. Gemini: Gemini Live is included free.

4. 4

   How does AI make images and video? Diffusion, simply

   Many image generators are built on an idea called diffusion. During training, you take real pictures and add random noise to them, a little at a time, until they are pure static, like an old TV with no signal. The model learns to undo one step of noise at a time.

   To make a new image, you start from pure static and let the model remove the noise step by step, guided by your text prompt, until a clear picture appears. It is like a sculptor who starts with a rough block and chips away until the statue matches the description. A 2021 paper showed you can do this in a compressed version of the image, which made it far cheaper, and used attention to follow the text prompt.

   Video works on the same idea across many frames. Google’s Veo 3 (May 2025) generates video together with sound, and Veo 3.1 followed in October 2025. ChatGPT Images 2.0 (April 2026) can plan before it draws and writes text inside images more accurately, with gains in Hindi and Bengali.

   1. Random noise↓→
   2. Remove a little noise (guided by your prompt)↓→
   3. Repeat many times↓→
   4. Finished image

   Key idea

   Diffusion models do not paste photos together. They learn to turn noise into a picture that fits your words.

5. 5

   Using multimodal AI well (and safely)

   Good input makes good output. Send clear, well-lit photos, crop to what matters but keep the context, and say exactly what you want back. For voice, a quiet room helps. For images, describe the subject, style, layout and any text you want, then ask for changes one at a time.

   Check before you trust. AI can misread a blurry digit on a bill or a word on a board. Do not upload ID cards, medical reports or other people’s photos unless you are comfortable with that app’s privacy terms, and never use image or voice tools to imitate a real person.

   • Ask it to mark anything it cannot read instead of guessing
   • Ask for a table when extracting numbers from a photo, then spot-check two or three
   • Treat generated images as illustrations, not evidence of real events

From the people who built it

Straight from the source

The original idea, then the same idea taken apart, then how you use it today.

OpenAI (the GPT-4o team)

Built GPT-4o, a single model for text, images and audio

Hello GPT-4o (announcement, May 2024)

1 · Understand

OpenAI trained one model end to end across text, vision and audio, so the same network handles every input and output. The “o” stands for “omni”.

2 · Break it down

• Before, voice mode chained three models, and the main model only ever saw text.
• That lost tone, laughter, background noise and who was speaking, and replies took seconds.
• One model hearing audio directly could reply in about 320 milliseconds on average, close to human conversation.

3 · Use it

In voice mode you can interrupt, change your mind mid-sentence or ask it to slow down, just as you would with a person. Use it to practise interview answers out loud or to quiz yourself on notes.

Jonathan Ho, Ajay Jain and Pieter Abbeel

Authors of the 2020 paper that made diffusion models produce high-quality images

Denoising Diffusion Probabilistic Models (paper, 2020)

1 · Understand

They showed that a model trained to remove noise from images, one small step at a time, can generate new high-quality pictures, reaching a “state-of-the-art FID score of 3.17” on a standard image test.

2 · Break it down

• Training: add noise to real images bit by bit, and teach the model to undo each step.
• Generating: start from pure noise and run the undo step many times.
• Later work added text prompts and ran the process on a compressed image, which led to today’s text-to-image tools.

3 · Use it

When an image comes out wrong, small prompt changes help because each generation starts from fresh noise. Generate a few versions, pick the closest one, then ask for one change at a time.

6 terms

Words to know

Multimodal AI

AI that can take in or produce more than one type of data, such as text, images, audio and video.

Vision (image input)

A model’s ability to read and reason about images you upload. The image is cut into patches that become tokens.

Speech-to-text

Turning spoken audio into written words. Also called automatic speech recognition (ASR).

Text-to-speech

Turning written text into spoken audio in a natural-sounding voice.

Speech-to-speech (realtime) model

A model that takes audio in and gives audio out directly, which is faster and keeps tone of voice.

Diffusion model

An image or video generator that starts from random noise and removes it step by step, guided by your prompt.

2 videos · plays here

Watch

Hand-picked from the people who explain this best. The video loads only when you press play.

Hands-on · 10 minutes or less

Try it

Three senses in ten minutes: see, hear and draw

1. 1See: take a photo of a page of your notes, a whiteboard or a bill. Upload it to Claude, ChatGPT or Gemini and ask: “Turn this into clean notes. Write [unclear] wherever you cannot read something.” Check three details against the photo.
2. 2Hear and speak: on your phone, open the Claude or ChatGPT app and start voice mode. Ask it to quiz you on those notes out loud. Try interrupting it mid-answer and see how it handles that.
3. 3Draw: in ChatGPT or Gemini, ask for “a simple poster for a college coding club with the words ‘Code Club Tuesday 5 pm’ in English and Hindi”. Check whether the text is spelled correctly.
4. 4Ask the same tool: “Explain in 3 sentences how you made that image.” Compare its answer with the diffusion steps in this lesson.

Open Gemini (free)

3 questions · instant feedback

Check yourself

Pick an answer to see if you are right, and why.

Question 1

Question 1 of 3

How does a multimodal model “see” an uploaded photo?

AIt looks the photo up on Google ImagesBIt cuts the image into small patches and turns them into tokens the model can processCA human describes it firstDIt reads only the file name

Question 2

Question 2 of 3

Why are single speech-to-speech models better for voice chat than chaining three models?

AThey are cheaper to trainBThey reply faster and keep information like tone of voice that is lost when speech is turned into plain textCThey do not need any training dataDThey only work in English

Question 3

Question 3 of 3

What does a diffusion model do when it creates an image?

ACopies and blends existing photos from the internetBStarts from random noise and removes it step by step, guided by the promptCDraws the image line by line like a penDSearches a database of stock images

Lesson 9 of 36

Done? Keep going

Mark this lesson complete to keep your place. Progress is saved on this device, no login.

Mark lesson complete

Up next · Lesson 10 · 9 minYour first promptA prompt is simply the message you type to an AI chatbot. Open any free one (ChatGPT, Claude or Gemini), describe your task the way you would brief a smart new colleague, then keep talking: correct it, add details and ask for changes. Treat the first answer as a draft, not the final word.← Previous: Reasoning models and thinking

People also ask

Questions people ask

What is an example of multimodal AI?+

Uploading a photo of a maths problem to ChatGPT or Claude and getting a worked solution is multimodal: image in, text out. Talking to Gemini Live or Claude’s voice mode (audio in, audio out) and asking ChatGPT to make a poster (text in, image out) are also examples.

Is ChatGPT multimodal?+

Yes. ChatGPT can read images and files, hold voice conversations, and generate images. Claude can read images and documents and has voice mode. Gemini can read images, hold voice chats with Gemini Live and generate images, with video generation on paid plans.

What is the difference between multimodal AI and generative AI?+

Generative AI creates new content, like text or images. Multimodal AI works with more than one type of data. Many tools are both, such as a chatbot that reads your photo and writes an answer, but a model that only reads images and labels them is multimodal without being generative.

16 sources · checked Nov 2026

Sources and further reading

• Dosovitskiy et al. (2020): An Image is Worth 16x16 Words (Vision Transformer)
• Radford et al. (2021): Learning Transferable Visual Models From Natural Language Supervision (CLIP)
• Claude Docs: Vision
• OpenAI (Sept 21, 2022): Introducing Whisper
• OpenAI (May 13, 2024): Hello GPT-4o
• OpenAI (Aug 28, 2025): Introducing gpt-realtime
• Claude Help Center: Use voice mode
• ChatGPT pricing (voice and image features by plan, Nov 2026)
• Google: Gemini subscriptions (features by plan, Nov 2026)
• Ho, Jain and Abbeel (2020): Denoising Diffusion Probabilistic Models
• Rombach et al. (2021): High-Resolution Image Synthesis with Latent Diffusion Models
• 9to5Google: Google Veo guide (version history)
• eWeek (Apr 2026): OpenAI debuts ChatGPT Images 2.0
• University of Southampton: What is multimodal AI?
• StartupHub.ai: IBM Master Inventor explains multimodal AI (video summary)
• Podwise: chapters of “But how do AI images and videos actually work?”

Related on this site

Previous: Reasoning models and thinkingNext: Your first promptTokens and context windowsHow voice AI agents workWhy AI makes things upFree AI resources