How Language Models Read Prompts: A Beginner Guide · Lesson 1 · 3 minutes
Words into tokens
Define a token as a text chunk.
Play this lesson in your browser. No account needed.
Text into chunks
Computers do not read words like humans do. A computer cuts text into small pieces called tokens.
How does an AI model read incoming text?
- It cuts the text into small pieces called tokens.
- It reads whole sentences all at once without stopping.
- It only looks at whole full words.
Answer: It cuts the text into small pieces called tokens.
Models cut text into smaller chunks called tokens before reading.
What is a token?
A token can be a single letter, a whole word, or part of a word.
Complete the rule about tokens.
Models do not read full words. They split text into ___.
Answer: Models do not read full words. They split text into tokens.
Tokens are text chunks, not always whole words or single letters.
Is a token always just one letter?
- Yes, every letter is one token.
- No, a token can be a word, part of a word, or a letter.
Answer: No, a token can be a word, part of a word, or a letter.
A token can be a letter, a word piece, or a whole word.
Build the statement explaining how AI processes text.
Answer: An AI model cuts text into tokens
Computers process text by breaking it into token chunks.
Put the steps in order for how a model reads text:
Answer: You type a prompt. → Text gets chopped into tokens. → The model processes the tokens.
Text must be chopped into tokens before the model processes it.
What to remember
- Computers do not read whole words like humans do.
- Text is chopped into small pieces called tokens.
- Tokens can be letters, whole words, or parts of words.
Next in this course: Rare and common words. It carries on in the Breni app.
The full How Language Models Read Prompts: A Beginner Guide path