What is a token?

What a token is

A token is a piece of text. Usually a whole common word. Often a fragment of a longer word. Sometimes just a space and a letter, or a single mark of punctuation.

Models do not read words. Before the model sees your message, a small program called the tokenizer chops the text into these pieces and swaps each piece for a number. That list of numbers is your message, as far as the model is concerned.

That step is tokenization: turning text into a list of tokens, and each token into its number. It happens before the model runs, it is not learned by the model, and it is entirely mechanical - the same text through the same tokenizer gives the same numbers every time. How the tokenizer decided where the cuts go is its own story, told in What is BPE?; this article stays on the reader's side of it.

So yes, your text becomes numbers - in two steps, not one. First the text is cut into pieces. Then each piece is swapped for its row number in a fixed list. That number is an address, not an amount: token 279 is not bigger or better than token 102, it just sits on a different line.

A token is not a word. It is the piece of text a model was taught to see - and the unit everything about the model is counted in.- the one line to keep from this article

The whole idea in one picture. Every shade below is one token - some are whole words, one is half a word, and the commas and the full stop are tokens of their own:

So the three numbers are never the same, and their order is predictable: more characters than tokens, and more tokens than words. That middle relationship is the one worth memorising.

Tokens almost always outnumber words. Budget about four tokens for every three words of English prose - and more than that for code, names, long numbers, and any language that is not English.- the sizing rule to keep in your head

Two things follow, and between them they explain why the word is on every pricing page. The model cannot look inside a token, because a token is the smallest thing it receives. And any piece of text has a token count of its own - not its word count, not its character count.

One more thing to know early: the tokenizer belongs to the model. Each model family ships its own list of tokens, so the same paragraph can be 180 tokens for one model and 210 for another. A token count means nothing without the name of the model that counted it.

Why not words

A token could just as easily have been a word. So why isn't it? Because there are only three ways to cut text, and two of them break.

• Cut on spaces - one token per word. Breaks on the first word the list was never built with: a product name, a typo, a variable name, any inflected form in a language richer than English.
• Cut into single letters. Covers everything and multiplies the bill, because the same text arrives as several times as many tokens, each costing memory and compute in every layer.
• Cut into pieces - what models actually do. Common words stay whole; rare ones are assembled from fragments that are themselves common, so tokenization arrives as Token + ization.

That is the whole trade: cover every word that will ever exist, at close to word-length cost. What is BPE? weighs the three options properly, with the price of each. What matters here is the consequence you feel on the invoice - once tokens stop matching words, your word count stops predicting anything.

Two lines can carry the same number of words and cost you twice as much. The words are what you wrote; the tokens are what you are charged for.- the rule that catches every team once

What the split does to your text

Two schemes produce the vocabularies you will actually meet: BPE (byte pair encoding) and WordPiece. Both are trained on a corpus before the model itself is trained, and both work bottom-up: start from the smallest pieces, then repeatedly glue together the pair that pays off most, until the vocabulary reaches its target size. BPE merges the pair that occurs most often; WordPiece merges the pair that most improves the likelihood of the training text.

What either one leaves behind is not a program: just a vocabulary and a merge list, two plain text files that ship with the model. What matters here is what they do to your text, and three details explain most of the surprises.

• A word is not always the same token. Whether a space sits in front of it changes its id, so the same word can cost differently at the start of a line than it does mid-sentence.
• Nothing you type can fail to encode. Emoji, unfamiliar scripts, broken markup: worst case they cost several tokens each, but they are never unknown.
• Numbers get chopped. Long digit strings are split into groups that owe nothing to arithmetic, so 1234567 may arrive as three unrelated pieces.

If you want to see the split rather than read about it, paste your own text into a tokenizer viewer - the references below include one. It is a five-minute exercise that permanently improves your intuition. And for the algorithm that chose those splits in the first place, carry on to What is BPE?.

How big is the list?

A fair question follows from section 01: if every token carries a number, and a dictionary runs to hundreds of thousands of words, shouldn't those numbers get enormous? They don't. The list is capped on purpose, and the cap is smaller than the dictionary.

Why a cap that small is enough is the payoff of section 02. A vocabulary of words would need a slot per word, and that list never closes - not once you count inflections, proper names, product names, typos and every identifier in every codebase. A vocabulary of pieces has no such problem. Around a hundred thousand fragments cover text that will never stop inventing new words, because anything unfamiliar is assembled from pieces already on the list, and the pieces bottom out at raw bytes.

The list is not a dictionary. It is a set of pieces chosen so that everything else can be built out of them - which is why a hundred thousand entries outlast a language of millions of word forms.- why the cap holds

And the size of the number itself costs nothing either way. Three details settle that:

• Low ids are the common pieces. The list is built in merge order, not alphabetical order, so single bytes and frequent words get the low ids and rare fragments the high ones. z is nowhere near the end.
• The width is fixed. An id is stored as a fixed-width integer - four bytes whether it reads 5 or 199,998. A large id is not a large thing to store or move.
• The id survives exactly one step. The model uses it as a row number in the embedding table, takes that row, and never looks at the number again. Nothing ever adds, compares or scales ids.

Which leaves the opposite question: if ids are this cheap, why not make the list enormous and cut every word down to one token? Because the vocabulary is paid for at both ends of the model. Going in, every entry needs its own row in that embedding table. Coming out, the model scores every token in the list on every single pass in order to pick the next one. Double the vocabulary and you double both, on a model that already runs once per token it writes. A bigger list buys shorter sequences and charges for it in width - and somewhere between 100,000 and 256,000 is where that trade currently settles.

So the number to watch is never the id. It is how many tokens you send.

LLMs and tokens

For a large language model, tokens are not an implementation detail. They are the entire interface. An LLM does one thing: given a list of tokens, predict the next token. Everything a chat assistant appears to do is that operation, repeated.

• Going in: input tokens. The system prompt, the conversation so far, a pasted document, the tool definitions - all of it becomes one list of token ids, and the model reads the whole list in a single pass.
• Coming out: output tokens. The reply is not written in one go. The model predicts one token, appends it to the list, and runs again over the slightly longer list. Then again. A 400-token answer is 400 passes over a growing input, which is why answers stream in piece by piece rather than appearing at once.

Three properties of the model follow directly from that, and each one is measured in tokens rather than words.

• The context window is a token budget. It is the maximum length of that list, and everything competes for the same space: instructions, history, documents, tool definitions, and the reply being generated. Fill it with a long file and the answer gets squeezed. Section 06 takes the budget apart.
• The vocabulary is a closed set. The model can only ever emit tokens from its own list. It cannot invent a new one, which is why a word outside the vocabulary comes back assembled from fragments.
• Structure and stopping are tokens. A chat is one long document, and the markers that separate the system prompt from your turn are special tokens - things like <|im_end|> that you never type and never see. The model finishes by emitting an end-of-turn token, or by hitting the maximum output length, or by producing a stop sequence you defined.

And nothing outside that list exists for the model. No memory of yesterday, no access to the file you mentioned, no knowledge of who you are - unless it arrived as tokens in this request. Inside an LLM follows one question through those passes in detail.

The context window

After its price, the context window is the number most often quoted about a model, and it is a token count: the maximum length of the list from section 05, reply included. It is not memory and it is not storage. It is how much text the model can have in front of it during one request - and every request starts from an empty window.

Everything competes for the same space, and all of it is counted in tokens before the model runs:

• The system prompt - your instructions, plus any the provider adds.
• Every earlier turn - your messages and the model's replies, sent again in full each time.
• Attachments - a pasted document, a PDF, an image. Each is converted to tokens and takes room in the window exactly as text does.
• Tool definitions and tool results - the schemas you declare, and whatever your tools return.
• The reply itself - including any thinking the model does before answering. The output has to fit in the same window as the input.

When the list is too long, the model does not quietly drop the oldest part. If the input alone is over the limit, the request is refused before a single token is read. If the input fits but the reply runs out of room, the reply stops where the window ends. Chat products manage this for you by trimming or summarising the oldest turns; over the API the budget is yours to manage, which is what What's in an agent's context? is about.

Two consequences are worth stating plainly. A bigger window is not free attention: as the list grows, the model's recall of what is in it degrades, so the window is a ceiling, not a target. And because every turn re-sends the whole list, the window also sets the shape of the bill - the subject of the next section.

The window is a per-request budget in tokens, not a memory. Whatever is not in this request's tokens does not exist for the model - however long the conversation has been running.- the context window, in one line

Why you pay per token, not per word

Every provider meters the same two numbers: tokens in and tokens out. Words appear nowhere on the invoice, and the two counts are not priced the same - output costs several times more than input, for the reason in section 05.

Rates are quoted per million tokens, written MTok, which keeps the arithmetic simple. Anthropic's published list rates in September 2026 give the shape of it - a ten-fold spread between the cheap tier and the expensive one, and output at five times input on every single tier.

Put a real job through the meter. Sonnet 5 at $2 in, $10 out, one question over a 40,000-token document - roughly 30,000 words, a long report:

That last line is the part that surprises people. Nothing was uploaded once and remembered; every turn re-reads everything. It is also why what an agent costs is a question about tokens rather than about requests - an agent loop re-sends its own history dozens of times. Prompt caching exists for exactly this case, and the last two lines above are why it is the first optimisation anyone reaches for.

Your quota is denominated the same way: maximum response length is a token count, and throughput limits are quoted in tokens per minute, not requests per minute.

The four-tokens-per-three-words rule from section 01 is enough to size a feature before you build it - 1,000 tokens is roughly 750 words of English - but treat it as a sanity check, never as a number to bill against.

Two practical habits follow. First, count instead of estimating when the number matters: every major provider publishes a tokenizer library or an endpoint that returns the exact count for a given model. Second, watch what language your text is in. Tokenizers are trained on corpora that skew English, so the same sentence in Russian, Hindi or Japanese frequently costs noticeably more tokens than its English translation - the same meaning, a longer bill, and less room left in the window.

Tokens, not megabytes, not effort

The meter counts one thing. It does not count bytes: a 1 MB file of prose and a 1 MB file of JSON are the same size on disk and nowhere near the same number of tokens, because braces, quotes, digits and indentation each cost separately - the JSON line in section 02 was four words and thirteen tokens. What you send is measured by the split, never by the file size.

It does not count difficulty either. A trivial question and a hard one of the same length cost the same to send, and the model does the same amount of arithmetic per token whichever it is. The only way hardness reaches the invoice is as more tokens: a longer answer, or - on models that think before they answer - the thinking tokens, which are billed as output at the output rate. A hard problem that produces a short answer is cheap; an easy one that produces a long answer is not.

You are not renting the model by the megabyte or by the problem. You are buying tokens in and tokens out, at a rate set per model.- what the invoice measures

Why Fable costs five times Sonnet

The rate is per token, but it is not the same rate for every model. Haiku 4.5 charges $1 per million input tokens, Sonnet 5 $2, Opus 5 $5 and Fable 5.1 $10, with output at five times input on each. Same kind of token, a rate that spans ten to one - so the difference cannot be in the tokens. It is in what one token sets in motion.

Every token you send passes through every layer of the network, and every token the model writes is a full pass over the whole list. A model in a higher tier is a bigger network - more weights, more layers, more arithmetic for the same token - and that arithmetic is hardware time, which is what the provider is actually selling. Anthropic does not publish parameter counts, but the price ladder tracks the tiers, and it moves in one direction: the same question, the same token count, and each of those tokens now costs a larger network's worth of compute.

A token is not priced by what it says. It is priced by how much machinery it has to pass through - and a bigger model is more machinery per token.- why the tiers exist

One further detail hides in the same table. The tokenizer belongs to the model, and Claude models from 4.7 onwards ship a newer one that cuts the same text into roughly 30% more tokens than the tokenizer Haiku 4.5 and Sonnet 4.6 use. So a paragraph does not have one token count across the lineup: the model's rate and its split move together, and a cost estimate needs both.

The tiers exist because most requests do not need the top one. Routing the easy majority to a cheaper model and reserving the expensive one for the work that needs it is the biggest cost lever after caching - and, like caching, it is a decision about tokens: which model reads them, and how many.

What tokens explain

Tokens explain the model's best behaviour and its silliest failures, and it is the same mechanism both times. Start with a question it handles perfectly.

That works because the answer is a token the model has seen follow that pattern countless times. Not one step of it needed a letter: the question went in as eight pieces, the answer came out as two, and the whole exchange is ten tokens plus the end-of-turn marker.

Now the other side. A specific class of failure stops looking mysterious once you know the input is tokens - the model is not being careless, it is answering a question about letters without having been shown any letters.

The same root cause shows up in several familiar places:

• Counting and reversing letters is guesswork about token spelling, not inspection.
• Arithmetic on long numbers is done over digit groups the tokenizer chose, which is why errors cluster in the middle of long numbers.
• Rhyme, syllables and wordplay are harder than their difficulty suggests, for the same reason.
• Whitespace and formatting cost money. Pretty-printed JSON and heavily indented code carry real token weight compared with their compact equivalents.

The fix for the first three is not a better prompt but a different tool: let the model call code to count, reverse or calculate, and let it read the result. Inside AI tool use covers that handshake.

What to keep: a token is the unit the model reads, the unit you are billed in, and the unit your limits are written in. When a number about text does not add up, convert it to tokens first - the answer is usually there.