What is BPE: Byte-Pair Encoding?

What tokenization is

A model cannot read text. It works only with numbers, so the text has to be turned into numbers first. That step is tokenization, and it matters more than it looks: it decides what the model can see, how long each input is, and how much each request costs. The pieces it produces are tokens. What a token is, why you are billed in them, and how many fit in a context window is What is a token?; what the model does with them once they arrive is Inside an LLM. This article is about the process in between: the algorithm that decides where the cuts go.

Tokenization produces one thing: a list of ids - line numbers in the tokenizer's vocabulary. The model never sees your words, only those numbers.- the whole job, in one line

Why numbers? Because a model is arithmetic and nothing else. Every layer inside it multiplies matrices of floating-point numbers, and there is no operation anywhere in that machinery that accepts the letter c. So each piece of text is swapped for an integer, and that integer is used as a row number into a table of learned vectors. What actually reaches the model is a list of row numbers.

So the numbers are not quantities, and nothing is calculated from them. They are ids - line numbers in a table, working the way a page number works: page 2061 does not turn into the words printed on it, you simply open the book there.

One note on words before going on. This article says symbol, piece and token for the same thing - an entry in that list. Symbol while the algorithm is still counting, token once the model is reading; nothing changes but the point of view.

Two runs, not one

Two tables sit between your text and the model: the tokenizer's vocabulary, which turns a piece of text into an id, and the model's embedding table, which turns that id into a row of decimals. Where do they come from? From two separate training runs, one after the other. Each run takes something in and leaves something behind.

Run 1: the tokenizer run. The run finds the two symbols that sit next to each other most often, glues them into one, and repeats until the vocabulary is as big as you asked for - the loop in section 08. It takes a few hours on one machine. No network, no weights, and the model it will serve does not exist yet.

• In: a corpus - a large sample of text, the same kind of writing the model will later read, boiled down to a list of distinct words with a count next to each.
• Out: two plain text files. vocab.json lists the pieces with their numbers; merges.txt lists the gluing rules. Together they are the tokenizer, and they are small - about 1.5 MB for GPT-2, a few megabytes for a 128,000-token vocabulary.

Run 2: the model run. An LLM is a very large set of numbers, and at the start of this run every one of them is a small value straight out of a random number generator - a placeholder that means nothing, there only because the numbers have to start somewhere. The model is not empty and does not grow: how many numbers there are is settled before the run begins, so GPT-2's 124 million all exist from the first second, and the file is the same 548 MB on day one as at the end. Training changes their values, never their count. The embedding table is part of that set, not a thing beside it: one row for each line of run 1's vocabulary, filled with those same random values. Training fixes them: guess the next id, check how far off the guess was, adjust, and do it again over the whole training set. The text is never kept - it is read, used to correct the numbers, and dropped. The details are their own article: How a model learns.

• In: those two files, plus the training set - the full collection of text the model learns from, far larger than the sample, and often the very text the sample came from. The model never reads it as text; run 1's files turn it into ids first.
• Out: one file, model.safetensors, 548 MB for GPT-2. That file is the LLM.

A word on weights, since the file is named after them. A weight is one single number the model multiplies something by - one of the numbers that started random and that training spent the whole run adjusting. Their count is what a model's size means: GPT-2 has about 124 million of them, its largest version 1.5 billion, and parameters is the same word for the same thing. Stored as 4-byte decimals, 124 million numbers account for most of the 548 MB on disk. And there is nothing else in the file - no code, no vocabulary, no text - so the weights and the LLM are two names for one thing, not two things shipped together.

The file is the weights, and the weights are the LLM - one thing, three names. The embedding table is not a fourth: it is one layer inside those numbers, about a third of them in GPT-2, and the layer whose height the vocabulary sets.- what contains what

The filename is convention, not meaning. Most open models, GPT-2 included, are published on Hugging Face, the public hub for sharing them (GPT-2's files sit at huggingface.co/openai-community/gpt2); model is what the hub calls the main weights file, and .safetensors is the format it introduced: a plain container of number arrays that cannot run code when it loads. What is not in it is the tokenizer: the vocabulary and the merge rules ship as their own files beside it, which is the whole point of section 03. GPT-2 is this article's example because OpenAI released its weights in 2019 - an open-weights model is a file anyone can download, run and inspect, where a closed model such as GPT-4 or Claude is reachable only through an API and its numbers never leave the vendor.

Two runs, in order · a sample in, two files out; those files and the full text in, one weights file out

The two outputs are the two things that ship. Notice the direction of travel. The vocabulary only ever goes in to run 2: fixed before the first step, never changed by it. The embedding table only ever comes out: it is not something the model is given, it is something the model ends up with.

Two rules follow. The tokenizer has to be frozen before model training starts, or the rows would end up pointing at the wrong pieces. And the model can only write a token that has a row, which is why its vocabulary is a closed set. The next two sections take the outputs in order: run 1's two files, then run 2's embedding table, which the model builds to match one of them.

Run one: the two files

What the tokenizer run leaves behind is not a program. It is two ordinary text files, shipped alongside the model - plain text, a megabyte or two all in, and between them the entire tokenizer. One of them, the vocabulary, will be paired with the model's embedding table in section 04, across the run boundary; here it is paired with the other file the same run wrote. They do two very different jobs, and only one of them is about numbers.

• merges.txt - where the text gets cut. The ordered merge rules - which pair of pieces gets glued into one, in the order the rules were learned - replayed from the top on every piece of new text. This is the only learned half of the tokenizer run and the hard half of the problem: deciding that Tokenizers breaks as Token + izers, and not as To + ken + izers.
• vocab.json - what number each piece gets. This is the vocabulary from section 01, as a real file on disk. A dictionary lookup, and the easy half: if ids were all a tokenizer had to produce, this file on its own would be the whole tokenizer.

The example below uses a toy corpus - cat, the, bat, sat - small enough to read whole, with ids numbered from 1.

Two files, one job · the merges decide where the splits go, the vocabulary turns the pieces into ids

Read the two together and the whole thing demystifies. The single characters were in the vocabulary from the start; at is there because a + t was worth merging, and cat because c + at was. Every id the model receives is a line in the first file, and every split the tokenizer performs is the second file replayed from the top - fast, deterministic, and with no neural network involved at all. This step is table lookup.

The same two files on a sentence the toy corpus never saw. Each word is split into characters, the merge rules are replayed from the top inside each word, and whatever pieces remain are looked up. One of the words is on the six-line list above; the rest use lines further down the real files.

Those are toy numbers. Here are the real ones - GPT-2's published files, listed next to the weights they ship with:

Two files you could open in a text editor, deciding what a 548 MB model - the whole of run 2's artifact, the LLM itself - is able to see at all. The 50,000 rules and the 50,257 entries are the same number twice, near enough: the vocabulary is 256 starting symbols - the byte values, for reasons section 06 gets to - plus one merged token per rule, plus a single end-of-text marker. Both files scale with the vocabulary and nothing else, so a 128,000-token tokenizer is a few megabytes rather than one.

One packaging note, because a modern repository often contains neither filename: the two tables are now usually shipped as a single tokenizer.json - 1.36 MB for GPT-2 - holding the vocabulary, the merge list and the pre-tokenization pattern together. Same two tables, one file.

Only one of the two files is about numbers. The other decides where the text breaks - and that is the half the tokenizer run exists to learn.- why a tokenizer is two files and not one

Run two: the embedding table

Of the two files, vocab.json is the one that crosses the run boundary: it is the tokenizer's table, and the model keeps a table of its own to match it, row for row. The two are owned by different halves of the system. The vocabulary belongs to the tokenizer: every piece it knows, one per line. The embedding table belongs to the model: it is the LLM's first layer, shipped inside the weights file itself, and it holds one row of numbers for each line of that vocabulary. That row - an ordered list of decimals, 768 of them in GPT-2 and a few thousand in larger models - is the piece's embedding vector. They stay separate because different runs build them - the vocabulary by counting text, the embedding table by training a model, the process How a model learns follows step by step - and the id is the only thing that ever passes between the two. It is also why a different tokenizer hands the same sentence different numbers.

One id, two tables · the tokenizer assigns the number, the model is addressed by it

One thing the embedding table is not: a vector database. Both hold vectors, and that is where the resemblance ends. A vector database stores embeddings of your documents and is searched by similarity - you hand it a vector, it returns the nearest ones, which is the retrieval step in Inside a RAG pipeline. The embedding table is never searched and nothing is ever compared: the id is a row number, the row comes back, done. Nor is it a file you can swap or a service you can query - it is part of the model's weights, learned during training and frozen with them. That is the sense in which it is the model's table: not a store the LLM consults, but the LLM's first layer.

The tokenizer is the dictionary that gives a piece a number. The embedding table is the table that gives that number a learned vector.- the two tables in one line

The tokenizer never touches the second table, and the model never sees your text. The id is the whole conversation between them. What is in those rows is Inside an LLM.

Back to the tokenizer's own two files. Both are fixed before a single word is processed - which pieces exist at all, and what number each one carries - and the only real decision behind them is the first one: where do you cut?

The tokenization challenge

There are only three answers to that question. The first two are the obvious ones, and both of them break. Take one sentence through each.

Option one: one token per word

Pros

1. Every unit carries meaning on its own.
2. The sequence is as short as text can be made.

Cons

1. The vocabulary has to be enormous, and frozen before training starts. English alone has more than 170,000 words in current use, before inflections, proper nouns, product names and typos.
2. Anything outside that list becomes <UNK>, and everything the word carried is lost.
3. Morphologically rich languages - German, Finnish, Turkish - generate word forms faster than any fixed list can hold them.

<UNK> is short for unknown: the one placeholder a word-level tokenizer writes for every word it has never seen. The word is not split or approximated - it is replaced, and whatever it carried is gone before the model sees it.- the unknown token

Option two: one token per character

Pros

1. A tiny vocabulary: about a hundred symbols covers everything.
2. Nothing is ever unknown, because anything can be spelled out.

Cons

1. The same text becomes several times more tokens - 44 here against 10, and around five times as many for English prose in general.
2. Attention cost grows with the square of the sequence length - the mechanism Inside an LLM walks through - so every one of those extra tokens is expensive.
3. A single character carries almost no meaning on its own, so the model rebuilds every word from scratch, every time.

The out-of-vocabulary problem

The word-level failure has a name. Suppose a translation system trained on news articles meets this:

That is the observation the whole field is built on, and Sennrich, Haddow and Birch stated it plainly in 2016: rare words are usually compositional. Split them into the right smaller units and a network can translate - and produce - words it has never once seen in training. The unknown token stops being necessary.

Option three: one token per piece

Which is where BPE comes in. Option three keeps frequent words whole and assembles rare ones out of fragments that are themselves common, and byte-pair encoding is the most widely used way of deciding which fragments those are. Every model traced through Inside an LLM reads its input this way. Everything after this section is how it picks them.

Pros

1. A manageable vocabulary, 50,000 to 256,000 entries.
2. Sequence lengths close to the word count.
3. Coverage of any text at all - anything unfamiliar is composed from pieces already on the list, so <UNK> disappears.

Cons

1. The pieces are chosen by counting, not by meaning, so they respect neither morphemes nor digits nor word boundaries unless you force them to.
2. The token count of a text is no longer predictable from its word count - a piece is not a word, so counting one tells you little about the other.

Cut text into pieces and any word ever written can be covered, at about one token per word. BPE is the most common way to choose the pieces.- the trade, in one line

The history of BPE

BPE was not designed for language, and it was not designed for models. It arrives in AI from data compression, by way of machine translation, and each hand-off changed what the loop counts while leaving the loop itself alone.

1994: a data-compression algorithm

In February 1994 Philip Gage published A New Algorithm for Data Compression in The C Users Journal. The target was byte streams, and the algorithm was four steps long.

1. Find the most frequent pair of consecutive bytes in the data.
2. Replace every occurrence of that pair with a new, unused byte.
3. Record the replacement in a lookup table.
4. Repeat until no pair occurs more than once, or until the compression is good enough.

Run it on a word you already know and the whole idea fits in three passes.

Two of those properties carried into tokenization. Two did not.

• The loop carried. Steps 1 and 2 are the whole of BPE: find the top pair, replace every occurrence of it. A frequent pair is worth a symbol of its own - as true of ing in English prose as of a repeated byte pattern in a file.
• The table carried. Step 3. Compression is worthless if you cannot invert it, so the substitution table ships with the data. In a tokenizer that table is merges.txt, and inverting it is how decoding works.
• The unused byte did not. Step 2 needed a genuinely unused byte for every rule, so Gage's compression runs out of room once the byte space is exhausted. A tokenizer just allocates the next integer, and never runs out.
• The stopping rule did not. Step 4 stops when nothing repeats, because the goal is a smaller file. A tokenizer stops when the vocabulary reaches a size you picked in advance - which is how that number ends up being the only real knob in the whole algorithm.

2016: subwords for translation

Twenty-two years later, Sennrich, Haddow and Birch pointed the same loop at a different problem: rare words in neural machine translation. Rather than compressing bytes for storage, they compressed characters into subwords, learned from a corpus - the same word-and-count list section 02 introduced, drawn here from the text the translation system would have to handle. Three modifications, all of which survive today.

• Characters, not bytes. The base vocabulary is the set of characters that appear in the corpus.
• An end-of-word marker. Writing </w> at the end of each word keeps est at the end of a word distinct from est in the middle of one, and makes the split reversible.
• Frequency weighting. Pairs are counted across the corpus weighted by how often each word occurs, not once per distinct word.

2019: byte-level BPE

Character-level BPE still has a hole in it: a character the training corpus never contained has no representation, so an emoji or an unfamiliar script falls back to <UNK>. GPT-2 closed the hole with three changes that are now standard.

• The base vocabulary is the 256 byte values. Not characters - bytes. Any text encodes to UTF-8, UTF-8 is bytes, so nothing can fail. An unfamiliar emoji can cost up to four tokens rather than one, but it is never unknown, and the unknown token disappears from the design entirely.
• The leading space belongs to the token. " the" and "the" are separate entries with separate ids. GPT-2 prints that space as Ġ in vocabulary dumps so it is visible; in the data it is an ordinary space byte.
• The text is split before BPE runs at all. A fixed pattern - pre-tokenization - cuts the input into runs of letters, runs of digits, runs of punctuation, contractions and whitespace, each run keeping the single space in front of it. Merges are then only ever allowed to act inside those chunks.

Pre-tokenization does more than tidy up. Without it the merge loop happily learns tokens that span word boundaries, because a space followed by a common word is a frequent pair like any other. Train a small BPE on a megabyte of Shakespeare with no pre-tokenization and the vocabulary fills with entries like " in the ", "lord, " and "OF YORK:\n" - each one a perfectly good compression of that corpus and a perfectly useless unit of language. Pre-tokenization spends the vocabulary on word pieces instead.

One inherited quirk is worth knowing. That pattern puts a run of digits in its own chunk but never caps how long the run can be, so long numbers get chopped into whatever groups the merge list happened to learn, owing nothing to place value. Later tokenizers patched it by splitting digits into fixed groups of one to three. It is the clearest case of a general rule: arithmetic weakness in a model is often a tokenizer decision, not a reasoning failure.

The cousins, and what ships today

BPE won, but it did not win alone, and it does not ship the way the 2016 paper described it. Two other algorithms build the same kind of list by a different route, and the tokenizers in production today pack the two files into one - or into none you are allowed to see.

Since then: two cousins

Different loops, same artifact - a fixed vocabulary of pieces the model is then built around - so everything in sections 02 to 04 holds for all three.

WordPiece is the tokenizer behind BERT (Google, 2018) and its descendants - DistilBERT, ELECTRA, and most of the encoder models still used for classification and search. It runs the same merge loop as BPE but scores a pair differently: not by how often it occurs, but by how much more often it occurs than its two halves would predict - roughly the count of the pair divided by the counts of its two parts. A pair of rare parts that almost always appear together beats a pair of very common parts that merely happen to sit next to each other. Two traces of it in the wild: a continuation piece is written with a leading ## (play, ##ing) where GPT-2 marks a word start with a leading space, and the list is small - about 30,000 entries for English BERT. Encoding new text is different too: WordPiece keeps no merge list to replay. It takes the longest piece in the vocabulary that matches the front of the word, then repeats on what is left.

Unigram works from the other end. Instead of growing a list from characters, it starts with a very large candidate vocabulary - every frequent substring in the corpus - and prunes: on each pass it estimates how much worse the corpus would be described if a piece were removed, drops the least useful few percent, and stops at the target size. The pieces that survive are the ones the corpus most needs. It ships in Google's SentencePiece library and is the tokenizer of T5, ALBERT and XLNet. Unigram also keeps something BPE and WordPiece throw away: a probability for every piece. One word can therefore be cut several valid ways, and the encoder picks the most likely split - or, during training, deliberately samples a less likely one, which makes the model more robust to typos and unfamiliar spellings.

Today: tiktoken, and tokenizers you cannot open

The tokenizer OpenAI ships is tiktoken: byte-level BPE exactly as section 06 described it, a Rust core with Python bindings, published as open source so anyone can count tokens before sending a request. Each model generation has its own encoding, and the names say how long the list is: r50k_base for GPT-2 and GPT-3 (the 50,257-entry vocabulary this article has been using), p50k_base for the Codex models, cl100k_base for GPT-3.5 and GPT-4 at about 100,000 entries, and o200k_base for GPT-4o, the o-series and GPT-5 at about 200,000. A longer list is why the same sentence costs fewer tokens on a newer model: Tokenization is two pieces under cl100k_base and one under o200k_base.

One packaging detail closes the loop with section 03. A .tiktoken file is a single list, one line per piece: the piece's bytes and its rank. There is no separate merges.txt, because the rank does both jobs at once - it is the piece's id, and it is the order its merge is replayed in. The two files were always one table read two ways. The pre-tokenization pattern travels alongside, and it is where cl100k_base caps a run of digits at three.

Anthropic has taken the other route. Claude's tokenizer is not published: there is no file to download, and the API's count_tokens endpoint returns a number without showing the split. The vocabulary and the merge list stay on the vendor's side - the same closed arrangement section 02 described for the weights - which is also why third-party Claude token counters built on cl100k_base are estimates, not measurements.

The algorithm, in depth

The whole idea fits in one sentence: two neighbours that keep turning up together deserve to become one piece. Everything else in this section is the bookkeeping that makes that sentence run.

BPE is a greedy algorithm, and the word is doing real work. A greedy algorithm builds its answer one step at a time, and at each step it takes the choice that looks best right now - the locally optimal one - without weighing how that choice constrains the steps still to come. It is deliberately short-sighted. It never backtracks, never revises, and what it produces is not guaranteed to be the best possible answer, only a good one reached quickly.

Here that means: merge whichever pair is most frequent at this moment, then look again. BPE never asks whether some other merge now would have produced a better vocabulary twenty thousand merges later, and it never undoes a merge it has already made. Given a corpus and a target vocabulary size, it does this and nothing else.

• Initialise the vocabulary with the base tokens - characters, or the 256 bytes.
• While the vocabulary is smaller than the target: count every adjacent pair, find the most frequent one, add the merged pair to the vocabulary as a new token, and replace every occurrence of it in the corpus.
• Record each merge, in order. That order is the tokenizer.

Written out, the loop is about a dozen lines. The only subtlety is that the pair counts have to be recomputed after every merge, because merging e+s destroys the pair (s,t) and creates the pair (es,t).

The same loop, line by line, for anyone who does not read Python:

• vocab = base_tokens(corpus) - the starting list: every single character in the corpus, or the 256 byte values. Nothing is merged yet.
• splits = {word: list(word) ...} - every distinct word written out as separate symbols, low as l o w. This is the working copy the loop keeps rewriting.
• merges = [] - an empty list that will become merges.txt. Its order is the whole tokenizer.
• while len(vocab) < vocab_size - keep going until the list is as long as you asked for. The target size is the only knob.
• counts = count_pairs(...) - walk every word, count every pair of neighbours, weighted by how often the word occurs. (e,s) scores 9 in the toy corpus because newest occurs six times and widest three.
• if not counts: break - no pair occurs any more, so there is nothing left to merge. Stop early.
• best = max(counts, ...) - take the pair with the highest count. Ties break the same way every run, so the same corpus always yields the same list.
• splits = apply_merge(splits, best) - rewrite every word: wherever the two symbols sit next to each other, glue them into one. n e w e s t becomes n e w es t.
• merges.append(best) - write the rule down. Its position in the list is its rank, and rank 1 is replayed before rank 2 for the rest of the tokenizer's life.
• vocab.append(best[0] + best[1]) - the glued pair is a new token; it gets the next free id.
• return vocab, merges - the two files from section 03. The loop never runs again; from here on everything is replay.

Nothing is fit, nothing is optimised, nothing converges. You count, you merge the winner, you write the rule down - and the vocabulary size is the only hyperparameter in the room.- why BPE training is not model training

Hold that against the other run. Model training, as How a model learns lays out, is a loss measured and a gradient stepped, over trillions of tokens, on hardware that costs a fortune. This is a frequency count that finishes in an afternoon. The two share the word training and nothing else.

At production scale nothing about that shape changes, only the numbers: a corpus of hundreds of gigabytes, a target somewhere between 50,000 and 256,000 tokens, and a few hours on a single machine. Choosing that target is the one judgement call in the process - what a bigger vocabulary costs is a question for the token article.

Walkthrough one: the paper's toy corpus

This is the corpus from the 2016 paper, and it is worth following line by line, because everything that later surprises people about tokenizers is visible in it. Four distinct words, each with a frequency, each split into characters, with </w> closing the word.

Four steps and the two files · the counts include word frequencies, and lowest comes out as two pieces it already has

Three details in that table do real work later.

• Counts are weighted by word frequency, not by distinct words. (e,s) scores 9 because it appears in newest six times and widest three. A pattern common in the corpus wins even if it occurs in only two distinct words - which is exactly why tokenizers are so sensitive to what they were trained on.
• Ties happen, and the tie-break is arbitrary but fixed. Iteration 1 has three pairs at 9: (e,s), (s,t) and (t,</w>). Implementations settle it by first-seen or by sort order; what matters is that the same corpus always yields the same merge list, because the merge list is the tokenizer.
• The order is the output. Each merge is appended to a list, and its position in that list is its rank. Rank 1 is applied before rank 2, always, for the rest of the tokenizer's life.

Walkthrough two: "the cat in the hat"

Same loop, no word markers, straight over a running string - which is closer to how a byte-level tokenizer sees its corpus.

Notice what the loop just built without being told anything about English: a determiner, a rhyme fragment, and a digraph. Nobody supplied a rule about spelling. Three passes of counting produced units a linguist would recognise, which is the whole reason the technique works.

After training: the same rules, replayed

Once the two files exist, the loop never runs again. Tokenizing new text is a replay: split the text into its base symbols, then walk down merges.txt from the top and apply each rule wherever its pair appears, in the order the rules were learned - never by what is most frequent in the new text, and never by position. Run the five merges above on lowest, a word the toy corpus never contained, and it comes out as low + est</w>: two pieces, both already on the list, ids 15 and 13. That is the compositional payoff from section 05, and it is deterministic - the same text through the same files gives the same ids, every time, with no network involved. Decoding is the same table read backwards: look up each id, concatenate the pieces, and the original text comes back exactly.

What to keep: BPE is a compression loop that stops early. It counts pairs over a sample of text, merges the winner, writes the rule down, and leaves two small files behind - the tokenizer - before the model exists. The model is then trained around that list and can never see past it. Every oddity downstream - the cost of an emoji, the arithmetic slips, the bill for a language the corpus barely contained - traces back to a frequency table built before the first gradient step. For what happens to the ids once they reach the model, read Inside an LLM; for the run that fills the table those ids address, read How a model learns.