Tokenization

Tokenization

2026-07-23

We know, computers work with numbers and numbers only. So, whenever we represent some other formats like Emojis or Strings, it is naturally represented using numbers inside the computer system.

The same goes for LLMs. They can't just work on raw text. They are first converted to numbers before they can be processed. The tool that does this exactly is known as a tokenizer. It is quite similar in concept but different than tokenizer used by compilers.

Tokenizer in the context of LLMs take a raw input string, convert it into vectors (set of numbers) and train using those vectors. Later, during text generation, the output in given vector is converted to text string that we humans can understand.

That said, there are different ways we can convert text to tokens. Each has their own pros and cons and with that their own use cases:

Types of Tokenizers

1. Character level tokenizer

This is the simplest and most naive form of tokenization. Here, we simply take each character in any string and convert it into a number.

Notice, that with this approach, we won't have any issues with any unknown words as well words are represented by characters. This approach might be suitable for small amount of training data. However, this approach quickly poses a problem. Imagine we are trying to tokenize all the contents in a book. Tokenizing each and every character will grow out of hand quickly. Now, imagine we are trying to tokenize PetaBytes(PB) of worth of data from the internet. It requires a lot of compute and context window and makes long range dependencies difficult. We are essentially trading off smaller vocabulary for much longer sequences.

2. Word level Tokenizers

We can try to be clever here and we can tokens per word instead of characters. This is exactly what word level tokenizers do. This is quite straightforward because the string: "Hello World" becomes ["Hello", "World"]

This solves the problem of large sequences because a sentence will be composed of relatively shorter number of tokens in word level as opposed to character level. However, since we use a vocabulary for tokenization, we may run into words that are not in the vocabulary. For example, the vocabulary might have the word for micro and processor but not for microprocessor. Also, similar words that imply similar meaning will each have different tokens (play, playing, played).

3. Subword level

This approach tries to fix the problems with word level tokenizers. As the name suggests, it uses subwords instead of raw words directly. By subwords, what I mean is, it breaks down commonly occuring patterns and phrases in a word (i.e. playing becomes play and ing).

This approaches avoids most unknown words problem because an unknown might be broken down into smaller known words (i.e. microprocessor becomes micro and processor). It does this while keeping relatively short sequences (much much shorter than character level tokenizer). It also captures useful word structure. For examples:

  • play → ["play"]
  • playing → ["play" + "ing"]
  • played → ["play" + "ed"]

Not all languages follow this pattern, so it can become language dependent. However, it is still the most popular approach for tokenization till date.

Subword level algorithms

1. BPE

BPE is one of the subword level tokenization scheme in which most frequently adjacent pairs are combined together. It is very common, mostly in Decoder only transformers like GPT. Here, the algorithm starts with individual characters. As the model sees more and more vocabularily, it identifies different subwords that comes together regularly. It is purely frequency driven greedy approach.

For example: If the training data frequently contains words like:

play, player, playing, replay

The tokenizer may learn following tokens:

  • play
  • ing
  • er
  • re

Later, if it encounters a new unseen word like replaying, following happens:

replaying = re + play + ing

2. WordPiece

It is very similar to BPE, but has one major difference: At each step, WordPiece tries to pick a pair which will maximize the likelihood of words appearing in the training data when merged. This is unlike BPE which is a greedy approach and always tends to pick wjkjjords that appears most frequently together in the training data. It is very common in BERT models.

For example, the algorithm will check if the probability of "es" coming together higher than probability of "e" or "s" in a word.

3. Unigram

Unigram on the otherhand, starts with a bunch of subwords. Then, it iteratively and gradually remove the least useful ones keeping only important subwords. It starts with a very large pool of subwords, assign a probability to each token and remove tokens that contribute the least. During tokenization, it selects the most probable combination of remaining pieces.

For example: Suppose the word is: unhappiness

During training, it may assign probabilities as:

Subword Probability
un 0.20
happy 0.30
happi 0.10
ness 0.25
s 0.05
unhappy 0.10

Possible segments include:

  • un + happi + ness
  • unhappy + ness
  • un + happy + ness
4. SentencePiece

Unlike other algorithms, it is more accurately a framework rather than an algorithm. It processes raw text directly (including spaces as well). This makes it adaptable and suitable for different domains. It may be implemented by either BPE or Unigram internally.

For example: Suppose we take the sentence: "I like playing football".

Internally, SentencePiece may split it as:

["▁I", "▁like", "▁play", "ing", "▁foot", "ball"]

Special Tokens

Just words and subwords alone are not enough for all our tokenization needs. Sometimes, we may need to represent additional information during tokenization process. This is done by using special tokens. This can be done by using tokens like:

  1. [CLS]

    • Classification Token
    • Usually added at the beginning of the sequence:
    • [CLS] The Odyssey was excellent [SEP]
    • In model like BERT, for sequence level tasks: (Classify the entire sentence), we want a single vector that summarizes the entire sequence in one vector.
    • The classifier head in the model uses that vector to predict a label (i.e. "positive" or "negative")

  2. [SEP]

    • Seperator token
    • As the name suggests, it seperates sequences or sections.
    • [CLS] What is the capital of France? [SEP] Paris is the capital of France. [SEP]
    • This helps a model understand when two different parts like question-answer, Sentence A-Sentence B or two different documents seperate.

  3. [PAD]

    • Padding Token
    • Models typically operate in batches.
    • Since all sequences may not have equal length, shorter sequences are padded to make the length equal.
    • INPUT 1: [CLS] Hello World [SEP]
    • INPUT 2: [CLS] Hi [SEP] [PAD]

  4. [UNK]

    • Unknown Token
    • This is used when models are unable to identify the token as a word or combination of subwords in the vocabulary.
    • "dsjkfh" -> [UNK]
    • Although, using character level or subword level tokenizer which breaks down unseen word to character level reduces this token as all the characters "d", "s", … "h" are known individually.

Vocab Size

  • The vocabulary size of the model is set and controlled using a Hyperparameter.
  • Hyperparameter means that the parameter is not learned but have to set manually by a person.
  • Too small → Loss of Detail
  • Too big → Memory Hungry

After Tokenization

  • After the tokenization process completes, we get Integer IDs from strings:
  • ["the"] → [ID: 234]
  • This alone doesn't give us any information about which words are closer to what. (i.e cat and dogs are similar as they are both household pets)
  • This task is done by Embedding layer, which during pre-training adjusts the embedding vectors such that tokens of "cats" and "dogs" are closer in a very high dimensional space.
  • Words like "the", "a" are extremely common and doesn't mean much in a sequence. So, they tend to act as glue and cluster near the origin (usually).

Tokenizers used by Popular Models

Model Tokenization Algorithm
BERT WordPiece
GPT-2 BPE
T5 SentencePiece
Llama 1/2 SentencePiece
Llama 3 BPE
Gemma Family SentencePiece