Tokenization

Tokenization is a foundational step in natural language processing (NLP) and generative AI. Before a large language model (LLM) can work with text, a tokenizer converts that text into discrete units called tokens. The number of tokens in a prompt and response then shapes how much work the model must do — and with it, the compute, memory and cost required to deliver an answer.

What is a token?

In AI tokenization, a token is a small unit of data that an AI model can process. For text, a token may be a full word, part of a word, a single character or a punctuation mark. For example, the word “tokenization” might be treated as one token or split into smaller pieces such as “token” and “ization,” depending on the tokenizer. Each token is assigned a numerical ID so the model has a consistent way to recognize and process that piece of language.

Tokens matter because AI models measure their inputs and outputs in tokens. Every model has a context window — the maximum number of tokens it can consider at once — so tokenization determines how much text fits within that limit. A short sentence may use only a few tokens, while a long document or detailed prompt may use thousands. The more tokens a model processes, the more work it must do to analyze the input and generate an output.

What is tokenization?

Tokenization is the process of converting text into tokens and assigning them numerical IDs that AI models can process mathematically. Some multimodal models apply analogous approaches to images, audio and video, converting them into token-like units, although the methods differ by modality and model architecture. Rather than reading human language directly, AI models use tokenization to turn raw input into structured sequences they can analyze, compare and generate from. This makes tokenization the bridge between human language and the numerical inputs on which a model computes.

Tokenization is the first step in most NLP tasks, from text classification and machine translation to sentiment analysis and question answering. It enables machine learning models to transform unstructured human language into structured, machine-readable representations. The choice of tokenization method influences how efficiently a model represents text across languages and domains. A tokenizer that needs more tokens to express the same content produces longer sequences, which increases compute, memory use and cost.

How does tokenization work?

A tokenizer, a preprocessing component whose vocabulary is typically learned from a large body of text before the model itself is trained, splits input text into subunits, maps each to a numerical ID and passes that sequence of IDs to the model for processing. On the output side, the tokenizer works in reverse: Token IDs generated by the model are decoded back into human-readable language. Everything in between — interpreting the input and generating a response — is done by the model, not the tokenizer.

For example, tokenization works a little like turning a sentence into a barcode or machine-readable code. A person can read “the cow jumped over the moon” and understand the words directly, but an AI model needs that sentence prepared step by step. First, during text normalization, the tokenizer may standardize the text format, such as character encoding or spacing. Many modern LLM tokenizers keep capitalization and punctuation intact because they carry meaning. Next, during token extraction, it splits the sentence into smaller pieces such as “the,” “cow,” “jumped,” “over,” “the” and “moon.” Then, during vocabulary mapping, each token is matched to a numerical ID from the model’s vocabulary. Those IDs do not carry meaning by themselves, but they give the model a consistent way to recognize and process each piece of language.

Once tokenization is complete, the model takes over. It first converts each token ID into an embedding — a list of numbers learned during training — and then refines those representations through its layers to capture relationships between words and their context. For example, the phrase “the cow jumped over the moon” comes from the nursery rhyme “Hey Diddle Diddle,” so a model that has seen that phrase in similar contexts may connect it to children’s literature, rhyme or imaginative language. If the model has also seen “cow” appear near words like “farm,” “animal,” “milk” and “pasture,” it learns that those ideas are related too.

Within those layers, a mechanism called attention lets the model weigh how strongly each token relates to the others in context. For example, “the cow jumped over the moon” and “the cow did not jump over the moon” contain many of the same tokens, but the token “not” changes the meaning of the sentence. Attention helps the model account for that difference when it builds its response.

Finally, the model generates its response one token at a time, predicting each new token ID based on everything that came before. During detokenization, the tokenizer converts those output token IDs back into readable text. If someone asks, “Where is that phrase from?” the model may identify the connection to “Hey Diddle Diddle” and create an answer such as, “The phrase comes from the nursery rhyme ‘Hey Diddle Diddle.’” The model looks at the words in the question and creates an answer that fits that context. In short, tokenization breaks language into pieces the model can process, the model does the work of interpreting those pieces and generating new tokens and detokenization turns the output back into language people can read.

The key steps in the tokenization pipeline include:

• Text normalization: Optional, tokenizer-specific cleanup, such as standardizing character encoding, white space or accents. Many LLM tokenizers preserve case and punctuation.
• Token extraction: Applying rule-based or algorithmic patterns to split text into words, subwords or characters.
• Vocabulary mapping: Assigning each token a unique numerical ID from a predefined vocabulary.
• Detokenization: Converting token IDs generated by the model back into readable text.

Tokenization itself is computationally lightweight. Most of the work in an AI system happens afterward, as the model processes input tokens and generates output tokens — work that depends on fast, high-capacity memory solutions close to the AI processor.

Why does token count matter for AI performance?

Every token a model reads or writes requires computation and data movement. AI inference typically happens in two phases. In the prefill phase, the model processes all of the input tokens in a prompt, which largely determines how quickly a response begins. In the decode phase, the model generates output one token at a time, which determines how quickly the rest of the response streams back to the user.

Each step of the decode phase requires the processor to read the model’s weights from memory, so memory bandwidth often limits how fast tokens can be generated. To avoid repeating earlier work, the model also stores intermediate results for every token in its context in a key-value (KV) cache, which grows with context length and the number of users served at once. As prompts get longer, AI agents chain more steps together and more users share the same infrastructure, memory capacity and bandwidth become central to AI performance. Micron’s memory and storage technologies help AI systems keep processors supplied with data, sustain high token throughput and support longer context across training and inference.

Common token metrics include:

• Context window: The maximum number of tokens a model can consider at once, including the prompt and the response.
• Time to first token (TTFT): How long a user waits before the first part of a response appears.
• Tokens per second: How quickly a model generates output, measured per user or across an entire system.
• KV cache: Memory that holds intermediate results for tokens already in context so the model doesn’t have to recompute them.

What is the history of tokenization?

Tokenization has evolved from simple rule-based text splitting into a sophisticated machine learning discipline central to modern AI. Early NLP systems used basic word-boundary rules to segment text. As AI models grew more capable, more advanced methods became essential.

• 1980s-1990s, rule-based NLP: Early tokenizers relied on white space and punctuation rules, which was effective for English but limited for multilingual text.
• 2000s, statistical methods: Probabilistic approaches enabled more nuanced segmentation, particularly for Asian languages without clear word boundaries.
• 2016, byte pair encoding (BPE): Originally a data compression algorithm published in 1994, BPE was adapted for NLP — enabling subword tokenization that balanced vocabulary size and linguistic coverage.
• 2018, transformer-era tokenizers: Bidirectional encoder representations from transformers (BERT) and generative pre-trained transformer (GPT) models popularized WordPiece and BPE-based tokenizers, establishing the industry standard for LLMs.
• 2020s-present, multimodal tokenization: Token-based approaches expanded beyond text. Vision transformers, for example, divide images into small patches that the model treats much like tokens, helping enable today’s multimodal AI systems.

What are the key types of tokenization?

Tokenization encompasses several distinct methods, each optimized for different languages, tasks and model architectures. Modern AI systems, especially LLMs, predominantly rely on subword tokenization for its balance of vocabulary efficiency and linguistic coverage.

1. Word tokenization

Word tokenization splits text into individual words based on spaces or punctuation boundaries. It delivers strong performance for languages with clear word boundaries, such as English, but struggles with compound words, languages without spaces (e.g., Chinese, Japanese) and out-of-vocabulary terms.

2. Character tokenization

Character tokenization segments text into individual characters, operating with very small vocabularies while producing longer sequences. This approach handles rare or invented words effectively and suits languages with complex scripts, though it increases sequence length and computational overhead.

Example: The word “cow” could be split into character tokens such as “c,” “o” and “w.”

3. Subword tokenization

Subword tokenization — the standard approach for modern LLMs — breaks words into frequently occurring fragments, which often, but not always, line up with prefixes, roots and suffixes. It balances vocabulary size with linguistic coverage, enabling models to handle rare words, technical terminology and new language by decomposing them into known subunits.

Example: The word “tokenization” could be split into subword tokens such as “token” and “ization,” allowing the model to process an unfamiliar or complex word using smaller known pieces. Common approaches include:

• BPE: Starts from individual characters or bytes and repeatedly merges the most frequent adjacent pairs into new tokens. Byte-level variants are widely used in GPT-family and many other LLMs.
• WordPiece: Similar to BPE, but chooses the merges that maximize the likelihood of the training data. It is used in BERT and related encoder models.
• SentencePiece: A tokenizer toolkit rather than a single algorithm. It treats text as a raw character stream, including spaces, so it doesn’t depend on language-specific word splitting. It can implement BPE or unigram tokenization and is widely used in multilingual models.

4. Sentence segmentation

Sentence segmentation, sometimes called sentence tokenization, divides paragraphs or documents into individual sentences. Unlike the methods above, it doesn’t create the tokens that an LLM processes; it’s a preprocessing step used in traditional NLP pipelines and document workflows. It supports tasks requiring document-level analysis, such as summarization, translation and long-context reasoning.

How is tokenization used?

Tokenization delivers value across a broad range of AI-powered applications, serving as a foundational preprocessing step for systems that interpret or generate human language. Organizations deploying AI at scale, from AI data centers to edge devices, must manage token volumes to sustain throughput and control inference costs.

• Search engines: Tokenization enables systems to index and match user queries with precision, improving search relevance and retrieval speed.
• Chatbots and virtual assistants: Every user input is tokenized before processing; output tokens are decoded into natural language responses in real time.
• Machine translation: Tokenization converts source sentences into processable units, enabling accurate cross-language reconstruction.
• Speech recognition: Voice inputs are transcribed to text, then tokenized for semantic interpretation by AI systems.
• Healthcare AI: Tokenization prepares clinical notes, pathology reports and medical literature for AI models that support diagnostics and research workflows.
• Financial document analysis: Tokenization prepares contracts, earnings reports and regulatory filings for AI models that extract structured insights.
• Legal document processing: AI tokenizers segment case filings, contracts and legislation for automated review and summarization.
• AI training pipelines: Models learn by predicting the next token in a sequence; efficient data storage and retrieval of training token datasets is a key performance factor in large-scale AI infrastructure.