Tokenization
[/ˌtoʊkənɪˈzeɪʃən/]
Definitions
The process of converting raw text into discrete units called tokens that a language model can process. Tokens are typically subword units — common words become single tokens while rare words split into multiple tokens. All LLM pricing and context limits are measured in tokens, not characters or words.
“The word "unbelievable" tokenized into three pieces: "un", "believ", "able".”
Related Terms
- Context WindowAI & Technology
The maximum amount of text (measured in tokens) that a language model can process and "remember" in a single interaction. Information outside the context window is inaccessible to ...
- AgenticAI & Technology
Describing AI systems capable of autonomous action, planning, and decision-making. An agentic AI can break down tasks, use tools, and work toward goals with minimal human intervent...
- AI AlignmentAI & Technology
The research field focused on ensuring that AI systems pursue goals that match human values and intentions. A misaligned AI might optimize for a metric that appears correct but pro...
- Chain of ThoughtAI & Technology
A prompting technique where a language model is encouraged or required to show its step-by-step reasoning before providing a final answer. Chain-of-thought prompting significantly ...
- Constitutional AIAI & Technology
A training methodology developed by Anthropic where a set of guiding principles (a "constitution") is used to self-supervise and refine AI outputs. The model critiques and rewrites...
- Diffusion ModelAI & Technology
A class of generative AI model that learns to create images, audio, or video by reversing a noise-adding process. During training the model learns to denoise progressively; during ...