Tokenization
[/ˌtoʊkənɪˈzeɪʃən/]
Definitions
Borrador de traduccion automatica (Spanish) for "Tokenization": The process of converting raw text into discrete units called tokens that a language model can process. Tokens are typically subword units — common words become single tokens while rare words split into multiple tokens. All LLM pricing and context limits are measured in tokens, not characters or words.
“Ejemplo en borrador: The word "unbelievable" tokenized into three pieces: "un", "believ", "able".”
Related Terms
- Context WindowAI & Technology
Borrador de traduccion automatica (Spanish) for "Context Window": The maximum amount of text (measured in tokens) that a language model can process and "remember" in a single inter...
- AgenticAI & Technology
Borrador de traduccion automatica (Spanish) for "Agentic": Describing AI systems capable of autonomous action, planning, and decision-making. An agentic AI can break down tasks, us...
- AI AlignmentAI & Technology
Borrador de traduccion automatica (Spanish) for "AI Alignment": The research field focused on ensuring that AI systems pursue goals that match human values and intentions. A misali...
- Chain of ThoughtAI & Technology
Borrador de traduccion automatica (Spanish) for "Chain of Thought": A prompting technique where a language model is encouraged or required to show its step-by-step reasoning before...
- Constitutional AIAI & Technology
Borrador de traduccion automatica (Spanish) for "Constitutional AI": A training methodology developed by Anthropic where a set of guiding principles (a "constitution") is used to s...
- Diffusion ModelAI & Technology
Borrador de traduccion automatica (Spanish) for "Diffusion Model": A class of generative AI model that learns to create images, audio, or video by reversing a noise-adding process....