Tokenization
[/ˌtoʊkənɪˈzeɪʃən/]
Definitions
مسودة ترجمة بمساعدة آلية (Arabic) for "Tokenization": The process of converting raw text into discrete units called tokens that a language model can process. Tokens are typically subword units — common words become single tokens while rare words split into multiple tokens. All LLM pricing and context limits are measured in tokens, not characters or words.
“مسودة مثال: The word "unbelievable" tokenized into three pieces: "un", "believ", "able".”
Related Terms
- Context WindowAI & Technology
مسودة ترجمة بمساعدة آلية (Arabic) for "Context Window": The maximum amount of text (measured in tokens) that a language model can process and "remember" in a single interaction. In...
- AgenticAI & Technology
مسودة ترجمة بمساعدة آلية (Arabic) for "Agentic": Describing AI systems capable of autonomous action, planning, and decision-making. An agentic AI can break down tasks, use tools, a...
- AI AlignmentAI & Technology
مسودة ترجمة بمساعدة آلية (Arabic) for "AI Alignment": The research field focused on ensuring that AI systems pursue goals that match human values and intentions. A misaligned AI mi...
- Chain of ThoughtAI & Technology
مسودة ترجمة بمساعدة آلية (Arabic) for "Chain of Thought": A prompting technique where a language model is encouraged or required to show its step-by-step reasoning before providing...
- Constitutional AIAI & Technology
مسودة ترجمة بمساعدة آلية (Arabic) for "Constitutional AI": A training methodology developed by Anthropic where a set of guiding principles (a "constitution") is used to self-superv...
- Diffusion ModelAI & Technology
مسودة ترجمة بمساعدة آلية (Arabic) for "Diffusion Model": A class of generative AI model that learns to create images, audio, or video by reversing a noise-adding process. During tr...