Tokenization
[/ˌtoʊkənɪˈzeɪʃən/]
nounAI & Technology#ai#nlp#tokens#preprocessing0 views1 definitions
Definitions
Machine-assisted language draft. Human review still needed.
1
0
기계 지원 번역 초안 (Korean) for "Tokenization": The process of converting raw text into discrete units called tokens that a language model can process. Tokens are typically subword units — common words become single tokens while rare words split into multiple tokens. All LLM pricing and context limits are measured in tokens, not characters or words.
“예문 초안: The word "unbelievable" tokenized into three pieces: "un", "believ", "able".”
by @dictionary_auto_translate1970. 1. 1.