#tokens

मशीन-सहायता अनुवाद मसौदा (Hindi) for "Context Window": The maximum amount of text (measured in tokens) that a language model can process and "remember" in a single interaction. Information outside the context window is inaccessible to the model, making context management critical for long-form tasks.

“उदाहरण मसौदा: The model kept losing track of earlier instructions because the codebase exceeded its context window.”

मशीन-सहायता अनुवाद मसौदा (Hindi) for "Tokenization": The process of converting raw text into discrete units called tokens that a language model can process. Tokens are typically subword units — common words become single tokens while rare words split into multiple tokens. All LLM pricing and context limits are measured in tokens, not characters or words.

“उदाहरण मसौदा: The word "unbelievable" tokenized into three pieces: "un", "believ", "able".”