Token / Tokenization

Full definition

Tokenization is the process of converting raw text into the integer IDs the model actually consumes. Modern LLMs use byte-pair encoding (BPE) or SentencePiece. As a rule of thumb, 1,000 tokens ≈ 750 English words, but the ratio is far worse for code, JSON, and non-Latin scripts. Tokenizer choice meaningfully affects cost and quality across languages.

Why it matters

Token economics drive AI unit cost. A 30% reduction in tokens-per-task — through prompt compression, caching, summarization, or a better-fitted model — directly reduces ongoing OPEX. Finance and engineering should both be tracking dollars-per-task, not just dollars-per-million-tokens.

Example

"Retrieval-Augmented Generation is a technique" tokenizes to roughly 8 tokens in GPT-4's tokenizer. The same sentence in Japanese can take 30+ tokens, materially raising cost-per-query for non-English markets.

Related terms

• Large Language Model (LLM)
  A Large Language Model (LLM) is a deep neural network — almost always a transformer — trained on hundreds of billions to trillions of words to predict the next token, and to generate, summarize, translate, or reason over text.
• Context Window
  A context window is the maximum amount of text — measured in tokens — a language model can read and reason over in one inference call, equivalent to the model's working memory for that turn.
• Inference Cost
  Inference cost is the dollar cost of running a trained AI model in production — per request, per user, or per business outcome — and the operating expense that determines whether an AI feature has positive unit economics at scale.
• Prompt Engineering
  Prompt engineering is the practice of designing the inputs to a language model — instructions, examples, role definitions, output formats, and constraints — to reliably produce a desired output without changing the model's weights.