The Tokenizer: What Runs Before the Model Does
When you send text to an AI model, it does not arrive as words. Before any processing happens, the text passes through a component called a tokenizer, which converts it into a sequence of numbers the model can work with. Each number corresponds to a token.
A token is a subword unit, not a full word and not a single character. The algorithm most modern models use to generate tokens is called Byte Pair Encoding, or BPE. The short version of how it works: BPE takes a large body of training text, scans it for the character pairs that appear most often side by side, and merges each pair into a single token. It keeps repeating that process until a vocabulary of roughly 32,000 to 100,000 tokens is fully assembled. Common short words map to a single token. Longer or rarer words get split into two or three pieces. Symbols, punctuation, and non-alphabetic characters often become tokens of their own.
The practical consequence is that one word does not equal one token. OpenAI’s own documentation gives the most widely used rule of thumb: one token is roughly four characters or three-quarters of a word in English. By that measure, 100 tokens equals about 75 words, and 1,500 words works out to approximately 2,048 tokens. These hold reasonably well for standard English prose. They shift for other content types.
Different content types tokenize differently. Code, technical terminology, and text in languages other than English typically produce more tokens per word than clean English prose does. That variation affects cost directly, and it is not visible in a flat word count.
One term worth defining before the table below: RAG, short for retrieval-augmented generation, is a pattern where an AI application fetches relevant documents or data from an external source and includes that content alongside the user’s question when calling the model. It is one of the most common designs in production AI applications, and retrieved documents can make up a large share of the input token count on every request.
The table below maps common content types to their approximate token counts and notes where the standard rule of thumb breaks down :
| Content Type | Approx. Words | Approx. Tokens | What to Know |
|---|---|---|---|
| Short user message | 10–30 words | 13–40 tokens | Usually the smallest piece of an API call |
| System prompt (instructions) | 200–500 words | 267–667 tokens | Sent on every request; often the largest fixed-cost component |
| Conversation history (5 turns) | 500–1,500 words | 667–2,000 tokens | Grows with each exchange; not trimmed automatically by default |
| RAG context chunk | 500–2,000 words | 667–2,667 tokens | Retrieved documents added before the user’s question |
| 50-line Python function | Not word-based | 300–600 tokens | Code tokenizes differently from prose; operators and symbols each count |
| Standard business email | 150–250 words | 200–333 tokens | US Declaration of Independence = 1,695 tokens, for reference |
| Non-English paragraph (e.g., Japanese) | 100 words equiv. | 300–700 tokens | Non-Latin scripts fragment more heavily; cost per sentence is higher |
Token estimates use the 0.75 words-per-token conversion from OpenAI’s published documentation. Actual counts vary by model tokenizer. OpenAI’s Tokenizer tool and Anthropic’s count-tokens API endpoint provide exact counts per model.
One pattern stands out immediately: in most production applications, the user’s message is the smallest contributor to the input token count. System prompts, conversation history, and retrieved context usually account for the large majority of what gets billed, before the user types anything.
Input Tokens and Output Tokens : Two Different Products
Every API call generates two token counts. Input tokens are everything the model reads before generating a response. Output tokens are everything the model writes back. Providers bill them separately, and at different rates.
On the input side, a typical production API call includes several distinct components. The table below breaks them down :
| Component | What It Contains | Typical Token Range | Grows Over Time? |
|---|---|---|---|
| System prompt | Instructions, persona, and behavioral rules set by the developer | 200–700 tokens | No (fixed per deployment) |
| User message | What the user actually typed or submitted | 10–150 tokens | Varies by request |
| Conversation history | All prior turns in a multi-turn session | 0–5,000+ tokens | Yes, with every exchange |
| RAG context | Documents or knowledge base chunks retrieved for this request | 500–10,000 tokens | Varies by query complexity |
| Tool definitions | Function names and schemas for agent or function-calling use | 100–500 tokens | No (fixed per deployment) |
Token ranges are approximate and workload-dependent. In agentic applications, multiple of these components can stack on a single request, and conversation history and RAG context can grow substantially across long sessions.
Output tokens are the model’s response: the text, code, or structured data it generates. They are also the more expensive side of the bill. Across every current-generation model from Anthropic and OpenAI, output tokens cost five times more than input tokens at standard list rates. That ratio reflects what is actually happening computationally: reading input can be processed in parallel, while generating each output token requires a full forward pass through the model.
A request with 3,000 input tokens and a 600-token response is not a 3,600-token bill at a single rate. It is 3,000 tokens billed at the input price plus 600 tokens billed at five times that rate. The output side of a generation-heavy workload is often where most of the cost is.
How to Count Tokens Before You Send a Request
Token counts are not something you have to estimate after the fact. Two approaches give you accurate numbers before a request is sent, and a third tells you exactly what happened after.
- OpenAI’s tiktoken library and Tokenizer tool. tiktoken is OpenAI’s open-source Python library for counting tokens across GPT model families. The interactive Tokenizer tool on OpenAI’s platform visualizes exactly how a string is split, token by token, using the encoding for a chosen model. Both are the standard approach for pre-flight estimation on OpenAI models.
- Anthropic’s count-tokens endpoint. Anthropic exposes a count-tokens API endpoint that returns the exact token count for a given message and model combination without running a full inference. This is the most reliable way to budget input costs before sending to a Claude model.
- API response metadata. Every API response from Anthropic and OpenAI includes usage data in the response body: input token count, output token count, and, where applicable, cached token counts. Logging this metadata on every request is the foundation of any accurate token cost model in production.
The key detail on estimation: token counts are model-specific. The same text can produce different token counts on GPT-5.4 versus Claude Sonnet 4.6 versus Llama 4 because each model uses a different tokenizer vocabulary. Estimations built for one model may be off by 10-30% on another. If you are running a multi-model architecture, count separately for each model family.
How CloudInvent Identifies This
CloudInvent’s 200+ AI-driven algorithms evaluate AI spend from billing and usage metadata, with no access required to prompts, completions, or model outputs. On the token side, that means analyzing input-to-output ratios by workload, flagging conversation history growth patterns that are driving unnecessary input token accumulation, and surfacing cases where retrieved context is being over-provisioned for the complexity of the requests being made.
For organizations new to token-level analysis, the first pass often surfaces input inefficiencies that were never visible in aggregate billing data. System prompts that are duplicated rather than cached, history windows that grow unbounded without trimming logic, and RAG pipelines returning far more context than any given query needs are among the most common findings. Our work with customers across AWS, Azure, and GCP consistently produces 25% to 40% reductions in cloud IT spend, achievable in the first 30 days, and token-level visibility is an increasingly important part of what drives those outcomes.
Practical Considerations
- Separate words from tokens in your planning. A document described as 10,000 words is approximately 13,300 tokens. If that document is sent as part of a RAG pipeline on every request, that is 13,300 input tokens on every call, not 10,000.
- Count system prompts as a fixed cost per request. Whatever a system prompt contains gets billed every time it is sent. A 500-token system prompt across one million requests costs the same as 500 million input tokens, which adds up quickly depending on the model tier.
- Build trimming logic for conversation history. Conversation history grows with every turn and is billed in full on each request unless it is actively managed. Without trimming or summarization, a long session’s history can exceed the combined cost of every other input component.
- Always use the model-specific tokenizer for estimates. Generic word-to-token conversion ratios are useful for quick ballparks. For any forecast that will be used for budgeting or capacity planning, use the tokenizer for the specific model in question.
The Takeaway
A token is the smallest unit of AI cost, and it is not the same as a word. The gap between those two things is where most early-stage AI cost estimates go wrong. System prompts, conversation history, and retrieved context are typically the largest contributors to input token counts, not the user’s message. And output tokens, which cost five times more than input on every current-generation model, are where generation-heavy workloads concentrate their spending.
Getting comfortable with token mechanics is the prerequisite for everything else in AI cost management. The second part of this series covers how providers price those tokens, why AI bills are so difficult to attribute, and what organizations need to put in place before they can manage this spending with real discipline.
The bill does not start when the user hits Send. It starts when the tokenizer runs, and most of what it counts was already there before the message arrived.
References
- Gartner, Inc. Gartner Forecasts Worldwide AI Spending to Grow 47% in 2026 (May 19, 2026).
- FinOps Foundation. State of FinOps 2026 Report.
- OpenAI Help Center. What Are Tokens and How to Count Them?
- OpenAI. Key Concepts, OpenAI API Documentation.
- Hugging Face. Byte-Pair Encoding Tokenization.
- Anthropic. Pricing Documentation, Claude Platform Docs.
- OpenAI. Tokenizer Tool.
- OpenAI. tiktoken (Open-Source Python Tokenizer Library).
- FinOps Foundation. FinOps X 2026 Day 1 Keynote: The Wild West of AI, Token Economics and the Evolving Role of FinOps.
- Stanford Institute for Human-Centered Artificial Intelligence. The 2025 AI Index Report.