LLM Tokens and Context Windows: Count, Budget and Avoid Truncation

A token is the chunk of text a language model reads and writes, and the context window is the maximum number of tokens it can handle in one request, counting your input and the model's output together. To stay inside the limit, count tokens before you send a request: call client.messages.count_tokens for Claude, use the tiktoken library for OpenAI encodings, and check the usage data that every response returns.

Who this is for and what you will have

This guide is for developers who call LLM APIs and want predictable behaviour for long prompts, documents and conversations. By the end you will know what counts toward the window, how to count tokens for Claude and OpenAI-style encodings, how to budget max_tokens, and which errors to expect when a request is too large.

Prerequisites

• Python with the anthropic package and an ANTHROPIC_API_KEY environment variable for the Claude examples.
• Optionally the tiktoken package for OpenAI encodings.

pip install anthropic tiktoken

What a token is and why the count matters

Models do not read characters or words. They read tokens, which are pieces of text produced by a tokenizer. The tiktoken documentation describes its byte pair encoding as reversible and lossless and gives a rough rule that each token corresponds to about 4 bytes on average. Treat that as an order-of-magnitude guide only, because code, non-English text and unusual formatting tokenize differently.

Token counts matter for three reasons: they set what fits in the context window, they drive cost because APIs bill by tokens, and they feed rate limits. Different model families use different tokenizers, so the same text has different counts on different models. For cost calculations see Claude API Pricing: Estimate Token Costs Before You Build.

What counts toward the context window

The Claude documentation defines the context window as all the text a model can reference when generating a response, including the response itself. Everything in the request counts:

• The system prompt.
• Every message in messages, including tool results, images and documents.
• Your tool definitions.
• The output Claude generates for the turn, including extended thinking.

Across turns, each user message and assistant response accumulates, and earlier turns are preserved completely unless you manage them. With prompt caching, the input count is split across input_tokens, cache_read_input_tokens and cache_creation_input_tokens, and all three count toward the window. Caching changes what you pay for those tokens, not whether they count.

Window sizes

According to the Claude documentation, current models including Claude Opus 5.5, Sonnet 5.5, Haiku 5.5 and the Fable 5 family have a 1M-token context window, and a single request can generate up to 128k output tokens (max_tokens). Other Claude models, including Claude Sonnet 4.5, have a 200k-token window. Model lists change, so confirm the figure for your model in the model comparison table before you hard-code a limit.

A larger window is not automatically better. The documentation notes that as token count grows, accuracy and recall degrade, which it calls context rot, so curating what goes into the request matters as much as how much space remains.

Thinking tokens

With extended or adaptive thinking, thinking tokens count toward the context window and are a subset of max_tokens. They are billed as output tokens. Whether thinking blocks from earlier assistant turns stay in context depends on the model: some keep them by default and they count as input, and others strip them automatically when you pass them back. Check the thinking documentation for your model.

Count Claude tokens before you send

The token counting endpoint accepts the same structured inputs as creating a message, including system prompts, tools, images and PDFs, and returns the total number of input tokens.

import anthropic
 
client = anthropic.Anthropic()
 
response = client.messages.count_tokens(
    model="claude-opus-5-5",
    system="You are a scientist",
    messages=[{"role": "user", "content": "Hello, Claude"}],
)
 
print(response.json())

The documentation shows this producing { "input_tokens": 14 }. Pass the model you will actually use, because the endpoint counts with that model's tokenizer.

Count tools

Tool definitions consume tokens on every request, so include them when you count.

response = client.messages.count_tokens(
    model="claude-opus-5-5",
    tools=[
        {
            "name": "get_weather",
            "description": "Get the current weather in a given location",
            "input_schema": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    }
                },
                "required": ["location"],
            },
        }
    ],
    messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
)
 
print(response.json())

The documentation's sample output for this request is { "input_tokens": 403 }, which shows how a single small tool adds to a short prompt.

What the endpoint does not accept

The endpoint returns an invalid_request_error for some inputs the Messages API accepts:

• Server tools such as web search, web fetch, code execution and tool search, with the advisor tool as the exception.
• The MCP connector.
• image or document blocks with a url or file source.

Send images and PDFs as base64 to count them. For requests that use server tools or MCP servers, read the usage object in the Messages API response instead.

Limits, cost and accuracy

Token counting is free, but it has requests-per-minute limits by usage tier.

Counting and message creation have separate, independent limits, so one does not consume the other. Two accuracy points matter:

• The count is an estimate and may differ by a small amount from the real usage. It may include system-added tokens that you are not billed for.
• Counting does not use prompt caching. You can include cache_control blocks, but caching only happens when you create a message.

The tokenizer changed

The documentation states that Claude 4.7 and later models, and the Fable and Mythos models, use a newer tokenizer. The same text produces approximately 30 percent more tokens than on earlier models, with the exact increase depending on the content. Do not reuse counts measured on an older model to estimate cost or window fit. To measure the difference for your workload, count the same request against both models and compare the two input_tokens values.

Count OpenAI-style tokens with tiktoken

For OpenAI models, tiktoken is the open source tokenizer. Get an encoding by name or by model, then encode the text and take the length.

import tiktoken
 
enc = tiktoken.get_encoding("o200k_base")
# or: enc = tiktoken.encoding_for_model("gpt-4o")
 
tokens = enc.encode("hello world")
print(len(tokens))
assert enc.decode(tokens) == "hello world"

The len(enc.encode(text)) pattern is the standard way to count, though the repository README does not show a counting example and points to an OpenAI Cookbook notebook for that. Use tiktoken only for OpenAI encodings. It does not count Claude tokens, so use the Claude endpoint for Claude prompts. Because tiktoken counts text, confirm real consumption from the usage field of the API response rather than relying on text-only counts.

Budget input, output and tools together

A simple budget for each request is:

available_for_input = context_window - max_tokens

Your system prompt, tool definitions, conversation history and retrieved documents must fit in that remainder. A practical routine is:

1. Count the fixed parts once: system prompt and tool definitions.
2. Count the variable parts per request: history, retrieved text, user input.
3. Compare the total plus max_tokens with the window and trim or summarise if it does not fit.

CONTEXT_WINDOW = 1_000_000  # confirm for your model
MAX_OUTPUT = 4_096
 
def fits(client, model, system, messages, tools=None):
    kwargs = {"model": model, "system": system, "messages": messages}
    if tools:
        kwargs["tools"] = tools
    used = client.messages.count_tokens(**kwargs).input_tokens
    return used + MAX_OUTPUT <= CONTEXT_WINDOW, used

You rarely need to run this for a 1M window with short chat turns, but it protects document-heavy workflows, and it is essential on models with a 200k window.

Ways to reduce what you send

• Drop or shorten old turns that no longer matter.
• Retrieve only the relevant passages instead of whole documents.
• Defer rarely used tool definitions. The documentation points to the tool search tool and a guide on managing tool context.
• Use server-side compaction for long conversations. The documentation describes it as automatically summarising earlier parts of the conversation, available in beta for Claude 4.6 and later models.
• Use context editing to clear old tool results or thinking blocks in agentic workflows.

What happens when you exceed the limit

Always check stop_reason in your code. A response that stops because of the window or max_tokens can be cut off in the middle of a sentence or a JSON document, so treat it as incomplete. If you parse JSON from the output, see Claude Structured Outputs: Get Valid JSON from the API in Python for the related failure cases.

Troubleshooting

• "prompt is too long" error. Count tokens first, then trim history or retrieved text, or compact the conversation.
• Count differs slightly from billed usage. The count is an estimate. Compare with the usage object from real responses.
• Counts went up after changing models. The newer tokenizer on Claude 4.7 and later produces more tokens for the same text. Recount.
• invalid_request_error from count_tokens but not from messages.create. Your request has a server tool, the MCP connector, or a url or file source for an image or document.
• Replies cut off. max_tokens or the window was reached. Raise max_tokens if the window allows, or shorten the input.

Checklist

• Count with the model you will call, not a different one.
• Include system prompt, tools, history, documents and max_tokens in the budget.
• Treat counts as estimates and read usage from real responses.
• Check stop_reason before using output.
• Recount after any model change.
• Use tiktoken for OpenAI encodings only.

References

• Token counting, Claude API documentation: https://platform.claude.com/docs/en/build-with-claude/token-counting
• Context windows, Claude API documentation: https://platform.claude.com/docs/en/build-with-claude/context-windows
• tiktoken repository: https://github.com/openai/tiktoken