What are AI tokens?
Text, in the model’s units.

A token is a unit in a model’s vocabulary, not a fixed-size word.

A tokenizer converts text into a sequence of token IDs that the model can process. Depending on the vocabulary and method, a token may represent a common word, part of a word, punctuation, whitespace, or a byte sequence. A word can be one token in one model and several in another.

Many tokenizers use subword methods such as byte-pair encoding or unigram models. Their vocabularies let them represent frequent patterns efficiently while still handling less familiar text. They do not all split the same sentence in the same places.

After tokenization, IDs are mapped into numerical representations that a neural network can process. Those vectors are related to, but distinct from, the visible words and from the sentence embeddings used for semantic search ↗.

Characters, words, and tokens are different measures.

A token count can affect how much input fits and, for some services, how usage is measured. Pricing and request accounting are provider-specific; token count alone does not describe the total cost of a useful task.

The window is the material available to one model call.

A context window is the finite amount of tokenized material a model can use while producing a response to a request. The application may send more than the latest message: system instructions, conversation history, retrieved passages, structured data, and tool results can all use space. The exact accounting rules and available response budget depend on the model and interface.

When a task grows beyond its budget, a product may trim old messages, summarize, retrieve only selected material, reject the request, or use a model with a different limit. Those are application choices; the model does not automatically preserve the complete conversation somewhere outside the request.

That distinction matters: a context window is not persistent memory. Memory requires the surrounding product to store information and decide when to provide it again. See how AI memory is designed ↗.

One provider-specific example: OpenAI’s Realtime API documentation ↗ describes truncating older conversation content when input grows beyond its limit. Other interfaces may handle overflow differently.

A larger window does not guarantee that every detail will be used.

In a 2023 study, researchers tested several language models on tasks that placed relevant evidence at different positions in long inputs. Performance often dropped when the key passage appeared in the middle. The finding describes the models and tasks evaluated in that paper, not every model available today; it is a useful reason to test information placement in your own workflow.

Later, the RULER benchmark tested long-context models on a broader set of tasks, including multiple-needle retrieval, aggregation, and multi-hop tracing. The paper found that a model’s success on a simple needle-in-a-haystack test did not reliably predict performance on those more varied tasks. Treat the result as evidence about the tested benchmark settings, then measure the work your own application must do.

More material can help when it adds relevant evidence. It can also add stale instructions, distractors, and retrieval noise. Design context around the next decision: retain source names and dates, label untrusted text, and include only material that the model needs to see.

For a hands-on evaluation of effective context length and recent long-context findings, read how to test whether AI can use its full context window ↗.

For a practical approach to selecting, ordering, and testing that material, read context engineering for agents ↗.

Measure the actual request before you send it.

1. Choose the target model first. Tokenization and context rules are model-specific.
2. Count with its tokenizer or API. Include message structure, tool definitions, and other content the service accounts for.
3. Reserve output space. The answer also consumes a budget; do not fill the entire allowance with input.
4. Keep context useful. Trim repeated history, retrieve relevant passages, and preserve the source and date of important evidence.
5. Test realistic cases. Include code, multilingual text, long conversations, missing evidence, updates to the source, and relevant passages placed at different positions.

For implementation details, inspect OpenAI’s tiktoken code ↗. For a broader look at language-model generation, continue with how large language models work ↗.