What is AI inference?
The moment a model answers.

Training changes the model. Inference uses what it learned.

During training, an optimization process adjusts a model’s parameters using data or feedback. During inference, the trained parameters are used to calculate an output for a particular request. The model generally does not rewrite its learned weights each time someone asks a question.

An application may still save a conversation, use a memory feature, record feedback, or update its own data. Those are separate product processes. “The model is inferring” does not tell you what the surrounding software stores or sends elsewhere.

For the broader lifecycle, start with how language models are trained and used ↗ and what machine learning means ↗.

A text model builds an answer from the context it receives.

1. Prepare the input. Text is converted into tokens the model can process. A chat system may also add role markers, instructions, and prior turns.
2. Process the context. The model calculates representations for the prompt. In common serving designs, this initial work is called prefill.
3. Choose a next token. The model computes scores for possible next tokens. A decoding rule—such as choosing the top-scoring token or sampling from a distribution—selects what comes next.
4. Continue until a stop condition. The selected token becomes part of the context for the next step. Generation repeats until a stop token, configured limit, or other stopping rule is reached.
5. Return the result. Tokens are decoded into readable text. The application can stream them as they arrive or wait and return the completed response.

Generation settings control this process. For example, the Transformers library documents token limits, sampling, stopping conditions, and caches as separate generation controls. Their names and defaults can vary across libraries and model providers.

This describes a common autoregressive text-generation path, not every AI system. Image, audio, video, and multimodal models can use different input encoders and output procedures.

The pause before the first word and the pace after it have different causes.

For many language-model services, the prompt is processed first; then the model generates output tokens one step at a time. Serving research often tracks time to first token (TTFT) for the initial phase and time per output token (TPOT) for the generation phase. These answer different questions: how long until a response starts, and how quickly it continues.

The model can reuse cached attention keys and values from earlier tokens instead of recomputing the entire history each time. This key-value (KV) cache helps with repeated decoding, but it occupies memory and grows with the context and generated sequence. Long prompts and long responses can therefore change the compute and memory required by a request. Read the KV cache guide for a memory estimate and serving strategies ↗; for a compact model’s complete device footprint, see how to evaluate memory for small language models ↗.

Some serving systems also use speculative decoding to propose several next tokens and verify them together. See how draft-and-verify generation works and what to measure ↗.

The DistServe study ↗ analyzes prefill and decoding as phases with different latency requirements. The PagedAttention paper ↗ examines how KV-cache memory management affects batching and throughput. These papers describe serving experiments, not a guarantee about any particular provider’s infrastructure.

The model call is one event in a larger workflow.

A product may retrieve documents before inference, filter or transform the input, call the model more than once, invoke a tool, or ask a person to review an action. A short answer can follow a long chain of work; a fast token stream can still be unhelpful if the application retrieved the wrong source or gave the model too much access.

Inference also does not mean the model has searched the web, remembered a previous session, or verified a fact. Those functions require additional data, tools, or product behavior. Ask what the runtime actually receives and what happens after it produces an output.

Some systems also spend additional inference-time computation on difficult requests. See how reasoning models use that extra compute ↗.

Related guides: how an agent moves from a model to an action ↗, where local inference does—and does not—protect data ↗, and how to measure the whole system’s cost ↗.

“Tokens per second” is not a complete answer.

Two systems are comparable only when the task and test conditions are clear. A benchmark that reports a fast output rate may use a different prompt length, response length, hardware, concurrency, or generation configuration from the workload you care about.

1. Use representative requests. Include the prompt sizes, output lengths, and input types your application actually sends.
2. Measure both phases. Track time to first token and generation pace, not only total runtime.
3. Report load and hardware. Note concurrent requests, batching, model version, and the hardware used for the comparison.
4. Check useful quality. Evaluate correctness and task completion alongside speed, cost, and memory.
5. Include the surrounding workflow. Retrieval, retries, tool calls, and human review may dominate the time or cost of a completed task.

For model selection, see how to choose for a real task ↗. For operational comparison, read the full cost of intelligence ↗.

Follow the computation to its consequences.

Inference turns model parameters into an output for a specific context. The quality and consequences of that output still depend on evidence, instructions, permissions, and evaluation. Continue with why fluent answers can fail ↗ or explore the AI field guide ↗.