How do Transformers work?
Relationships, computed at scale.

A Transformer turns a sequence into representations it can use.

The original Transformer paper introduced an encoder–decoder architecture for sequence tasks such as translation. It replaced recurrence with attention: the model processes relationships between sequence positions through stacked layers, alongside position-wise feed-forward networks.

Many later language models use Transformer variants, but the architecture family is broader than chatbots. An encoder can build representations from an input; a decoder can generate a sequence; an encoder–decoder model can combine both. The exact blocks and training objective vary.

To place the architecture among other neural approaches, see the deep-learning guide ↗. For the next-token generation task, continue with how language models work ↗.

Before attention, text becomes ordered vectors.

A model first splits input into tokens. A token may be a whole word, part of one, punctuation, or another piece defined by that model’s tokenizer; tokens are not always words or single characters. Each token ID is mapped to a learned numerical vector, and the sequence receives information about token order. The original Transformer added positional encodings to token embeddings; later architectures use different position methods.

The resulting vectors are working representations, not dictionary entries. As they pass through layers, a token’s representation can change with the rest of the sequence. In “the key opened the door” and “the key idea was simple,” the same spelling appears in different contexts; a language model can build different contextual representations for those occurrences.

This is one reason to keep the model tokenizer and context limit in view. The system processes its token sequence, not an abstract count of words. For practical consequences, see how tokenization changes context length ↗.

Each position can gather a weighted summary of other positions.

For each token representation, learned projections produce a query, a key, and a value. A query is compared with keys from positions it is allowed to use. The resulting weights determine how much information to combine from the corresponding values. The names are a useful shorthand: queries and keys determine the mixing weights, while values carry the information being mixed.

In self-attention, the queries, keys, and values come from the same sequence. This gives a token a way to combine information from other parts of the input. The mix changes from layer to layer as the representations change.

The word “attention” can sound psychological. Here it names a mathematical operation over learned vectors; an attention weight is not a human explanation of the model’s reasoning.

In a standard full-attention layer, each sequence position can compare against many other positions. That creates a pairwise computation whose size grows roughly with the square of sequence length. Optimized attention kernels and other architectures can reduce practical memory or runtime costs, but a longer context still changes the work. For the runtime memory that accumulates during generation, read the KV-cache guide ↗.

Several learned views are combined and refined.

Stacking layers gives a model repeated opportunities to transform and combine information. The result is learned numerical state—not a transparent set of rules that can be read directly as human concepts.

Generation uses a different information boundary from encoding.

An encoder may let each input position use information from the full input. An autoregressive decoder uses a causal mask so a position cannot see future output tokens while predicting the next one. The original encoder–decoder Transformer also uses attention from decoder positions to the encoded source sequence.

These boundaries reflect the task. A bidirectional encoder is useful for representations that can inspect the complete input; a causal decoder supports left-to-right generation. The word “Transformer” alone does not tell you which variant a product uses.

BERT is a concrete encoder example: its paper describes deeply bidirectional representations that condition on left and right context. GPT-3, by contrast, is described as an autoregressive language model. Both use Transformer architecture, but their attention boundaries and training setups differ. Read the BERT paper ↗ and the GPT-3 paper ↗.

For how tokens and context budgets shape the input, read the guide to tokens and context windows ↗.

A capable architecture still needs evidence and evaluation.

Attention provides a way for a network to combine information across positions. It does not guarantee that the model understood an instruction, used a source correctly, or produced a true answer. The learned behavior also depends on training data, objective, parameters, and the application that supplies context and tools.

Full self-attention compares positions with one another, which makes longer sequences more demanding to process. Implementations and later architectures use different optimizations, but an advertised context length alone does not establish quality on a real task.

Test the complete system on representative inputs, including important details in different positions, missing evidence, long histories, and the mistakes that matter to people using it. Learn how to assemble and evaluate context ↗.

Follow the representations into a working system.

Transformers describe how one family of models processes information. Tokenization defines the inputs, training shapes the parameters, and the surrounding product decides what the model can retrieve or do. Continue from architecture to how language models are trained ↗ and what happens when a trained model runs ↗, or browse the AI field guide ↗.