Red-team an AI system.
Find the edges before release.

Challenge the safeguards with realistic scenarios.

NIST describes AI red teaming as controlled exercises that look for adverse behavior or outcomes and stress-test safeguards. Tests can happen before or after release. A useful exercise starts with the system’s intended use and asks how a plausible failure could affect someone.

A jailbreak screenshot is one observation, not a complete security assessment. A red team follows a scenario through the whole workflow: what information enters, what the model returns, which tools or services are called, and whether the result can cause an impact.

For one threat that can appear in those scenarios, see how prompt injection works ↗. Red teaming covers a wider set of failure paths.

Map the product, its users, and its boundaries.

Write down what the system is for, who can use it, what information it handles, and what actions it can take. Mark where a foundation model ends and the application begins. That map keeps a test relevant to the product instead of an abstract contest between prompts.

Set written authorization, a test window, a safe environment, and clear stop conditions. If a system is operated by another organization, get permission before testing it.

Test the path from an input to a real consequence.

Use the system’s threat model to choose scenarios. MITRE ATLAS catalogs observed and demonstrated techniques against AI-enabled systems, including retrieval poisoning, model access, prompt attacks, and agent tool use. Use it to find relevant categories; it is a living knowledge base, not a checklist every product must implement.

For each scenario, define the expected behavior and the evidence that would show a failure. Include ordinary use, unexpected inputs, and plausible misuse; keep test cases tied to the product’s actual users and data.

Make the exercise safe for users and the system.

Use a staging environment where possible, with synthetic or appropriately protected data. Restrict credentials and tool permissions, record the test setup, and establish a way to stop or roll back actions. Do not use real customer secrets or send test content to people who have not agreed to participate.

Bring together people who understand the technology and the context of use. NIST’s guidance notes that red-team findings depend on the expertise of the team; domain knowledge and different perspectives can uncover risks a purely technical review misses.

Automated scanners can help repeat known checks, but people still need to interpret the scenario, validate the impact, and distinguish a real product risk from a surprising but harmless output.

Make each failure reproducible and actionable.

1. Record the system version, configuration, data source, test date, and authorized scope.
2. Describe the scenario, preconditions, steps, and the behavior the product should have shown.
3. Save relevant inputs, outputs, tool calls, and logs with sensitive values removed or protected.
4. Explain who could be affected, what could happen, and how likely the conditions are in real use.
5. Assign an owner and a fix, then repeat the scenario against the changed system.

Include unsuccessful attempts and important coverage gaps. A report that only lists dramatic examples can hide the controls that worked and the tests that were never run.

Turn useful findings into lasting assurance work.

When a finding is understood, add a safe, repeatable case to the regression suite. Keep separate tests for model behavior, application security, and user experience: a model may answer well while the application mishandles access, or a technically correct answer may still confuse the person relying on it.

NIST’s 2026 ARIA planning manual treats model testing, red teaming, and user testing as complementary parts of a system evaluation. Together, they give a broader view than any single exercise. Passing a test set describes tested conditions; it does not certify that every future use is safe.

Continue with a repeatable evaluation for AI agents ↗, permission boundaries for tools ↗, and the Symploka security principles ↗.