What is explainable AI?
Make the explanation answer to the model.

An explanation is evidence for a particular audience and purpose.

Explainable AI (XAI) describes methods and system designs that make an AI outcome or process easier to examine. A useful explanation names or shows something relevant about a result, in a form its intended reader can understand. It should also represent the process that produced the result and make clear when the system is outside its limits.

NIST’s four principles make the standard concrete: provide an explanation; make it meaningful to the person receiving it; make it accurate to the system’s process; and define the conditions in which the system is expected to operate. A developer debugging a model and a person affected by its decision need different explanations.

In practice, interpretability and explainability are often used together. A useful distinction is whether a person can understand the model’s decision structure itself, or whether another method summarizes a model that is harder to inspect. Ask what a particular explanation covers instead of relying on the label.

A feature score, a heat map, and a written rationale are not interchangeable.

Each method has its own scope, assumptions, and failure modes. None alone proves that a prediction is correct, fair, causal, or safe to use. For those questions, pair the explanation with outcome evaluation and a review path. Continue with AI bias and fairness ↗ and AI governance ↗.

A persuasive picture can still explain the wrong thing.

Adebayo and colleagues proposed sanity checks that deliberately change a model’s learned parameters or the data used to train it, then examine whether the explanation changes. In their experiments, some popular saliency methods were insensitive to both the model and the data. A human could find those maps plausible even though they did not reflect what that model had learned.

Another study showed how a wrapper around a classifier could be constructed to make tested versions of LIME and SHAP display benign-looking explanations while the underlying classifier behaved differently on the relevant cases. This is a specific adversarial demonstration, not evidence that every use of LIME or SHAP is deceptive. It does show why checking the model and the explanation together matters.

Test the claim, the reader, and the decision that follows.

1. Name the reader and the decision. State who needs the explanation, what they should understand, and what action it is meant to support.
2. Write down the method and scope. Record the model version, explainer, baseline or reference, input representation, and whether the result is local or intended to describe broader behavior.
3. Check dependence on the model. Use appropriate randomization or ablation checks. If parameters or learned signals change while the explanation stays fixed, investigate before relying on it.
4. Probe the input carefully. Compare plausible variations and equivalent examples. Explain what the test can establish; a perturbation that creates an unrealistic input can produce a misleading result too.
5. Check whether people can use it. Ask representative readers what the explanation means and whether it improves the decision. Technical fidelity and human usefulness are separate requirements.
6. Show uncertainty and boundaries. Say when an explanation is unavailable, unstable, or outside the tested conditions. Preserve a way for a person to question or appeal a consequential result.
7. Measure the prediction separately. Test accuracy, errors, subgroup outcomes, and workflow impact on suitable cases. A strong explanation score cannot substitute for these measures.

For high-impact systems, keep the explanation review inside a broader evaluation plan. The guides to reading AI benchmarks ↗ and evaluating AI agents ↗ cover how to define representative cases and evidence before making a release decision.

For a consequential decision, ask whether the model itself can be made understandable.

Cynthia Rudin argues that in high-stakes settings, teams should consider an intrinsically interpretable model instead of defaulting to a black box and adding a post-hoc explanation. That is a strong design position, not a universal rule for every domain. It makes a useful question unavoidable: does a more complex model earn its place after a suitable interpretable alternative is measured on the same task?

Compare candidates on representative data, relevant errors, operating conditions, and the consequences of mistakes. If an opaque model remains necessary, treat explanations as one tested part of the safety and accountability case—not as a certificate. Tie them to human oversight, documented limits, and ongoing monitoring.

For the wider system around a model, continue to governance and ownership ↗ or making intended behavior testable ↗.

One explanation cannot carry the whole case.

Check the system’s outcomes in context, trace its evidence, and give people a clear route to question a result. Start with fairness and the decisions that follow ↗, then explore the limits of generated reasoning traces ↗.