SHAP for explaining model predictions

How it works

SHAP expresses a prediction relative to a baseline and distributes the difference among features. The explainer and background data affect the values you see.

What you need

Choose an explainer suited to the model and a background sample that matches the comparison you want. Record which prediction output is being explained.

What to try first

Explain several correct and incorrect examples and compare explanations across two background samples. Check how correlated features share attribution.

Use the same background data when comparing two model versions. Otherwise a change in attribution can reflect a different baseline rather than a change in model behavior.

Where to look for failures

Attribution describes a fitted model under specific assumptions. It cannot establish that changing an input would cause an outcome to change.

Related Franklin pages

An explanation of a prediction and an estimate of a causal effect ask different questions. Inspect the data and assumptions required by the selected analysis before interpreting its output.

• Estimate causal effects under a stated model of relationships and assumptions. DoWhy for explicit causal analysis
• Fit explainable models and inspect feature effects with shared visualization tools. InterpretML for explainable predictive models

For a primer on learning from data and evaluating predictions, see Franklin’s glossary. Machine learning

Project documentation

• Current project README
• README at the reviewed revision