Explainability Methods and Their Honest Limits
Explainability methods attempt to show why a model made a particular prediction. Common methods include feature importance, SHAP values, LIME and gradient-based attribution. These methods are useful for understanding model behaviour but have honest limits that you must acknowledge and understand.
In a real-world scenario, a financial institution using a machine learning model to assess loan applications may rely on explainability tools to justify decisions to regulators and customers. However, even when these tools are applied correctly, they must be interpreted with caution and in context.
Feature importance scores tell you which variables the model relies on most when making predictions. If a credit model has high feature importance for income and credit history, the model considers those variables most when deciding. For example, in a UK mortgage lending model, income and credit score may be flagged as the most influential features.
But feature importance does not tell you the direction of effect (does higher income lead to approval or rejection?) or how variables interact with each other. It is incomplete information. A lending officer might see that age is a low-importance variable and assume it is irrelevant, when in fact age may interact with other features like employment status to influence outcomes.
This misunderstanding can lead to incorrect conclusions about model fairness or decision-making logic.
SHAP (Shapley additive explanations) values estimate how much each variable contributed to pushing the prediction away from the model’s average prediction. SHAP values are mathematically principled and grounded in game theory. However, they are computationally expensive to calculate.
They can be misleading when variables are correlated, because the Shapley value distributes credit between correlated variables in ways that may not match your causal intuition about which variable truly matters. For instance, in a UK healthcare model predicting patient readmission risk, variables such as length of stay and number of previous admissions may be highly correlated.
SHAP may attribute a portion of the prediction to each variable, even though one may be a proxy for the other. This can mislead clinicians into thinking that both variables are independently important when one is merely a consequence of the other.
LIME (local interpretable model-agnostic explanations) builds a local linear approximation around a specific prediction to show which variables matter for that particular case. LIME is useful for explaining individual predictions but the local linear approximation may not be accurate if the model is highly nonlinear or has complex interactions.
A UK insurance company using a model to price motor policies may use LIME to explain why a specific customer was charged a higher premium. However, if the underlying model is complex and non-linear, the local explanation may not reflect the true decision process. A customer may receive an explanation that seems reasonable but is based on a simplified model that doesn’t capture the full complexity of the decision.
This can lead to confusion and a false sense of understanding.
The honest limits are these: explainability methods show what the model did, not why humans should trust it. They do not prove the model is fair. They do not prove the model is accurate on data the model has never seen.
They do not prove the model is making a good decision. A model can be easily explainable and still dangerously unfair. A model can produce explanations that seem reasonable while making systematic errors for underrepresented subgroups.
For example, a UK recruitment model that is explainable but systematically under-represents candidates from certain ethnic backgrounds may still appear to be making fair decisions based on its explanations. This highlights the need to combine explainability with fairness audits and human oversight.
Do not claim that explainability alone solves fairness or safety. Regulators are increasingly sceptical of claims that a model is safe simply because it is explainable. In the UK, the Information Commissioner’s Office (ICO) and the Financial Conduct Authority (FCA) have expressed concern over over-reliance on explainability as a proxy for responsible AI. Explainability is useful for debugging and for oversight but is not a substitute for thorough testing, fairness analysis and human review.
A UK retail company using a model to predict customer churn must not rely solely on explainability to justify its decisions. Instead, it must combine explainability with bias testing, performance benchmarking and stakeholder feedback. This multi-step process ensures that the model is not only explainable but also dependable and ethical.
Practical steps to manage explainability include: regularly auditing model decisions using multiple explainability tools; training staff to interpret explanations critically; and integrating explainability into model development workflows. For example, a UK energy company deploying a model to predict customer demand must ensure that explainability is part of its model governance process.
This means that before deploying a model, it must be tested using SHAP and LIME to understand how it behaves. Then, it must be reviewed by a diverse team to ensure that explanations align with business logic and ethical standards. Only then should the model be considered for production use.
This process helps prevent overconfidence in explainability and ensures that models are both interpretable and responsible.