Learn how to interpret SHAP values to identify hidden bias in machine learning models. Understand why differing feature weights across demographics signal disparate impact in credit scoring algorithms.
Question
You’re analyzing a credit scoring model using SHAP values. For applicants from Group A, you notice that the feature “years_at_current_address” has a mean absolute SHAP value of 0.12, while for Group B applicants, the same feature has a mean absolute SHAP value of 0.03. What does this difference most likely indicate?
A. The model treats both groups fairly because the feature is used for both.
B. The model may be exhibiting disparate impact by weighting this feature differently across demographic groups.
C. The model is more accurate for Group A than Group B.
D. Group A applicants have more stable addresses than Group B applicants.
Answer
B. The model may be exhibiting disparate impact by weighting this feature differently across demographic groups.
Explanation
To understand why this happens, you have to look at what SHAP (SHapley Additive exPlanations) values actually measure. They do not measure the raw data itself; they measure how the machine learning model interprets that data to make a decision.
A mean absolute SHAP value of 0.12 for Group A means that the feature “years_at_current_address” plays a heavy role in determining their credit score. In contrast, a score of 0.03 for Group B means the model barely considers their address history when calculating their creditworthiness.
When a model applies drastically different weights to the exact same feature depending on the applicant’s demographic group, it creates an uneven playing field. This is a classic indicator of disparate impact. The algorithm is essentially using two different sets of rules to judge two different groups of people.
Why This Leads to Disparate Impact
In the context of credit scoring, disparate impact occurs when a seemingly neutral policy disproportionately harms a protected class.
If Group A represents a demographic that historically rents rather than owns property, they naturally move more frequently. If the AI model heavily penalizes Group A for having a short address history (high SHAP value), while essentially ignoring address history for Group B (low SHAP value), the model reinforces systemic bias. The algorithm has learned a proxy for demographic identity and is using it to gatekeep financial resources.
Engineering Fair AI Systems
For infrastructure engineers and AI architects scaling models across multi-cloud environments, detecting this kind of bias early is critical. You cannot simply deploy a model and hope it acts fairly.
When you discover varying SHAP values across demographic segments during your evaluation phase, you must pause deployment. You will need to investigate the training data for historical bias, test feature dropping (removing “years_at_current_address” entirely to see if the model’s accuracy remains stable without it), and apply fairness constraints during the model training process to ensure equal predictive parity.