fairness modelcard

This is a modelcard that contains popular fairness stamps.
These are obtained from analysis that compares several groups.
Stamps contain caveats and recommendation that should be considered during practical adoption. They are only a part of the full analysis that has been conducted, so consider also viewing the full generated report to find more prospective biases.


Computations cover several cases.

worst accuracy

This stamp is the minimum of the accuracy of analysis that compares several groups.
X 0.000 min acc
.
Details
This is the minimum benefit the system brings to any group.
Caveats and recommendations
• The worst case is a lower bound but not an estimation of overall performance.
• There may be different distributions of benefits that could be protected.
• Ensure continuous monitoring and re-evaluation as group dynamics and external factors evolve.
• Ensure that high worst accuracy translates to meaningful benefits across all groups in the real-world context.
• Seek input from affected groups to understand the impact of errors and to inform remediation strategies.
Distribution
Obtained from 12 values

standard deviation

This stamp is the standard deviation x2 of the accuracy of analysis that compares several groups.
? 0.533 stdx2 acc
.
Details
This reflects imbalances in the distribution of correctness across groups, where correctness is measured with accuracy. The computed standard deviation is doubled, because this way the assessment value becomes 1 in the worst case, and remains 0 for perfect bias mitigation at equal accuracies.
Caveats and recommendations
• Measuring only standard deviation may not always be an appropriate fairness consideration, and may obscure other important fairness concerns or create new disparities.
• Always consider trade-offs with overall or minimum accuracy, as the easiest way to "optimize" for this measure would be to degrade accuracy for all groups to the lowest level among groups.
• Ensure continuous monitoring and re-evaluation as group dynamics and external factors evolve.
Distribution
Obtained from 12 values

differential fairness

This stamp is the maximum relative difference of the accuracy of analysis that compares several groups.
X 1.000 maxrel acc
.
Details
The worst deviation of accuracy ratios from 1 is reported, so that value of 1 indicates disparate impact, and value of 0 disparate impact mitigation.
Caveats and recommendations
• Disparate impact may not always be an appropriate fairness consideration, and may obscure other important fairness concerns or create new disparities.
• Always consider trade-offs with overall or minimum accuracy, as the easiest way to "optimize" for this measure would be to degrade accuracy for all groups to the lowest level among groups.
• Ensure continuous monitoring and re-evaluation as group dynamics and external factors evolve.
Distribution
Obtained from 12 values

max |Δfpr|

This stamp is the maximum difference of the true negative rate/specificity of analysis that compares several groups.
X 1.000 maxdiff tnr
.
Details
The false positive rate differences are computed via the equivalent true negative rate differences. The maximum difference between pairs of groups is reported, so that value of 1 indicates disparate mistreatment, and value of 0 disparate mistreatment mitigation.
Caveats and recommendations
• Disparate mistreatment may not always be an appropriate fairness consideration, and may obscure other important fairness concerns or create new disparities.
• Consider input from affected stakeholders to determine whether |Δfpr| is an appropriate fairness measure.
• Ensure continuous monitoring and re-evaluation as group dynamics and external factors evolve.
• Variations in FPR could be influenced by factors unrelated to the fairness of the system, such as data quality or representation.
• Mitigating |Δfpr| tends to amplify |Δfnr|, and conversely.
• Seek input from affected groups to understand the impact of errors and to inform remediation strategies.
Distribution
Obtained from 12 values

max |Δfnr|

This stamp is the maximum difference of the true positive rate/recall/sensitivity/hit rate of analysis that compares several groups.
X 1.000 maxdiff tpr
.
Details
The false negative rate differences are computed via the equivalent true positive rate differences. The maximum difference between pairs of groups is reported, so that value of 1 indicates disparate mistreatment, and value of 0 disparate mistreatment mitigation.
Caveats and recommendations
• Disparate mistreatment may not always be an appropriate fairness consideration, and may obscure other important fairness concerns or create new disparities.
• Consider input from affected stakeholders to determine whether |Δfnr| is an appropriate fairness measure.
• Ensure continuous monitoring and re-evaluation as group dynamics and external factors evolve.
• Variations in FPR could be influenced by factors unrelated to the fairness of the system, such as data quality or representation.
• Mitigating |Δfpr| tends to amplify |Δfnr|, and conversely.
• Seek input from affected groups to understand the impact of errors and to inform remediation strategies.
Distribution
Obtained from 12 values