Metric comparisons

FairBench's main goal is to generate fairness reports that contain many types of evaluations. Reports look like the example below, where rows correspond to base metrics computed for each sensitive attribute dimension. Columns, on the other hand, include expansions and reductions that summarize metric comparisons across all protected groups or subgroups (for details on how to replicate this process with custom metrics, see see here). Below we present the comparison mechanisms computed by built-in report methods, as well as the value at which comparisons indicate full bias; scan reports for particularly bad values to notify stakeholders and explore why they occur.

--- Example multireport --- 
Metric          min             wmean           gini            minratio        maxdiff         maxbarea        maxrarea        maxbdcg        
auc             0.861           0.877           0.013           0.948           0.047           0.048           0.071           0.055          
avgscore        0.110           0.239           0.234           0.363           0.193           0.682           0.631           0.749          
tophr           0.667           0.778           0.100           0.667           0.333           nan             nan             nan            
toprec          0.001           0.002           0.392           0.121           0.004           nan             nan             nan            
avghr           0.389           0.592           0.220           0.389           0.611           0.611           0.611           0.696          
avgrepr         0.000           1.000           0.500           0.000           1.499           1.499           1.000           1.499   

Performance

These report columns assess the performance of analysed systems while accounting explicitly for the existence of subgroups.

min wmean

Computes the minimum base metric value across all groups or subgroups. Should have a high value to indicate that the base metric has high value for even then less priviledged group.
Full bias: 0

Computes the average of base metric values across groups and subgroups, weighted by group size. This places greater emphasis to more populous groups or subgroups. High values indicate high performance across groups, and low values worse performance. This mostly helps get sense of whether a min comparison is substantially smaller than the average.
Full bias: 0

Performance comparisons

These report columns directly compare the values of base metrics between groups. Depending on the report type, the groups being compared could be the pairs of groups or subgroups you indicated in the sensitive attribute fork, each group against the total population, or each group against its complement in the population.

gini minratio maxdiff minratio[vsAny] maxdiff[vsAny]

Computes the gini coefficient between metric values across all groups or subgroups. Value of 0 indicates that all groups achieve the same evaluation, whereas value of 1 indicates full imbalance between metric values.
Full bias: 1

The minimum ratio between base metric values across groups and subgroups. The comparison is made pairwise between each pair of groups. Values closer to zero indicate that imbalances between at least one pair of groups are large, and value of 1 indicates perfect equality in the measure assessments.
Full bias: 0

The maximum difference between base metric values across groups and subgroups. The comparison is made pairwise between each pair of groups. Values closer to zero indicate that all groups achieve the same evaluation, and larger values indicate the worst deviation between groups. This comparison has the same units as the measure being evaluated.
Full bias: 1

The minimum ratio of base metric values across groups and subgroups, when they are compared to the total population. Values closer to zero indicate that imbalances are large between at least one metric value and the population as a whole and value of 1 indicates that all groups perfectly follow the whole population for the specific metric.
Full bias: 0

The maximum difference of base metric values across groups and subgroups, when they are compared to the total population. Values closer to zero indicate that all groups achieve the same evaluation as the population as a whole, and larger values indicate the worst deviation between a group and the population. This comparison has the same units as the measure being evaluated.
Full bias: 1

Curve comparisons

These report columns compare the underlying curves of base measures. Which groups are compared is again determined by the report type. The base measures should keep track of curve explanations, otherwise this comparison is not possible.

maxbarea maxbdcg maxrarea

The maximum betweeness area between the curves used to compute metrics. In detail, if the base metric contains a curve in its explanation, these curves are retrieved and compared between pairs of groups (the metric's value itself is not used). The comparison considers the areas between the curves. The worst area across pairs of groups is reported. Zero area difference indicates that all groups have similar areas.
Full bias: 1

A similar strategy to maxbarea where differences are made to matter more through a normalized discounted comulative gain (NDCG) weighting. This is ideal for comparing the top hit rates of recommendation systems, but the formula has also been generalized to account for ROC curves, where false positive rate differences matter more when they occur for the same small negative rates. Zero relative area difference again indicates that all groups have similar ROC curves.
Full bias: 1

A similar strategy to maxbarea where the comparison between curve points is made not through an absolute difference but how much their ratio deviates from 1. Zero relative area difference again indicates that all groups have similar ROC curves.
Full bias: 1