Report manipulation

1. Manipulating metrics
2. Reduction
3. Combining and comparing reports

Editing metrics

Reports are forks of dictionaries and you can use normal dictionary methods to access and edit their elements (given that forks provide access to any possible methods of internal objects). For instance, you can use the following code to calculate a notion of total disparate mistreatment as the sum of dfnr and dfpr of a binary report in all brnaches and remove these entries from all branch dictionaries using Python's dictionary entry deletion:

import fairbench as fb

sensitive = fb.Fork(case1=..., case2=...)
report = fb.binreport(predictions=..., labels=..., sensitive=sensitive)
fb.describe(report)

report["mistreatment"] = abs(report.dfpr) + abs(report.dfnr)
del report["dfpr"]
del report["dfnr"]
fb.describe(report)

This will print the following to the console:

Metric          case2           case1          
accuracy        0.938           0.938          
prule           0.667           0.571          
dfpr            0.056           0.071          
dfnr            0.167           0.500          

Metric          case2           case1          
accuracy        0.938           0.938          
prule           0.667           0.571          
mistreatment    0.222           0.571

Reduction

Reports can be reduced alongside branches. Again, this operation is in general applicable to all variable forks, although this time usage is discouraged outside of report manipulation, as reduction creates new -and potentially unforeseen- data branches, but constitutes the main mechanism for summarizing multi-attribute reports into one measure. Reduction internally runs three types of functions obtained from its arguments:

• transform values found in the report for each metric, which can be either None or abs.
• expand the list of branch values for each metric, namely None, a pairwise ratio between values, or absolute differences between branch values.
• reducer method that takes a list of all branch values for each metric and summarizes them into one value. These can be mean,max,min,sum,budget, where the last one is the logarithm of the maximum declared in differential fairness formulations.

To demonstrate usage, we compute the mean, and budget of the absolute value ratio via the following code. Reduction creates new reports that comprise only one branch. The branch's name is dynamically derived by parameters (e.g., "budgetratioabs"), but you can also set a specific one with the argument name="ReductionName".

import fairbench as fb

sensitive = fb.Fork(case1=..., case2=...)
report = fb.accreport(predictions=..., labels=..., sensitive=sensitive)

mean_across_branches = fb.reduce(report, fb.mean, name="avg")
max_abs_across_branches = fb.reduce(report, fb.budget, expand=fb.ratio, transform=fb.abs)

Combining and comparing reports

Reports, including reduced ones, can be combined to create a super-report with all sub-branches. This is demonstrated in the following snippet:

new_report = fb.combine(report, mean_across_branches, max_abs_across_branches)
fb.describe(new_report)

Metric          case2           case1           avg             budgetratioabs 
accuracy        0.938           0.938           0.938           0.000          
fpr             0.056           0.071           0.063           0.251          
fnr             0.167           0.500           0.333           1.099    

Sometimes, you may want to compare the same report generated for multiple algorithms. To do this, you need to generate a fork where each branch holds a respective algorithm's default. Then, you can extract and combine values for each metric as shown in the following snipper:

reports = fb.Fork(ppr=report, lfpro=fair_report)
rep = fb.extract(acc=reports.mean.accuracy, prule=reports.minratio.pr)
fb.describe(rep)

Metric          ppr             lfpro          
acc             0.819           0.826          
prule           0.261           0.957    

When extracting values from reports, you can optionally omit the final value getter as long as is the same as the new name. For example, fb.extract(accuracy=reports.mean) is equivalent to fb.extract(accuracy=reports.mean.accuracy) given that reports.mean also returns a fork.