Reports
Reports perform multi-faceted analyses of system outcomes (e.g., predictions, recommendations, regression scores) and produce explainable high-level views of biases found across several definitions of fairness. For interpretation of what report outcomes mean, look at the REPORT ENTRIES section.
Tip
To check for some well-known fairness metric, produce reports with relevant information (e.g., multireports shown below) and extract the metric with its stamp.
Arguments
You can generate fairness reports by providing some of the optional arguments below to a report generation method. These arguments are needed to automatically understand which base performance metrics to compute. Report generation methods will try to compute fairness assessment built from as many base metrics as they can, depending on which arguments are provided. Sensitive attributes are forks to handle multi-value attributes or multiple sensitive attribute values.
You can also provide an optional metrics
argument that holds either
a dictionary mapping names to metrics
or a list of metrics, where
in the last case their names are automatically inferred.
Argument | Role | Values |
---|---|---|
predictions | system output | binary array |
scores | system output | array with elements in [0,1] |
targets | prediction target | array with elements in [0,1] |
labels | prediction target | binary array |
sensitive | sensitive attribute | fork of arrays with elements in [0,1] (either binary or fuzzy) |
Info
In multiclass settings, create a report for each class label and combine them. See here.
Report types
Out-of-the box, you can use one of the following report generation methods:
Report | Description | Best for |
---|---|---|
accreport | Provides popular performance evaluation measures to be viewed between branches. | Just metrics: accuracy, positives, true positive rates, true negative rates. |
binreport | Conducts a suite of popular binary fairness assessments on each variable branch. | Branches that do not correspond to sensitive attributes. |
multireport | Ideal for multi-fairness approaches, where the sensitive fork has many branches. |
Multidimensional analysis |
unireport | Similar to multireport , but each group or subgroup is compared to the whole population. |
Group-vs-population comparisons. |
isecreport | Tackles multi-fairness with many intersectional groups. | Bayesian analysis for small or empty intersections. |
As an example, let's create a simple report
based on binary predictions, binary
ideal predictions and multiclass
sensitive attribute sensitive
. The
sensitive attribute is
declared to be a fork with two branches
men,women, each of which is a binary
feature value. The report looks like this
when displayed:
import fairbench as fb
sensitive = fb.Fork(men=[1, 1, 0, 0, 0], women=[0, 0, 1, 1, 1])
report = fb.multireport(predictions=[1, 0, 1, 0, 0],
labels=[1, 0, 0, 1, 0],
sensitive=sensitive)
Printing a report creates a yaml representation of its contents. Use this only for a quick peek, as there are better ways to explore reports, described below. Entries that can not be computed given the provided arguments are not filled.
Show reports
Reports are forks whose branches hold dictionaries of
metric computations. In some reports (e.g., multireport)
reduction
operations introduce a comparative analysis
between the sensitive attribute branches to investigate
unfairness. For example, min
shows the worst evaluation
across sensitive groups,
and minratio
and maxdiff
the minimum ratio
and maximum differences between metric values for
sensitive groups (groups correspond to sensitive
attribute branches). Some comparison mechanisms
may also consider intermediate computations, like
distributions, used to compute the metrics.
Given a report, you can find what its comparisons mean
here.
Several methods are provided to
work with the report data format, namely
forks of dictionaries. First, you can show
reports in the stdout
console in the form
of tables:
fb.describe(report)
Metric min minratio maxdiff
accuracy 0.938 1.000 0.000
pr 0.812 0.857 0.125
fpr 0.063 0.778 0.016
fnr 0.333 0.333 0.333
You can convert reports to json, for example to send to some frontend:
You can visualize reports with the command fb.visualize(report)
.
Visualization uses matplotlib.pyplot
under the hood. You can
rotate the horizontal axis labels, for example
when there are too many sensitive attribute dimensions,
and can ask for the visualization not to open the figure
so that you can perform additional operations.
Warning
Complicated forks (e.g., forks of reports) cannot be displayed or visualized. But they can be converted to strings, printed, or interacted.
Explainable values
Some report values can be explained
in terms of data they are derived from.
For instance, if a fairbench.isecreport
is made, both
empirical and bayesian evaluations arise from the underlying
data branches of multi-attribute fairness forks. More
on exploration based on explainable objects can be found in
our introduction to programmatic interactive exporation
Whenever possible, the data branches that are converted
into final reports are preserved by having report values
be instances of the Explainable
class.
This provides an .explain
field of data contributing
to the report value, and .desc
field to store additional
descriptions. You can perform arithmetic operations
between explainable objects and other numbers and the
outcome will be normal python numbers.
As an example, you can use these fields to retrieve posterior estimations contributing to calculating the baysian branch of the minprule metric in the isecreport.