FairBench

This is a comprehensive AI fairness exploration framework developed by the MAMMOth project.

Why are you here?

⚡ Although principled exploration by generating full fairness reports is encouraged, see how to quickly compute a fairness metric in quickstart.
🤖 You may want to just investigate LLM fairness. In this case, jump to assessing LLMs.
📚 For exploring the library, visit other tutorials in the overview, or read the documentation.
⭐ Consider starring or submitting issues on GitHub.

Try it in your browser

▶ Run 🔄 Clear powered by pyodide # some sensitive attribute with values M,F,O, binary target # labels y, and predicted ones yhat (see docs for multiclass) s = ["M","F","M","F","M","F","M", "O", "O"] y = [1, 1, 0, 0, 1, 0, 1, 1, 0] yhat = [1, 1, 1, 0, 0, 0, 0, 1, 0] # one of the patterns to unpack sensitive attributes into dimensions s = fb.Dimensions(fb.categories @ s) # a fairness report for base measures (all applicable to the # arguments) computed and compared across all groups report = fb.reports.pairwise(predictions=yhat, labels=y, sensitive=s) # show in an environment (default: Console) & focus on max differences # report.help() shows what to focus on report.show(env=fb.export.ConsoleTable(legend=False)) report.maxdiff.show(env=fb.export.Console(ansiplot=False))