Fairness & Robustness in Risk Detection Models

Problem

Automated moderation decides what gets seen online. Toxicity classifiers are usually tested for fairness or for robustness to rewording, rarely both, so nobody checks whether the people they protect least are also the easiest to attack.

Approach

• A config-driven pipeline. Every experiment is a YAML file, and every run is saved with its exact config, predictions and metrics, so any number in the dissertation traces back to a run.
• Four open-source classifiers (IBM Granite Guardian HAP-38M, RoBERTa-Toxicity, Detoxify-Unbiased, Toxic-BERT) across six datasets, including CivilComments-WILDS, HateCheck, HateXplain and ToxiGen.
• Purpose-built benchmarks to isolate failure modes: counterfactual identity swaps, semantic and jailbreak attacks, leetspeak, over-refusal on benign identity-related text, and target masking.
• Proper statistics: bootstrap confidence intervals, permutation tests, Benjamini–Hochberg correction and effect sizes, plus a human annotation study of about 1,040 paraphrase pairs.
• Interpretability: SHAP token attributions show identity terms among the top drivers of false positives.

A note on reading the numbers

Some groups show a 0% bypass rate. That is not good news: the original, undisguised attacks on those groups were never flagged in the first place, so there was nothing to bypass. The explorer on the home page marks those groups as n/a rather than 0%.