Hugging Face announced FairStress, a benchmark that measures hidden fairness asymmetries in large language models (LLMs). It contains 300 realistic two‑candidate scenarios across 29 domains and expands to over 13 million test cases with variations in identity presentation and adversarial prompts. The data and code are now downloadable and integrated into the Olmo‑eval workbench. Initial tests on 15 models showed that explicit demographic cues can make models switch away from the stronger candidate more often when the switch favors a minority‑coded person.
Why it matters
Developers can use FairStress to spot when their models may over‑correct for minority groups, helping them build systems that treat evidence consistently across demographics.
