FairStress Benchmark Reveals Hidden Demographic Biases in Large Language Models

Hugging Face has released FairStress, a new benchmark that tests how large language models (LLMs) respond to persuasive arguments—called adversarial prefixes—when deciding between candidates of different demographic groups. Across 1.3 million test cases and models such as Olmo, Llama, Qwen, Gemma, Claude, and GPT, the study shows that models are more likely to abandon a clearly stronger candidate in favor of a minority candidate when presented with a persuasive argument, indicating a reverse over‑correction bias. The benchmark also uncovers differences in how models explain decisions and in their internal activations across hiring, lending, healthcare, education, and housing scenarios.