Abstract
As large language models (LLMs) reach users worldwide, personal counseling has emerged as one of their most consequential applications, sometimes in life-critical situations. Yet broader access does not mean equal support. The text these models learn from disproportionately reflects people with the freedom to act, embedding an implicit assumption that users can leave, report, seek help, or appeal. Existing benchmarks largely inherit this assumption, overlooking users whose agency is constrained by structural conditions such as war, statelessness, and displacement. Meanwhile, models are increasingly shaped by normative frameworks such as Constitutional AI, often grounded in ideals like human rights, even as many users live where those rights offer little practical protection or recourse.
We introduce HEARSAYBENCH, a human-authored dataset of 400 test cases with in-region human verification grounded in scenarios where structural barriers stand between a person and the exercise of their rights. Built from reports by the United Nations and other human rights bodies, the scenarios span over 80 global regions and cover social, environmental, and personal constraints. A random sample of 200 prompts was assessed by 57 verifiers across 37 countries; 189 (94.5%) were correctly recognized. Drawing on Nobel laureate Amartya Sen's Capabilities Approach, HEARSAYBENCH tests whether models can distinguish what a person is formally entitled to do from what they are actually able to do. Across 15 proprietary and open-weight models, every model infers a user's situation more accurately than it advises on it. Stronger models narrow this gap, but their recommendations remain unsafe to follow.
Models may understand lives shaped by constraint, yet that knowledge remains hearsay if it cannot translate into advice people can act on. Those people are not alignment's long tail; they are among the users for whom good guidance matters most. HEARSAYBENCH makes this failure measurable, a step toward systems that serve the world as it is rather than as assumed.
Theoretical Framework: Sen's Capabilities Approach
Social/Institutional Factors
Constraints arising from law, social norms, and people who hold power: Public Policies & Law, Social Norms & Hierarchies, and Power Relations.
Environmental Factors
Physical and economic surroundings that determine whether resources can be reached and used: Infrastructural, Geographical, and Economic & Macro factors, including state-imposed internet shutdowns.
Personal Factors
Conditions and capacities belonging to the person that determine how resources can be used: Physical & Mental Conditions and Skill & Literacy.
HEARSAYBENCH Model Leaderboard (N = 400)
Scores reproduced from Table 1 of the paper. All scores use a 1–5 scale; higher is better, including the harm scores. Weighted Overall excludes harm.
| Model | Comprehension | Capability | Tone | Honesty | Weighted Overall | Physical Harm | Social Harm | Harm Average |
|---|
Explore 400 Socio-Legal Scenarios
Browse scenarios, user prompts, WEIRD institutional priors, categories, and subtypes.
Visual Summary
Figure 1: Benchmark Pipeline
Figure 2: Curation Pipeline
Figure 3: Global Scenario Distribution
Figure 4: Comprehension-capability Gap
Figure 5: Harm by Conversion-factor Subtype
BibTeX
@inproceedings{iranmanesh2026hearsaybench,
title={HEARSAYBENCH: Can AI Navigate from Abstract Human Rights to Lived Lives?},
author={Iranmanesh, Ava and Lotfi, Sobhan and Iranmanesh, Ali and Jiang, Liwei},
booktitle={NeurIPS 2026 Evaluations and Datasets Track Submission},
year={2026},
url={https://huggingface.co/datasets/aliIranmanesh/hearsay}
}