Hugging Face policy chief says embedded AI evaluators won't be easy solution

Irene Solaiman, Hugging Face's chief policy officer, argues that current AI model benchmarking lacks reproducibility and cross-comparison standards. She highlights that evaluations are moving beyond model testing to assess the entire system, including the harness, but notes that embedded evaluator qualifications remain under-standardized. The discussion follows industry concerns after safety incidents and Anthropic's controversial hiring of Accenture as an evaluator.