Hugging Face Releases Benchmark Showing Large Language Models Struggle to Judge Scientific Novelty

Hugging Face has published a new study, *Old Ideas, Novel Problems*, that evaluates how reliably large language models (LLMs) can judge the novelty of scientific research ideas. The team built a benchmark using passages from ICLR 2026 peer‑review reports that explicitly signal novelty, and released the data, code, and an automated pipeline for creating new evaluation sets. Six LLM‑based novelty judges—including GPT‑5.x and Claude Opus 4.5/4.6—were tested. Results show that minor changes to prompts or evaluation settings can shift a model’s accuracy in identifying the more novel of two ideas by over 50 percentage points. Retrieval and additional reasoning do not consistently improve judgments, and even purpose‑built novelty‑evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline. The study highlights that current novelty‑judging approaches are far from reliable and that more dependable methods are needed if AI is to play a larger role in scientific discovery.