AllenAI’s Old Ideas, Novel Problems Study Questions LLMs’ Ability to Judge Research Novelty
Current overview (not historical)
AllenAI released the "Old Ideas, Novel Problems" study and the accompanying Novelty Judge benchmark, which was peer‑reviewed at ICLR 2026. The study mined novelty signals from research papers and built two evaluation settings—Human‑Only and Human+Generated—to test large language models (LLMs) on novelty assessment. The experiments revealed that even minor prompt tweaks can shift an LLM’s accuracy by over 50 percentage points. Adding retrieval or extra reasoning steps did not consistently improve performance, and the specialized novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline. The core conclusion is that an LLM’s ability to judge research novelty is highly sensitive to the evaluation setup and is not yet reliable.
Written with AI from the reporting · updated 10/27/2026
Expand the detailed fact route and sources
Story route
A station-by-station route from the first public evidence onward; every station carries its own evidence and uncertainty.
- 1Unclassified fact stationOct 28, 00:00recorded Oct 28, 00:45single sourcerevised
发布Old Ideas, Novel Problems研究
Read original source ↗ Hugging Face BlogReplay this moment in the explorerEvidence & revisions · 1 source records
- Subject：
- allenai
- Action：
- 发布研究报告及基准测试
- Object：
- 评估大语言模型判断研究创新性的能力
- Occurred (fact side)：
- Oct 28, 00:00
- Published (valid side)：
- Oct 28, 00:12
- First seen (known side)：
- Oct 28, 00:45
Fact stationSingle source · one-sided claimRevisedCopy may have been revised later; time replay is bounded. Layer labels group reading only, not causation between layers.
Uncertainty marks flag where to check the sources; they are not proven causation.
Report timeline
Explore every side through dated source reports.
- Hugging Face BlogFirst-party sourceRelated fact: single sourceSource report revisedHugging Face Releases Benchmark Showing Large Language Models Struggle to Judge Scientific Novelty
Report first recorded Oct 28, 00:45
Hugging Face has published a new study, *Old Ideas, Novel Problems*, that evaluates how reliably large language models (LLMs) can judge the novelty of scientific research ideas. The team built a benchmark using passages from ICLR 2026 peer‑review reports that explicitly signal novelty, and released the data, code, and an automated pipeline for creating new evaluation sets. Six LLM‑based novelty judges—including GPT‑5.x and Claude Opus 4.5/4.6—were tested. Results show that minor changes to prompts or evaluation settings can shift a model’s accuracy in identifying the more novel of two ideas by over 50 percentage points. Retrieval and additional reasoning do not consistently improve judgments, and even purpose‑built novelty‑evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline. The study highlights that current novelty‑judging approaches are far from reliable and that more dependable methods are needed if AI is to play a larger role in scientific discovery.
Read original source ↗
Event activity trend
- Latest comparable
- 3
- Comparable peak
- 7Oct 28, 14:00
- 24h change
- -50%
The chart compares consistently observed participants only. Hover or tap to inspect each hour; use arrow keys to move between observations.
Event discussion
Reader opinions are hosted on GitHub Discussions; posting requires a GitHub account. Comments are not verified facts.