Study finds coding agent decoder harnesses increase cost without significant performance gains
Current overview (not historical)
A study found that using different coding agent decoder harnesses has a negligible impact on model performance, comparable to noise from a single run. OpenCode lagged by about 9 points in large-scale tasks due to output limitations, which could not be fixed with continuation prompts. The primary cost driver for these harnesses is the system prompt and tool descriptions. Claude Code's system prompt, around 16,600 characters, resulted in a single task cost up to three times that of mini-SWE-agent, but this did not translate into improved performance.
Written with AI from the reporting · updated 10/24/2026
Expand the detailed fact route and sources
Story route
A station-by-station route from the first public evidence onward; every station carries its own evidence and uncertainty.
- 1Unclassified fact stationOct 24, 00:00recorded Oct 24, 17:30single sourcerevised
研究发现精细化编码编排器对AI编码代理成本影响显著但效果有限
Read original source ↗ The DecoderReplay this moment in the explorerEvidence & revisions · 1 source records
- Subject：
- Liu and Han
- Action：
- 进行了多模型多编排器基准测试
- Object：
- AI编码模型在SWE-bench Verified基准上的表现
- Occurred (fact side)：
- Oct 24, 00:00
- Published (valid side)：
- Oct 24, 17:21
- First seen (known side)：
- Oct 24, 17:30
Fact stationSingle source · one-sided claimRevisedCopy may have been revised later; time replay is bounded. Layer labels group reading only, not causation between layers.
Uncertainty marks flag where to check the sources; they are not proven causation.
Report timeline
Explore every side through dated source reports.
- The DecoderRelated fact: single sourceSource report revisedStudy finds complex coding agent harnesses increase cost without improving performance
Report first recorded Oct 24, 17:30
Research indicates that different coding agent harnesses have a negligible impact on model performance, with differences comparable to the noise from a single run. For instance, Zhipu AI's advertised 2.4-point lead over Claude Code falls within the measurement noise range. While heavier harnesses like Claude Code consume two to three times more tokens than the leanest option, pass rates remain within measurement uncertainty across the larger pool of 447 tasks, with only OpenCode showing a statistically significant deficit on this pool.
Read original source ↗
Event discussion
Reader opinions are hosted on GitHub Discussions; posting requires a GitHub account. Comments are not verified facts.