University of Michigan team releases CloudOpsBench, identifies reasoning as top AI cloud ops bottleneck

CloudOpsBench is a new open-source benchmark that measures how well AI agents handle typical DevOps tasks in the cloud, including fixing services, removing resources, restricting access rights, rolling out updates, and decommissioning projects. Version 1.0 includes 45 tasks on AWS and was built by experienced DevOps engineers. A team led by Ang Chen at the University of Michigan developed the benchmark. Even the best models fail at many tasks: Claude Fable 5.1 solved 60 percent and GPT-6 Astra managed 53 percent, while no model completed every task. The team said the main problem is not a lack of knowledge but reasoning. About 80 percent of failures occur because agents misjudge task scope, fail to check all relevant services, or stop too early. Roughly 20 percent of errors are wrong actions, some with irreversible consequences for infrastructure. All tasks run identically in the real AWS cloud and in an emulator. Extensions for GCP, Azure, and OCI are planned. The GitHub project is licensed under Apache 2.0, and a detailed failure analysis is available.