AI now mathematically checks software safety

The software industry has long relied on manual testing to ensure code runs correctly, a process that is both slow and expensive. This traditional approach is being challenged by a new method that uses artificial intelligence to mathematically verify that software is safe.

Anthropic and its partner Theorem have demonstrated a system where the company's Claude model generates formal proofs of correctness for existing software. In a recent trial, the AI produced approximately 180 million lines of proof code for 123 open-source libraries over a few weeks. This output was derived from test programs within Google's OSS-Fuzz security project.

Historically, formal verification was reserved for high-stakes environments like aviation because of the high cost of specialized labor. For context, a team of experts required 20 person-years to verify the seL4 operating system kernel. By automating this work, the new system aims to make such rigorous checks more accessible to a broader range of software applications.

The process involves Claude translating crash tests into mathematical problems and analyzing compiled machine code. It then produces extensive proofs that are checked by an independent verification program called Rocq. Anthropic describes the process as fully autonomous, meaning it requires minimal human review, although the underlying tools and formal foundations were developed by humans at Theorem.

The financial and operational stakes are significant because the system identified hundreds of bugs that conventional testing had missed. These errors often only appear under specific conditions, such as unusually large input files or particular access sequences. During the verification of a widely used PDF reader, the team found as many security bugs in one week as are typically reported for that software in an entire year.

In another instance, the model detected a logical error that had remained undetected for over a decade. Anthropic notes that even when a complete proof fails, the attempt still surfaces real bugs. This capability suggests that AI could reduce the time developers spend on code review, saving costs across the entire coding lifecycle.

However, the technology has practical limits. Anthropic clarifies that formal verification does not mean the software is entirely error-free. The proofs only hold under strict conditions, including a specific processor architecture, single-threaded programs, and inputs under 64 MB. In real-world scenarios, these libraries often run on different hardware configurations.

The compute cost for these proofs is substantial, with individual runs consuming hundreds of millions of tokens. Anthropic did not disclose the total project costs. Nevertheless, the system is becoming more efficient. A smaller test run showed that proofs are reusable, and newer models like Opus 5.5 solve half the proofs with about 40 percent fewer tokens than the previous version.

This development shifts the focus of AI in software from simply writing code to proving that the code is safe. If AI can guarantee correctness under defined conditions, it could lead to greater trust in AI-generated software. Anthropic expects this method to become as routine as standard testing in the near future.

The partnership is already planning to apply this technology to critical infrastructure components. Theorem's roadmap targets major libraries such as glibc, OpenSSL, and cURL. This expansion suggests that the industry is moving toward a model where automated mathematical verification becomes a standard part of the software development process.