METR finds Anthropic developing AI twice as fast

Anthropic is improving the capabilities of its artificial intelligence models roughly twice as quickly as it did a year ago, according to an analysis released Thursday by the independent nonprofit Model Evaluation and Threat Research, known as METR.

The findings come as the San Francisco-based company prepares for an initial public offering later this month, which could become the largest stock offering in history. METR conducted its assessment by reviewing internal data and interviewing staff members at Anthropic.

CEO Dario Amodei has publicly urged the AI industry to reduce the pace of its work so that safety measures can catch up. In an essay published on Sept. 12, Amodei asked the sector to slow the rate at which it improves model capabilities. The METR report suggests that the opposite is currently happening.

Researchers at METR believe much of this speedup is driven by AI itself helping to build newer, more advanced versions of the models. The nonprofit stated that it could confidently rule out an increase in computing power as the primary cause of the acceleration. Instead, they point to the use of AI tools in the development process.

METR noted that it found no significant bottlenecks, such as limits on data or processing power, holding back progress. The group described the potential for AI to improve its own training process as a key area of concern. They observed that while AI is not yet making fully autonomous contributions to the most critical parts of model training, researchers at Anthropic are seeing signs that this capability is approaching.

Sam McCandlish, a co-founder of Anthropic and its responsible scaling officer, responded to the report on the social platform X on Thursday. He stated that many of METR's conclusions align with his own views. McCandlish acknowledged that the company's decision to redact certain details from its public risk report makes it harder for external parties to verify METR's specific estimates.

McCandlish explained that Anthropic kept some information private because it is commercially sensitive and because sharing too much detail could provide a roadmap for competitors to increase their own speed. He added that the collaboration with METR supports the company's commitment to having independent evaluators with deep access to internal operations.

The nonprofit's assessment was based on data provided by Anthropic over an 11-week period from mid-August through late October. METR said that Anthropic's public risk report from August understated the level of acceleration occurring during that time. The group emphasized that its work is separate from the external evaluators the company is bringing on board as part of its broader safety strategy.