OpenAI Releases PaperBench, a Benchmark for Evaluating AI Agents

AIbase基地

Published inAI News · 4 min read · Apr 3, 2025

The OpenAI team has introduced PaperBench, a benchmark designed to evaluate the ability of AI agents to replicate cutting-edge AI research. This test requires AI agents to replicate, from scratch, 20 key and oral papers from the 2024 International Conference on Machine Learning (ICML). The process involves understanding the paper's contributions, developing the codebase, and successfully executing the experiments.

OpenAI, ChatGPT, Artificial Intelligence, AI

To ensure objective evaluation, researchers developed detailed scoring criteria. These criteria break down each replication task into multiple sub-tasks with clear scoring standards. PaperBench contains a total of 8316 individually scorable tasks, all scoring metrics developed in collaboration with the authors of each paper to guarantee accuracy and validity.

For large-scale evaluation, the research team also developed an automated scoring system based on a large language model (LLM). This system scores the AI agent's replication attempts according to the predefined scoring criteria. The team also established an independent benchmark for this scoring system to assess its performance.

After evaluating several leading AI models, the study found that the best-performing agent was Claude3.5Sonnet (a new version), achieving an average replication score of 21.0%. To further validate these results, researchers invited several top machine learning PhD students to attempt parts of PaperBench. The results showed that current AI models have not yet surpassed human replication capabilities.

To foster further research, the OpenAI team has decided to open-source their developed code, allowing more researchers to utilize this platform and explore the engineering capabilities of AI agents and their potential in replicating AI research.

Project code: https://github.com/openai/preparedness/tree/main/project/paperbench

Key Highlights:
🌟 PaperBench is a new benchmark for evaluating the ability of AI agents to replicate AI research, encompassing 20 ICML 2024 papers.
🔍 The test features 8316 individually scorable tasks, with scoring criteria developed in collaboration with paper authors.
🤖 Claude3.5Sonnet was the best-performing model in the test, but still hasn't surpassed top human researchers.

Breaking Free from Stiff AI: Midjourney and NYU Unlock New Dimensions in Creative Text Generation, Diversity Soars by 23%!

Researchers from Midjourney and New York University have collaborated on a novel approach to significantly enhance the diversity of creative text generated by language models while minimizing quality loss. Detailed in a recent research paper, this technique centers on incorporating a 'deviation metric' into the AI's training process. It works by quantifying the difference between each generated text and other texts created for the same prompt. Researchers utilize text embeddings and their pairwise cosine distances to calculate these differences, thereby providing the system with...

Anthropic and Databricks Partner in $100 Million Deal to Develop AI Agent Tools

Anthropic and Databricks announced a five-year, $100 million partnership to develop AI agent tools for enterprise task automation. Databricks CEO Ali Ghodsi stated that Anthropic's Claude model will be directly available on the Databricks platform, enabling clients to leverage their own data to develop custom AI solutions.

FuriosaAI, a South Korean AI Chip Startup, Rejects $800 Million Acquisition Bid from Meta

South Korean media reports that FuriosaAI, a startup specializing in AI application chips, recently rejected an $800 million acquisition offer from tech giant Meta. FuriosaAI stated it will continue focusing on independent R&D and production of its AI chips. The report indicates that the breakdown of acquisition talks stemmed from disagreements on business strategy and organizational structure, not the acquisition price itself. Similar to many tech companies building large language models (LLMs), Meta...

High School Student Creates AI Model Evaluation Website Using Minecraft

In today's rapidly advancing AI landscape, effectively evaluating and comparing the capabilities of different generative AI models is a significant challenge. Traditional AI benchmarking methods are increasingly showing their limitations, prompting AI developers to explore more innovative evaluation approaches. Recently, a website called "Minecraft Benchmark" (MC-Bench) has emerged, uniquely leveraging Microsoft's sandbox game Minecraft to facilitate model assessment.