Recently, Vectara released a report titled "Hallucination Leaderboard," which compares the performance of different large language models (LLMs) in generating hallucinations while summarizing short documents. This leaderboard utilizes Vectara's Hughes Hallucination Evaluation Model (HHEM-2.1), which is updated regularly to assess the frequency with which these models introduce false information in their summaries. According to the latest data, the report highlights key metrics such as the hallucination rate, factual consistency rate, response rate, and average summary length of various popular models.
In the latest ranking, Google's Gemini 2.0 series performed exceptionally well, particularly the Gemini-2.0-Flash-001 model, which topped the list with a low hallucination rate of 0.7%, demonstrating its capability of introducing almost no false information when processing documents. Additionally, the Gemini-2.0-Pro-Exp and OpenAI's o3-mini-high-reasoning models followed closely with hallucination rates of 0.8%, also showing commendable performance.
The report also indicates that although the hallucination rates of many models have increased, most still maintain a relatively low level, with the factual consistency rates of multiple models exceeding 95%, indicating their strong ability to ensure the accuracy of information. Notably, the response rates of the models are generally high, with the vast majority nearing 100%, signifying their excellent performance in understanding and responding to questions.
Furthermore, the leaderboard mentions the average summary lengths of different models, highlighting the variations in their ability to condense information. Overall, this leaderboard not only provides important reference data for researchers and developers but also offers ordinary users an easy way to understand the current performance of large language models.
Specific ranking access: https://github.com/vectara/hallucination-leaderboard
Key Points:
🌟 The latest hallucination leaderboard evaluates the performance of different large language models in document summarization.
🔍 Google's Gemini series models stand out, with a hallucination rate as low as 0.7%.
📊 The models' response rates are close to 100%, showcasing their efficiency in information processing.