Hugging Face recently released LightEval, a lightweight AI evaluation tool designed to assist enterprises and research institutions in more accurately assessing large language models. This tool emphasizes the importance of evaluation in the application of AI technologies, helping to ensure the accuracy, impartiality, and alignment of model outputs with business objectives. Addressing the limitations of traditional evaluation methods, LightEval offers customized evaluation solutions and integrates with other tools from Hugging Face, such as Datat.

On September 4, 2024, the Beijing ZhiYuan Artificial Intelligence Research Institute (BAAI) announced the launch of the world's first model combat evaluation service featuring text-to-video, called FlagEval Large Model Arena. This service is open to users and covers approximately 40 large models from both domestic and international sources, supporting customizable online or offline evaluations for four major tasks: language Q&A, multimodal image and text understanding, text-to-image, and text-to-video.

Recently, DingTalk announced the launch of several new products that integrate artificial intelligence (AI) capabilities, aimed at helping small and medium-sized enterprises use AI technology at a lower cost and in a more efficient way. Among them, the 'Super Work Order Assistant' powered by AI technology in DingTalk's multidimensional table is particularly noteworthy.

According to the evaluation results from the China Academy of Information and Communications Technology, Baidu Smart Cloud's Keway intelligent customer service product received the highest rating in the first batch of 'Intelligent Customer Service Based on Large Models,' marking authoritative recognition of Baidu's technical strength and application practices in this field. Intelligent customer service, as an important application of artificial intelligence, holds significant importance for industries such as finance, government, and telecommunications, helping institutions achieve cost reduction and efficiency enhancement. Keway, as the country's first intelligent customer service product based on large models, offers advantages such as comprehensive functionality, superior performance, and mature systems, efficiently solving complex problems and enhancing problem-solving autonomy.

The Sinan OpenCompass team at the Shanghai Artificial Intelligence Laboratory has collaborated with the Modao ModelScope to launch the Compass Multi-Modal Arena, a new section of a large model evaluation platform focusing on multi-modal large models. Users can upload images and input questions to have two anonymous multi-modal large models generate answers, which can then be subjectively evaluated based on the quality of the generated content, allowing users to select the better-performing model. The platform offers an easy-to-use interface and a unique question bank.

In the field of natural language processing, large language models perform exceptionally well on complex tasks, but model evaluation heavily relies on expensive and time-consuming human-annotated data. As models advance, the utility of existing data declines, necessitating the continuous collection of new data for scalable and sustainable evaluation. The Meta FAIR research team has proposed the 'Self-Taught Evaluator' to address this issue. This innovative approach trains on synthetic data, eliminating the need for human annotation by generating contrasting synthetic preferences.

Baidu Intelligent Cloud introduced the financial intelligence application "ZhiJin" at the 2024 China International Financial Expo, aiming to deeply integrate its leading AI large-model technology with core business scenarios in the financial industry to enhance operational efficiency and revenue for financial institutions.ZhiJin is tailored for four core scenarios: wealth management, asset evaluation, business compliance, and over-the-counter (OTC) trading, offering intelligent and efficient bus

According to Bloomberg, OpenAI has created an internal scale to track the progress of its large language models in the field of Artificial General Intelligence (AGI). This move not only demonstrates OpenAI's ambition in the AGI field but also provides the industry with a new standard for measuring AI development. The scale is divided into five levels: 1. Level 1: Current chatbots, such as ChatGPT, fall into this category. 2. Level 2: Systems capable of solving basic problems at the level of a Ph

Anthropic announced on Monday the launch of a new initiative aimed at funding the development of new benchmarks capable of assessing the performance and impact of artificial intelligence models, including its own generative model, Claude. According to information released in Anthropic's official blog, the company will provide financial support to third-party organizations to develop tools that "effectively measure the advanced capabilities of artificial intelligence models." Interested organizat

Hugging Face has updated its Open LLM Leaderboard, a move that will have a significant impact on the landscape of open-source AI development. This improvement comes at a crucial moment in AI development as researchers and companies are facing a plateau in the performance enhancement of large language models (LLMs).

Anthropic's latest model, Claude 3.5 Sonnet, demonstrated remarkable performance in a recent technical evaluation, surpassing even the professional doctorate level. In the Graduate-Level Question Answering (GPQA) test, Claude 3.5 Sonnet achieved a score of 67.2%, marking the first time a large language model has exceeded 65% in such assessments, and indicating its proficiency in understanding and answering advanced scientific knowledge.

["Patronus AI Releases Copyright Detection Tool", "OpenAI's GPT-4 is Accused of Serious Infringement Issues", "Microsoft Engineers Warn that AI Image Generation Tools Could Pose Societal Risks"]

ZhiYuan Research Institute has released a code generation training dataset called TACO, aimed at providing more challenging training data and evaluation benchmarks for code generation models. TACO has advantages in terms of data scale, quality, and evaluation schemes, including a larger training and testing set, diverse problem-solving answers, and fine-grained labels. Experimental results show that current popular code generation models show significant differences compared to GPT-4 in TACO evaluations, indicating that there is still room for improvement in this field. TACO is not just a challenging...

The Midjourney V6 model is about to be released, with the community starting style evaluations. The evaluations show that V6 has significantly improved in restoring complex visual information and content creation. During the evaluation, randomly generated images from the V6 model are displayed, and users select their favorite. The new model is expected to be released before Christmas, supporting more complex content creation. According to reports from Webmaster Home, the long-awaited Midjourney V6 model, anticipated for half a year, is about to be launched.

["Zhipu AI has released an evaluation benchmark AlignBench for Chinese large models", "AlignBench can evaluate the alignment level of models and human intentions in multiple dimensions", "The dataset is divided into 8 major categories, including knowledge Q&A, writing generation, role-playing, and various types of questions", "Developers can use AlignBench for evaluation and score using a robust scoring model", "By logging into the AlignBench website, users can submit their evaluations."]

Amazon AWS has launched a Human Benchmark Testing team to enhance AI model evaluation. Amazon aims to help users better assess AI models and encourages more people to participate in this process. AWS provides model evaluation on Bedrock to assess models in its repository. Model evaluation consists of both automated and human assessments, which can evaluate model performance based on different metrics. AWS also offers a human evaluation team to collaborate with users and detect metrics that automated systems cannot.

["📊 Evaluation System of Large Models: The current evaluation system for large models has issues such as open-source datasets that can be manipulated, fairness problems arising from closed evaluation datasets, and evaluation metrics that are not sufficiently scientific and comprehensive.", "💡 Trend of Large Model Applications: The article mentions that large models have evolved from model-level development to innovation at the application level.", "🔎 Commercialization Issues of Large Models: For large model teams, achieving commercialization is far more important than rankings and parameters." ]

ZhiYuan Research Institute has open-sourced the JudgeLM evaluation model, which can efficiently assess various large models and provide scores. Compared to GPT-4, JudgeLM's cost is only 1/120, with a consistency rate of over 90% for evaluation results. JudgeLM can be applied in various assessment scenarios including pure text and multimodal contexts, generating scores and justifying reasons. The consistency of JudgeLM with reference answers exceeds 90%, approaching human performance. ZhiYuan Research Institute has also released datasets for training and validation samples for in-depth research on large models.

Ant Group, in collaboration with Peking University, has released a benchmark for evaluating large language models in the DevOps field. This benchmark includes a total of 4850 multiple-choice questions across 8 categories such as planning, coding, building, testing, and releasing. The benchmark also provides detailed evaluations for AIOps tasks, showing that the score differences among various models are minimal.

["The U.S. government plans to introduce new regulations requiring the evaluation of algorithms used in healthcare, law enforcement, and housing to prevent potential discrimination or other negative impacts on human rights.", "The new regulations may force the U.S. government to make changes in AI, such as the facial recognition technology used by the FBI, which has faced criticism for not implementing measures required by Congress to protect civil liberties.", "The new regulations will require government agencies to evaluate existing algorithms by August 2024 and discontinue the use of algorithms that do not comply with the regulations."]