Tencent has released an open-source MOE large language model, Hunyuan-large, with a total parameter count of 398B and an active parameter count of 52B. Public evaluation results indicate that Tencent Hunyuan Large outperforms models like Llama3.1 and Mixtral in comprehensive evaluations across multiple disciplines including CMMLU, MMLU, CEva1, MATH, and in tasks involving Chinese and English NLP, coding, and mathematics, among other dimensions.
It is understood that the model's technological innovations enable the generation of high-quality synthetic data, effectively addressing the shortage of natural data through enhanced training with synthetic data. In terms of context processing capabilities, the pre-trained model supports text sequences up to 256K in length, significantly enhancing the ability to handle long-context tasks.
Additionally, Tencent Hunyuan has announced plans to open-source the PenguinScrolls evaluation set to address the industry's lack of real long-text evaluation sets, thereby aiding application research in the field. The self-developed PenguinScrolls is based on various natural long texts such as public financial, legal, and academic papers, with lengths ranging from 1K to 128K, covering a wide range of deep reading comprehension and long-text reasoning tasks.
The release of the Tencent Hunyuan Large language model and the open-source PenguinScrolls evaluation set will provide the industry with more powerful language models and evaluation tools, propelling the development of natural language processing and artificial intelligence.