Kimi Open Platform Announces Price Adjustments: Model Inference and Context Cache Prices Reduced

DeepSeek and Tsinghua University Joint Research: Innovative Reward Model Inference Method Improves Scalability

Researchers from DeepSeek and Tsinghua University recently published a new paper exploring scaling methods for reward model inference, seemingly advancing DeepSeek R2. Reinforcement learning is widely used in the large-scale post-training phase of large language models, but faces the challenge of obtaining accurate reward signals for these models. The researchers found that using pointwise generative reward modeling (GRM) improves model adaptability and scalability during inference. To this end, they propose Self-Principle Calibration Tuning (SPCT) learning.

NVIDIA AI Researchers Introduce FFN Fusion Technology: Accelerating Large Language Model Inference

Tsinghua Team Open-Sources Chitu Inference Engine to Boost Domestic AI Ecosystem

Recently, Professor Zhai Jidong's team from the Institute of High Performance Computing at Tsinghua University, in collaboration with Tsinghua-affiliated innovative enterprise, Qingcheng Extreme Intelligence, announced the open-sourcing of Chitu, a groundbreaking large model inference engine. This innovative technology marks another significant breakthrough for China in the AI field, particularly in inference engine development. A core highlight of the Chitu engine is its ability to natively run FP8 precision models on non-Nvidia Hopper architecture GPUs and various domestic chips.

Nous Research Unveils New API: Outshining OpenAI and Anthropic?

New York-based AI team Nous Research, known for its personalized, unrestricted language models, has launched a new model inference API. This marks a significant step for the unconventional AI group, making its advanced language models accessible to a wider range of developers and researchers through a programmatic interface. The unrestricted nature of these models has always set Nous Research apart, challenging the likes of OpenAI and Anthropic...

ByteDance's UltraMem Architecture Reduces Large Model Inference Costs by 83%

The ByteDance Doubao large model team announced today the successful development of a new sparse model architecture called UltraMem. This architecture effectively addresses the high memory access issues during the inference of MoE (Mixture of Experts) models, improving inference speed by 2 to 6 times compared to MoE, and reducing inference costs by up to 83%. This groundbreaking advancement opens a new path for efficient inference of large models. The UltraMem architecture successfully resolves the memory bottleneck during inference of MoE architectures while maintaining model performance. Experimental results show that the parameters and activation conditions are the same.

Google DeepMind Launches New Framework InfAlign: Enhancing Alignment Capability of Language Model Inference

Generative language models face many challenges during the transition from training to practical applications. One major issue is how to achieve optimal performance during the inference stage. Current strategies, such as Reinforcement Learning from Human Feedback (RLHF), primarily focus on improving the model's win rate but often overlook decoding strategies during inference, such as Best-of-N sampling and controlled decoding. This gap between training objectives and actual usage may lead to inefficiencies and affect the quality and reliability of outputs. To address these issues, Google D

Kimi and Tsinghua University Launch Open Source Model Inference Architecture Mooncake

Kimi Technology Co., Ltd. and Tsinghua University's MADSys laboratory have jointly released an open-source project called Mooncake, aimed at collaboratively building a large model inference architecture centered around KVCache. In June 2024, both parties released the design plan for the Mooncake inference system based on Kimi, which significantly enhances inference throughput through PD separation and a storage-computation architecture, garnering widespread attention in the industry.

Kimi Collaborates with Tsinghua University to Launch Mooncake Open Source Model Inference Architecture to Enhance AI Inference Efficiency

In an era of rapid development in artificial intelligence, the intelligence level of large models continues to improve, but the challenges of inference system efficiency are becoming increasingly apparent. Addressing high inference loads, reducing inference costs, and shortening response times has become an important issue faced by the industry. Kimi has collaborated with the MADSys laboratory at Tsinghua University to launch the KVCache-based Mooncake inference system design plan, which will be officially released in June 2024. The Mooncake inference system enhances efficiency through its innovative approach.

The Innovative Open Source Framework OpenR Effectively Enhances Large Model Inference Capabilities

Apple AI Research Team Discovers Limitations of Large Model Inference, Rendering OpenAI's o1 Ineffective with Just One Sentence

In the world of artificial intelligence, the reasoning capabilities of machine learning models, particularly large language models (LLMs), have been a focal point for scientists. Recently, Apple's AI research team published a paper titled 'Understanding the Limitations of Large Language Models in Mathematical Reasoning,' shedding light on these models' limitations when addressing logical problems. In the paper, researchers demonstrate this through a simple mathematical question. They first posed a question about Oliver picking kiwis: as follows: Oliver picked 44 on Friday.