Search AI Products and News

Kimi Technology Co., Ltd. and Tsinghua University's MADSys laboratory have jointly released an open-source project called Mooncake, aimed at collaboratively building a large model inference architecture centered around KVCache. In June 2024, both parties released the design plan for the Mooncake inference system based on Kimi, which significantly enhances inference throughput through PD separation and a storage-computation architecture, garnering widespread attention in the industry.

In an era of rapid development in artificial intelligence, the intelligence level of large models continues to improve, but the challenges of inference system efficiency are becoming increasingly apparent. Addressing high inference loads, reducing inference costs, and shortening response times has become an important issue faced by the industry. Kimi has collaborated with the MADSys laboratory at Tsinghua University to launch the KVCache-based Mooncake inference system design plan, which will be officially released in June 2024. The Mooncake inference system enhances efficiency through its innovative approach.