Kimi Technology Co., Ltd. and Tsinghua University's MADSys laboratory have jointly released an open-source project called Mooncake, aimed at collaboratively building a large model inference architecture centered around KVCache. In June 2024, both parties released the design plan for the Mooncake inference system based on Kimi, which significantly enhances inference throughput through PD separation and a storage-computation architecture, garnering widespread attention in the industry.

Generating high-quality, temporally continuous videos requires a significant amount of computational resources, especially for longer time spans. The latest Diffusion Transformer model (DiTs) has made significant progress in video generation, but its reliance on larger models and more complex attention mechanisms has resulted in slower inference speeds, exacerbating the challenge. To address this issue, researchers at Meta AI have proposed a training-free method called AdaCache to accelerate video DiTs.

Anthropic has announced the launch of a new 'Prompt Caching' feature in its Claude series of large language models, aimed at significantly reducing AI costs for enterprises and enhancing performance. This feature allows users to store and reuse specific context information without incurring additional costs or delays. In public testing, the Claude 3.5 Sonnet and Claude 3 Haiku models will implement this feature, expected to achieve up to 90% cost reduction and double the response speed in certain scenarios. This feature is particularly suited for use cases that require...

The Anthropic API introduces a new prompt caching feature aimed at optimizing the user experience with the Claude model. This feature allows users to cache frequently used contextual information between API calls, significantly reducing the costs and latency of long prompts by up to 90% and 85%, respectively. Currently, it supports the public beta versions of Claude 3.5 Sonnet and Claude 3 Haiku, with plans to expand to Claude 3 Opus in the future. It is suitable for use cases such as conversational agents, coding assistants, handling large documents, and agent searches.

The Kimi Open Platform under Dark Side of the Moon has announced a 50% reduction in context cache storage costs. Specifically, the cache storage cost has been reduced from 10 yuan/1M tokens/min to 5 yuan/1M tokens/min.

Yesterday, Kimi Open Platform under the Moon's Dark Side announced the public test of Context Caching technology. This technology can reduce the cost of using large text flagship models by up to 90% for developers, without changing the API price, and significantly improve the response speed of the model. Context Caching is an efficient data management technology that allows the system to pre-store large amounts of data or information that may be frequently requested. This way, when you request t

The ModelCache architecture under Ant Group's CodeFuse includes the adapter, embedding, similarity, and data_manager modules. ModelCache can reduce the inference cost of large model applications and enhance user experience. Cache hits can reduce average latency by 10 times, with speed improvements of up to 14.5%. ModelCache will continue to optimize performance and accuracy, improving recall time.