2023-11-06 16:16:07.AIbase.2.9k
Ant Group CodeFuse Open Source ModelCache for Large Model Semantic Caching
The ModelCache architecture under Ant Group's CodeFuse includes the adapter, embedding, similarity, and data_manager modules. ModelCache can reduce the inference cost of large model applications and enhance user experience. Cache hits can reduce average latency by 10 times, with speed improvements of up to 14.5%. ModelCache will continue to optimize performance and accuracy, improving recall time.
