The Transformer architecture, a star in the field of artificial intelligence, has led a revolution in natural language processing with its self-attention mechanism at its core. However, when handling long contexts, the resource consumption of self-attention calculations becomes a bottleneck. To address this issue, researchers have proposed the Tree Attention method, which decomposes the calculation tasks through tree reduction to improve efficiency. This method not only reduces communication overhead and memory usage but is also 8 times faster than existing methods in a multi-GPU environment. The introduction of Tree Attention not only enhances performance.

Zyphra has launched the Zamba2-2.7B language model, which is a milestone in the small language model domain. Its performance and efficiency have significantly improved, with a training dataset size of around 30 trillion tokens, reducing resource requirements during inference and making it an efficient solution for mobile applications. Key highlights include a twofold increase in response generation speed, a 27% reduction in memory usage, and a 1.29 times decrease in generation latency, particularly suited for applications requiring real-time interaction, such as virtual assistants and chatbots.