Popular GitHub Repositories for Efficient Attention

Quantized Attention that achieves speedups of 2.1-3.1x and 2.7-5.1x compared to FlashAttention2 and xformers, respectively, without lossing end-to-end metrics across various models.

Implementation of ? Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch

Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M context keypass retrieval

Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".

Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).