Popular GitHub Repositories for Attention Mechanism

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

A concise but complete full-attention transformer with a set of promising experimental features from various papers

Graph Attention Networks (https://arxiv.org/abs/1710.10903)

Pytorch implementation of the Graph Attention Network model by Veličković et. al (2017, https://arxiv.org/abs/1710.10903)

Show, Attend, and Tell | a PyTorch Tutorial to Image Captioning

Keras Attention Layer (Luong and Bahdanau scores).

My implementation of the original GAT paper (Veličković et al.). I've additionally included the playground.py file for visualizing the Cora dataset, GAT embeddings, an attention mechanism, and entropy histograms. I've supported both Cora (transductive) and PPI (inductive) examples!

Reformer, the efficient Transformer, in Pytorch

To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released

Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute

Implementation of ? Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

Implementation of various self-attention mechanisms focused on computer vision. Ongoing repository.

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

A curated list of NLP resources focused on Transformer networks, attention mechanism, GPT, BERT, ChatGPT, LLMs, and transfer learning.

Text classifier for Hierarchical Attention Networks for Document Classification

My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.

pytorch implementation of "Get To The Point: Summarization with Pointer-Generator Networks"