Popular GitHub Repositories for Multi Head Attention

[VLDB'22] Anomaly Detection using Transformers, self-conditioning and adversarial training.

Exploring attention weights in transformer-based models with linguistic knowledge.

A Faster Pytorch Implementation of Multi-Head Self-Attention

Multi^2OIE: Multilingual Open Information Extraction Based on Multi-Head Attention with BERT (Findings of ACL: EMNLP 2020)

several types of attention modules written in PyTorch for learning purposes

Attention-based Induction Networks for Few-Shot Text Classification

Performance of the C++ interface of flash attention and flash attention v2 in large language model (LLM) inference scenarios.

Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using CUDA core for the decoding stage of LLM inference.

Image Captioning with Encoder as Efficientnet and Decoder as Decoder of Transformer combined with the attention mechanism.

This project aims to implement the Scaled-Dot-Product Attention layer and the Multi-Head Attention layer using various Positional Encoding methods.