Popular GitHub Repositories for Efficient Transformers

[CVPR 2022--Oral] Restormer: Efficient Transformer for High-Resolution Image Restoration. SOTA for motion deblurring, image deraining, denoising (Gaussian/real data), and defocus deblurring.

Mask Transfiner for High-Quality Instance Segmentation, CVPR 2022

Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time"

Official PyTorch Implementation of Long-Short Transformer (NeurIPS 2021).

[MICCAI 2023] DAE-Former: Dual Attention-guided Efficient Transformer for Medical Image Segmentation

Master thesis with code investigating methods for incorporating long-context reasoning in low-resource languages, without the need to pre-train from scratch. We investigated if multilingual models could inherit these properties by making it an Efficient Transformer (s.a. the Longformer architecture).

This repository contains the official code for Energy Transformer---an efficient Energy-based Transformer variant for graph classification

Demo code for CVPR2023 paper "Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers"