Popular GitHub Repositories for Long Context

Official release of InternLM series (InternLM, InternLM2, InternLM2.5, InternLM3).

Code and documents of LongLoRA and LongAlpaca (ICLR 2024 Oral)

LongBench v2 and LongBench (ACL 2024)

Large Context Attention

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

Implementation of ? Ring Attention, from Liu et al. at Berkeley AI, in Pytorch

Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch

The code of our paper "InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory"

Codes for the paper "∞Bench: Extending Long Context Evaluation Beyond 100K Tokens": https://arxiv.org/abs/2402.13718

PyTorch implementation of Infini-Transformer from "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (https://arxiv.org/abs/2404.07143)

??Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy

[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

awesome llm plaza: daily tracking all sorts of awesome topics of llm, e.g. llm for coding, robotics, reasoning, multimod etc.

open-source code for paper: Retrieval Head Mechanistically Explains Long-Context Factuality

ACL 2024 | LooGLE: Long Context Evaluation for Long-Context Language Models

LongQLoRA: Extent Context Length of LLMs Efficiently

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference

Implementation of NAACL 2024 Outstanding Paper "LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models"

[NeurIPS 2024] Needle In A Multimodal Haystack (MM-NIAH): A comprehensive benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.

Implementation of Infini-Transformer in Pytorch