Popular GitHub Repositories for Longformer Models

Master thesis with code investigating methods for incorporating long-context reasoning in low-resource languages, without the need to pre-train from scratch. We investigated if multilingual models could inherit these properties by making it an Efficient Transformer (s.a. the Longformer architecture).

Convert pretrained RoBerta models to various long-document transformer models

LongPegasus package is used for inducing longformer self attention over base pegasus abstractive summarization model to increase the token limit and performance.