Popular GitHub Repositories for Multilingual Nlp

Crosslingual Generalization through Multitask Finetuning

Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023

Repo accompanying our paper "Do Llamas Work in English? On the Latent Language of Multilingual Transformers".

TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages

Generate synthetic labeled data for extremely low-resource languages using bilingual lexicons.

? Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing

Winning Solution for the Bangla Complex Named Entity Recognition Challenge - BDOSN NLP Hackathon 2023

M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysis

Whisper is an automatic speech recognition (ASR) system trained on 680,000 hours of multilingual and multitask supervised data collected from the web. We show that the use of such a large and diverse dataset leads to improved robustness to accents, background noise and technical language. Moreover, it enables transcription in multiple languages, as

This Python package is used for calculating ROUGE scores and supports over 100 languages by utilizing a multilingual BPE tokenizer. It leverages the mBERT tokenizer and was developed to support our work XL-HeadTags.

A multilingual data-driven method for generating reading comprehension questions

Data Repository for LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexicons

The project involves creating a transformer-based classifier for a multilingual text classification task.

Repository for research project exploring the benefits of cross-lingual transfer learning and pseudo-labeling for multilingual named entity recognition.