Popular GitHub Repositories for Tokenizer

Persian NLP Toolkit

Solves basic Russian NLP tasks, API for lower level Natasha projects

专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

A Japanese tokenizer based on recurrent neural networks

JS tokenizer for LLaMA 1 and 2

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

Text2Text Language Modeling Toolkit

Ready-made tokenizer library for working with GPT and tiktoken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

A Unified Tokenizer for Visual Generation and Understanding

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit

DadmaTools is a Persian NLP tools developed by Dadmatech Co.

AAAI 2025: Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model

phoneme tokenizer and grapheme-to-phoneme model for 8k languages

.NET 7 SDK for OpenAI with a Blazor Server playground

Implementation of MambaByte in "MambaByte: Token-free Selective State Space Model" in Pytorch and Zeta

JS tokenizer for LLaMA 3 and LLaMA 3.1

? Cutting-edge experimental spaCy components and features

A collection of tools for working with OpenAI