Implementing-BPE-and-WordPiece-Tokenizers

Easy token price estimates for 400+ LLMs. TokenOps.

A suite of image and video neural tokenizers

Ravencoin Core integration/staging tree

Unsupervised text tokenizer focused on computational efficiency

? spaCy building blocks and visualizers for Streamlit apps

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language