tokenmonster

Easy token price estimates for 400+ LLMs. TokenOps.

A suite of image and video neural tokenizers

Ravencoin Core integration/staging tree

Unsupervised text tokenizer focused on computational efficiency

? spaCy building blocks and visualizers for Streamlit apps

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

? vibrato: Viterbi-based accelerated tokenizer