tokenx

Easy token price estimates for 400+ LLMs. TokenOps.

A suite of image and video neural tokenizers

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

Online playground for OpenAPI tokenizers

Ravencoin Core integration/staging tree

Unsupervised text tokenizer focused on computational efficiency

? spaCy building blocks and visualizers for Streamlit apps

go version of tiktoken

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript