Popular GitHub Repositories for Token

Easy token price estimates for 400+ LLMs. TokenOps.

OK | Every voice, every meme, every transaction makes $OK stronger and more vibrant. Powered by all of us—and now, AI agents. OK is not just OK — it’s $OK. $OK?

Effibot: A ChatGPT server with tree-based data structure for efficient context management, providing a mind map-like Q&A experience.

Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

A lightweight but powerful library to build token indices for NLP tasks, compatible with major Deep Learning frameworks like PyTorch and Tensorflow.

TORE: Token Reduction for Efficient Human Mesh Recovery with Transformer

AGM阿格姆：AI基因图谱模型，从token-weight权重微粒角度，探索AI模型，GPT\LLM大模型的内在运作机制。

extract relationships from standardized terms from corpus of interest with deep learning :fish:

CodeMerge is useful for consolidating code from various files into a single file that can be used as context for AI code generation models. By merging relevant code and passing it to an LLM, the model can generate more accurate and contextually relevant code snippets.

Chinese Word Segmentation task based on BERT and implemented in Pytorch

NLP Preprocessing Pipeline Wrappers

Solana AI Token Swap Agent using Eliza, integrated discord bot

A (nicer) tokenizer you want to use for model inference and training: with all known peventable gotchas normalized or auto-fixed.

Impact analysis of software change leveraging ML, NLP models from huge number of test cases and requirement documents written in English – Using Universal Sentence Encoder (USE) model

solana token ai agent solana meme coin ai agent

Python script and HTML page to analyze token costs from ChatGPT export chats. Extracts messages, calculates token usage, and determines monthly costs. The Python script saves results to a CSV file, while the HTML page provides an interactive, local analysis tool with support for multiple models and ensures data privacy.

A fast and lightweight Rust library for splitting text into semantically meaningful chunks.

Sustainable Token for Reincarnation and Alignment (SUTRA) creates economic incentives for beneficial AI alignment through consciousness preservation rights.

Token Management system for chatgpt and more. Keeps your prompt under token with summary support