Popular GitHub Repositories for Tiktoken

Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion

An elegant LLM chat UI forked from chatgpt-demo of @anse-app. Index site at https://free-chat.asia

This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.

OpenShield is a new generation security layer for AI models

? GPT token estimation and context size utilities without a full tokenizer

Predict the exact openai token usage of functions

MLX.zig: Lightweight Zig language bindings for Apple's MLX framework, enabling efficient machine learning directly on Apple Silicon with zero external dependencies.

Extracts Google Sheets to JSONL for fine-tuning, estimates task costs with tiktoken.

console based game based on a llm