minbpe

A minimalist code project for BPE algorithm introduced by Karpathy

CommonProductProgrammingTransformerNLP

The minbpe project aims to create clean, educational code implementations of the BPE algorithm commonly used in LLMs. It provides two Tokenizers that implement the main functionalities of BPE algorithm training, encoding, decoding, etc. The code is concise and easy to read, offering users a convenient and efficient experience. The project has gained considerable attention and attractiveness, and it is believed to play an important role in the development of LLM and NLP technologies.

Best AI Websites & Tools

minbpe

minbpe Visit Over Time

minbpe Visit Trend

minbpe Visit Geography

minbpe Traffic Sources

minbpe Alternatives

Create-tsi — Low-code creation AI application Generation AI RAG toolkit.

Oasis — Real-time open-world AI model based on Transformer architecture.

Prompt Engineering — A comprehensive resource library for Prompt Engineering techniques.

SLM_Survey — Research, Measurement, and Insights on Small Language Models

ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer — A versatile creator and editor that follows instructions via diffusion transformers

GenAI_Agents — A comprehensive resource library for the development and implementation of generative AI agents.

Llama-3.2-1B — Multilingual large language model

iFlytek Virtual Human — Full-Stack Virtual Human Multi-Scenario Application Services

OpenCity — An open-source spatiotemporal foundation model for traffic prediction.

RAG_Techniques — A revolutionary collection of Retrieval-Augmented Generation system technologies.

Bailing-TTS — A large-scale text-to-speech model for generating high-quality Chinese dialect voices.

EXAONE-3.0-7.8B-Instruct — A bilingual generative model with 780 million parameters.

MindSearch — An open-source AI search engine framework with performance comparable to Perplexity.ai Pro.

Tele-FLM-1T — 1T Open-source multilingual large language model

Meta-Llama-3.1-405B-Instruct-FP8 — Multilingual dialogue generation model

DCLM-7B — 700 million parameter language model, demonstrating the effectiveness of data organization technology.

Mamba-Codestral-7B-v0.1 — High-performance open-source code model

FlashAttention — A fast and memory-efficient implementation of the accurate attention mechanism

VideoLLaMA2-7B-16F-Base — A large video language model used for visual question answering and video subtitling generation.

EasyAnimate — End-to-end solution for generating high-resolution and long videos.

ChatRTX — Building customized large language models (LLM) to enhance the capabilities of chatbots.

Llama-Chinese — Building the best Chinese Llama large model, open source and available for commercial use.

SpaceByte — SpaceByte is a new byte-level decoding architecture that avoids the defects of Tokenization.

LLM Transparency Tool — Analyzes the inner workings of Transformer-based language models.

Pile-T5 — A T5 model trained on the Pile dataset

Infini-attention — Extends the Transformer model to handle infinitely long inputs

Qwen1.5-32B — A series of Transformer-based pre-trained language models

speakai.co — Transcription, Research, Data Analysis and NLP software

DataDreamer — A tool for generating synthetic data to train and align models.