minbpe

A minimalist code project for BPE algorithm introduced by Karpathy

CommonProductProgrammingTransformerNLP

The minbpe project aims to create clean, educational code implementations of the BPE algorithm commonly used in LLMs. It provides two Tokenizers that implement the main functionalities of BPE algorithm training, encoding, decoding, etc. The code is concise and easy to read, offering users a convenient and efficient experience. The project has gained considerable attention and attractiveness, and it is believed to play an important role in the development of LLM and NLP technologies.

Implementation of BPE algorithm training
Implementation of BPE encoding for text
Implementation of decoding of BPE-encoded text
Functionality to save and load

Applied in Transformer-based language models
Used as a tokenizer for models like BERT

BPE encoding of text using minbpe
Implementing a custom BPE tokenizer using minbpe
minbpe can be used to train language models

Visit

minbpe Visit Over Time

Monthly Visits

494758773

Bounce Rate

37.69%

Page per Visit

5.7

Visit Duration

00:06:29

minbpe Visit Trend

minbpe Visit Geography

minbpe Traffic Sources

minbpe Alternatives

Create-tsi — Low-code creation AI application Generation AI RAG toolkit.

Programming•nlp•machine-learning

132

FineWeb2 — Multilingual Pretrained Dataset

Programming•Multilingual•Pretrained

Llama-3.3-70B-Instruct — A large multilingual language model with 70 billion parameters.

Programming•Large Language Model•Multilingual

102

OLMo 2 13B — High-performance English academic benchmark language model

Productivity•Language Model•Natural Language Processing

Star-Attention — EfficientInference Technology for Long Sequence Large Language Models

Programming•NVIDIA•Large Language Models

Extractous — A fast and efficient tool for unstructured data extraction

Programming•nlp•rust

126

MobileLLM-1B — A sub-billion parameter language model developed by Meta, suitable for device-side applications.

Programming•Language Model•Transformer

MobileLLM-600M — An efficient and optimized 600M parameter language model designed for device applications.

Programming•Language Model•Transformer

MobileLLM-350M — An efficiently optimized language model with sub-billion parameters, specifically designed for device-side applications.

Programming•Language Model•Transformer

Oasis — Real-time open-world AI model based on Transformer architecture.

Video•Transformer•Real-Time Interaction

432

Prompt Engineering — A comprehensive resource library for Prompt Engineering techniques.

Productivity•Prompt Engineering•Machine Learning

SLM_Survey — Research, Measurement, and Insights on Small Language Models

Education•Small Language Models•Transformer

ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer — A versatile creator and editor that follows instructions via diffusion transformers

Image•Visual Generation•Diffusion Model

144

GenAI_Agents — A comprehensive resource library for the development and implementation of generative AI agents.

Productivity•Generative AI•Agents

186

Llama-3.2-1B — Multilingual large language model

Productivity•Multilingual•Text Generation

144

iFlytek Virtual Human — Full-Stack Virtual Human Multi-Scenario Application Services

ChineseSelection•AI Virtual Image•Speech Recognition

288

OpenCity — An open-source spatiotemporal foundation model for traffic prediction.

Productivity•Traffic Prediction•Spatiotemporal Model

RAG_Techniques — A revolutionary collection of Retrieval-Augmented Generation system technologies.

Programming•RAG•NLP

168

Bailing-TTS — A large-scale text-to-speech model for generating high-quality Chinese dialect voices.

Others•text-to-speech•dialects

1644

EXAONE-3.0-7.8B-Instruct — A bilingual generative model with 780 million parameters.

chatting•NLP•Text Generation

102

MindSearch — An open-source AI search engine framework with performance comparable to Perplexity.ai Pro.

Programming•Search•Transformer

2298

Tele-FLM-1T — 1T Open-source multilingual large language model

Programming•Multilingual•Transformer

Meta-Llama-3.1-405B-Instruct-FP8 — Multilingual dialogue generation model

Programming•NLP•Multilingual

498

DCLM-7B — 700 million parameter language model, demonstrating the effectiveness of data organization technology.

Programming•language model•Transformer

228

Mamba-Codestral-7B-v0.1 — High-performance open-source code model

Programming•Code generation•Transformer

222

FlashAttention — A fast and memory-efficient implementation of the accurate attention mechanism

Programming•Deep Learning•Transformer

VideoLLaMA2-7B-16F-Base — A large video language model used for visual question answering and video subtitling generation.

Video•Video Question Answering•Video Subtitling

204

EasyAnimate — End-to-end solution for generating high-resolution and long videos.

Video•AI video generation•transformer

450

ChatRTX — Building customized large language models (LLM) to enhance the capabilities of chatbots.

InternationalSelection•NLP•Chatbot

906

Best AI Websites & Tools

minbpe

minbpe Visit Over Time

minbpe Visit Trend

minbpe Visit Geography

minbpe Traffic Sources

minbpe Alternatives

Create-tsi — Low-code creation AI application Generation AI RAG toolkit.

FineWeb2 — Multilingual Pretrained Dataset

Llama-3.3-70B-Instruct — A large multilingual language model with 70 billion parameters.

OLMo 2 13B — High-performance English academic benchmark language model

Star-Attention — EfficientInference Technology for Long Sequence Large Language Models

Extractous — A fast and efficient tool for unstructured data extraction

MobileLLM-1B — A sub-billion parameter language model developed by Meta, suitable for device-side applications.

MobileLLM-600M — An efficient and optimized 600M parameter language model designed for device applications.

MobileLLM-350M — An efficiently optimized language model with sub-billion parameters, specifically designed for device-side applications.

Oasis — Real-time open-world AI model based on Transformer architecture.

Prompt Engineering — A comprehensive resource library for Prompt Engineering techniques.

SLM_Survey — Research, Measurement, and Insights on Small Language Models

ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer — A versatile creator and editor that follows instructions via diffusion transformers

GenAI_Agents — A comprehensive resource library for the development and implementation of generative AI agents.

Llama-3.2-1B — Multilingual large language model

iFlytek Virtual Human — Full-Stack Virtual Human Multi-Scenario Application Services

OpenCity — An open-source spatiotemporal foundation model for traffic prediction.

RAG_Techniques — A revolutionary collection of Retrieval-Augmented Generation system technologies.

Bailing-TTS — A large-scale text-to-speech model for generating high-quality Chinese dialect voices.

EXAONE-3.0-7.8B-Instruct — A bilingual generative model with 780 million parameters.

MindSearch — An open-source AI search engine framework with performance comparable to Perplexity.ai Pro.

Tele-FLM-1T — 1T Open-source multilingual large language model

Meta-Llama-3.1-405B-Instruct-FP8 — Multilingual dialogue generation model

DCLM-7B — 700 million parameter language model, demonstrating the effectiveness of data organization technology.

Mamba-Codestral-7B-v0.1 — High-performance open-source code model

FlashAttention — A fast and memory-efficient implementation of the accurate attention mechanism

VideoLLaMA2-7B-16F-Base — A large video language model used for visual question answering and video subtitling generation.

EasyAnimate — End-to-end solution for generating high-resolution and long videos.

ChatRTX — Building customized large language models (LLM) to enhance the capabilities of chatbots.