minbpe

A minimalist code project for BPE algorithm introduced by Karpathy

CommonProductProgrammingTransformerNLP

The minbpe project aims to create clean, educational code implementations of the BPE algorithm commonly used in LLMs. It provides two Tokenizers that implement the main functionalities of BPE algorithm training, encoding, decoding, etc. The code is concise and easy to read, offering users a convenient and efficient experience. The project has gained considerable attention and attractiveness, and it is believed to play an important role in the development of LLM and NLP technologies.

Implementation of BPE algorithm training
Implementation of BPE encoding for text
Implementation of decoding of BPE-encoded text
Functionality to save and load

Applied in Transformer-based language models
Used as a tokenizer for models like BERT

BPE encoding of text using minbpe
Implementing a custom BPE tokenizer using minbpe
minbpe can be used to train language models

Visit

minbpe Visit Over Time

Monthly Visits

515580771

Bounce Rate

37.20%

Page per Visit

5.8

Visit Duration

00:06:42

minbpe Visit Trend

minbpe Visit Geography

minbpe Traffic Sources

minbpe Alternatives

Create-tsi — Low-code creation AI application Generation AI RAG toolkit.

Programming•nlp•machine-learning

132

Extractous — A fast and efficient tool for unstructured data extraction

Programming•nlp•rust

MobileLLM-1B — A sub-billion parameter language model developed by Meta, suitable for device-side applications.

Programming•Language Model•Transformer

MobileLLM-600M — An efficient and optimized 600M parameter language model designed for device applications.

Programming•Language Model•Transformer

MobileLLM-350M — An efficiently optimized language model with sub-billion parameters, specifically designed for device-side applications.

Programming•Language Model•Transformer

Oasis — Real-time open-world AI model based on Transformer architecture.

Video•Transformer•Real-Time Interaction

324

Prompt Engineering — A comprehensive resource library for Prompt Engineering techniques.

Productivity•Prompt Engineering•Machine Learning

SLM_Survey — Research, Measurement, and Insights on Small Language Models

Education•Small Language Models•Transformer

ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer — A versatile creator and editor that follows instructions via diffusion transformers

Image•Visual Generation•Diffusion Model

120

GenAI_Agents — A comprehensive resource library for the development and implementation of generative AI agents.

Productivity•Generative AI•Agents

162

Llama-3.2-1B — Multilingual large language model

Productivity•Multilingual•Text Generation

132

iFlytek Virtual Human — Full-Stack Virtual Human Multi-Scenario Application Services

ChineseSelection•AI Virtual Image•Speech Recognition

270

OpenCity — An open-source spatiotemporal foundation model for traffic prediction.

Productivity•Traffic Prediction•Spatiotemporal Model

RAG_Techniques — A revolutionary collection of Retrieval-Augmented Generation system technologies.

Programming•RAG•NLP

150

Bailing-TTS — A large-scale text-to-speech model for generating high-quality Chinese dialect voices.

Others•text-to-speech•dialects

1398

EXAONE-3.0-7.8B-Instruct — A bilingual generative model with 780 million parameters.

chatting•NLP•Text Generation

102

MindSearch — An open-source AI search engine framework with performance comparable to Perplexity.ai Pro.

Programming•Search•Transformer

2226

Tele-FLM-1T — 1T Open-source multilingual large language model

Programming•Multilingual•Transformer

Meta-Llama-3.1-405B-Instruct-FP8 — Multilingual dialogue generation model

Programming•NLP•Multilingual

486

DCLM-7B — 700 million parameter language model, demonstrating the effectiveness of data organization technology.

Programming•language model•Transformer

228

Mamba-Codestral-7B-v0.1 — High-performance open-source code model

Programming•Code generation•Transformer

222

FlashAttention — A fast and memory-efficient implementation of the accurate attention mechanism

Programming•Deep Learning•Transformer

VideoLLaMA2-7B-16F-Base — A large video language model used for visual question answering and video subtitling generation.

Video•Video Question Answering•Video Subtitling

186

EasyAnimate — End-to-end solution for generating high-resolution and long videos.

Video•AI video generation•transformer

426

ChatRTX — Building customized large language models (LLM) to enhance the capabilities of chatbots.

InternationalSelection•NLP•Chatbot

882

Llama-Chinese — Building the best Chinese Llama large model, open source and available for commercial use.

Programming•NLP•Open Source

360

SpaceByte — SpaceByte is a new byte-level decoding architecture that avoids the defects of Tokenization.

Programming•Byte-level Model•Large-Scale Language Model

LLM Transparency Tool — Analyzes the inner workings of Transformer-based language models.

Programming•Language Model•Transformer

282

Pile-T5 — A T5 model trained on the Pile dataset

Programming•NLP•Machine Learning

252

Best AI Websites & Tools

minbpe

minbpe Visit Over Time

minbpe Visit Trend

minbpe Visit Geography

minbpe Traffic Sources

minbpe Alternatives

Create-tsi — Low-code creation AI application Generation AI RAG toolkit.

Extractous — A fast and efficient tool for unstructured data extraction

MobileLLM-1B — A sub-billion parameter language model developed by Meta, suitable for device-side applications.

MobileLLM-600M — An efficient and optimized 600M parameter language model designed for device applications.

MobileLLM-350M — An efficiently optimized language model with sub-billion parameters, specifically designed for device-side applications.

Oasis — Real-time open-world AI model based on Transformer architecture.

Prompt Engineering — A comprehensive resource library for Prompt Engineering techniques.

SLM_Survey — Research, Measurement, and Insights on Small Language Models

ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer — A versatile creator and editor that follows instructions via diffusion transformers

GenAI_Agents — A comprehensive resource library for the development and implementation of generative AI agents.

Llama-3.2-1B — Multilingual large language model

iFlytek Virtual Human — Full-Stack Virtual Human Multi-Scenario Application Services

OpenCity — An open-source spatiotemporal foundation model for traffic prediction.

RAG_Techniques — A revolutionary collection of Retrieval-Augmented Generation system technologies.

Bailing-TTS — A large-scale text-to-speech model for generating high-quality Chinese dialect voices.

EXAONE-3.0-7.8B-Instruct — A bilingual generative model with 780 million parameters.

MindSearch — An open-source AI search engine framework with performance comparable to Perplexity.ai Pro.

Tele-FLM-1T — 1T Open-source multilingual large language model

Meta-Llama-3.1-405B-Instruct-FP8 — Multilingual dialogue generation model

DCLM-7B — 700 million parameter language model, demonstrating the effectiveness of data organization technology.

Mamba-Codestral-7B-v0.1 — High-performance open-source code model

FlashAttention — A fast and memory-efficient implementation of the accurate attention mechanism

VideoLLaMA2-7B-16F-Base — A large video language model used for visual question answering and video subtitling generation.

EasyAnimate — End-to-end solution for generating high-resolution and long videos.

ChatRTX — Building customized large language models (LLM) to enhance the capabilities of chatbots.

Llama-Chinese — Building the best Chinese Llama large model, open source and available for commercial use.

SpaceByte — SpaceByte is a new byte-level decoding architecture that avoids the defects of Tokenization.

LLM Transparency Tool — Analyzes the inner workings of Transformer-based language models.

Pile-T5 — A T5 model trained on the Pile dataset