LLMs-from-scratch

Deep dive into the inner workings of large language models.

CommonProductProgrammingLanguage ModelsDeep Learning

LLMs-from-scratch takes you on a journey to understand how LLMs work. This book guides you step-by-step in creating your own LLM, explaining each stage with clear text, diagrams, and examples. The described methods for training and developing small but functional models for educational purposes share similarities with techniques used to create large-scale foundation models like ChatGPT.

Deep understanding of large language models
Processing text data
Implementing attention mechanisms
Building a GPT model from scratch
Pre-training with unlabeled data
Fine-tuning for text classification
Fine-tuning with human feedback
Practical applications of large language models

Education & Research

Textbook for university deep learning courses
Reference guide for researchers
Self-learning resource for AI engineers

Visit

LLMs-from-scratch Visit Over Time

Monthly Visits

503747431

Bounce Rate

37.31%

Page per Visit

5.7

Visit Duration

00:06:44

LLMs-from-scratch Visit Trend

LLMs-from-scratch Visit Geography

LLMs-from-scratch Traffic Sources

LLMs-from-scratch Alternatives

rStar — Enhances problem-solving capabilities of small language models through self-play mutual reasoning.

Programming•Machine Learning•Natural Language Processing

Aixploria — AI tools directory for discovering the best AI tools

Productivity•AI tools•AI navigation

228

RWKV — The new generation of large-scale model architecture, surpassing transformer.

Productivity•Open Source•Deep Learning

114

HippoRAG — A novel RAG framework inspired by human long-term memory

Programming•Natural Language Processing•Knowledge Integration

174

llama3-from-scratch — Implement the Llama3 model from scratch

Programming•Deep Learning•Natural Language Processing

792

RAGFlow — An open-source Retrieval-Augmented Generation (RAG) engine based on deep document understanding

Productivity•Natural language processing•Machine learning

516

llava-llama-3-8b-v1_1 — A LLaVA model optimized by XTuner, which combines image and text processing capabilities.

Programming•Artificial Intelligence•Multimodal Learning

420

EasyContext — EasyContext demonstrates how to leverage existing technologies to train language models with 700K and 1M context lengths.

Programming•Language Models•Context Length

114

nasa-smd-ibm-st — Enhances natural language technologies for information retrieval and intelligent search in the NASA Science Mission Directorate (SMD).

Programming•NASA•Deep Learning

216

LaVi-Bridge — Connects different language models and generative visual models for text-to-image generation

Image•Text-to-Image Generation•Language Models

588

MovieLLM — MovieLLM is an AI-powered film framework designed to enhance long video understanding.

Video•Image Processing•Video Creation

1950

KarpathyLLMChallenge — Deep dive into the tokenization process within language models

Education•Language Models•Tokenization

BASE TTS — Amazon's Large-scale Voice Synthesis Model

Others•Voice Synthesis•Natural Language Processing

1206

ASPIRE — A framework to enhance the selective prediction capability of large language models

Productivity•Artificial Intelligence•Natural Language Processing

DenserBot — Transform data into knowledge

chatting•Chatbot•Large Language Model

150

PromptPoint — A tool for efficiently designing, testing, and deploying prompts.

Productivity•Language Models•Natural Language Processing

210

newsanalyst — An AI news analyst that updates global political, technological, and business news every hour

Productivity•News Analysis•Global Events

126

Claude — Building the future of AI

Productivity•Artificial Intelligence•Deep Learning

708

Quickchat — Let your AI assistant communicate like a human.

Productivity•Generative AI•GPT-3

114

DeepLearning.AI — A platform offering professional courses and resources in the field of AI.

InternationalSelection•Machine Learning•Deep Learning

OneGen — An efficient single-pass unified generation and retrieval framework suitable for large language models.

Programming•Natural Language Processing•Large Language Models

Qwen2.5-LLM — An open-source high-performance language model that supports multi-platform applications.

Productivity•Natural Language Processing•Machine Learning

Qwen2.5 — An open-source large language model that supports multilingual and specialized domain applications.