AIbase
Product LibraryTool Navigation

PyTorch-Scratch-LLM

Public

Simple and easy to understand PyTorch implementation of Large Language Model (LLM) GPT and LLAMA from scratch with detailed steps. Implemented: Byte-Pair Tokenizer, Rotational Positional Embedding (RoPe), SwishGLU, RMSNorm, Mixture of Experts (MOE). Tested on Taylor Swift song lyrics dataset.

Creat2024-10-30T08:47:53
Update2025-01-14T04:19:23
3
Stars
0
Stars Increase