Popular GitHub Repositories for Tokenizers

Building applications with LLMs through composability, in Kotlin

Develop DL models using Pytorch and Hugging Face

Multi-arch bindings for @huggingface/tokenizers.

Visualize some important concepts related to LLM architectures.

Question and Answer web applicaiton using fine-tuned and pre-trained T5 models. Application runs on Streamlit.

NLP Dataset Creation and Semantic Search Demonstration

Create prompts with a given token length for testing LLMs and other transformers text models.