Crawl4LLM

An efficient web crawler for LLM pre-training, focused on crawling high-quality web data effectively.

CommonProductProgrammingLLMWeb Crawler

Crawl4LLM is an open-source web crawling project designed to provide an efficient data crawling solution for the pre-training of Large Language Models (LLMs). It helps researchers and developers obtain high-quality training corpora through intelligent selection and crawling of web data. The tool supports various document scoring methods and allows flexible adjustment of crawling strategies based on configurations to meet different pre-training needs. Developed in Python, the project boasts good scalability and ease of use, making it suitable for both academic research and industrial applications.

Best AI Websites & Tools

Crawl4LLM

Crawl4LLM Visit Over Time

Crawl4LLM Visit Trend

Crawl4LLM Visit Geography

Crawl4LLM Traffic Sources

Crawl4LLM Alternatives

langroid — Langroid is a lightweight LLM framework based on Python

DocWrangler — An open-source interactive development environment for building and optimizing LLM-based data processing pipelines.

AI-Data-Analysis-MultiAgent — AI-Driven Multi-Agent Data Analysis System

marimo — Next-Gen Python Notebook

browser-use — An open-source web automation library that supports any large language model (LLM)

Laminar.ai — An open-source full-stack platform that provides support for building top-tier LLM products.

ComfyUI-MochiWrapper — ComfyUI wrapper for the Mochi video generator

promptic — A simple yet powerful Python library for using large language models (LLMs).

Data-Juicer — A one-stop data processing system that provides high-quality data for large language models.

SuperCoder 2.0 — Open Source Autonomous Software Development System

Composio — A toolkit for AI agents enabling complex task handling.

RAGElo — Elo ranking tool for RAG-based LLM agents

June — A local voice chatbot that protects privacy and does not require internet access.

AI Math Notes — An interactive graphing application for drawing and calculating mathematical equations.

llama-recipes — A collection of libraries and example scripts for fine-tuning the Meta Llama model.

X-Force IDE — Create agent workgroups through drag-and-drop to customize business logic.

Confident AI — Open-source evaluation infrastructure that provides confidence for LLMs.

llamafile — Pack an LLM model into an executable file

Cline.bot — Cline is a collaborative AI programming partner for VSCode, helping developers develop efficiently.

The Ultra-Scale Playbook — A tool focused on ultra-scale system design and optimization, providing efficient solutions.

hallucination-leaderboard — A leaderboard for comparing the hallucination rates of large language models when summarizing short documents.

Fluent Reader — An AI-powered browser bilingual translation extension that provides smart translation and privacy protection.

SkyReels-V1-Hunyuan-I2V — SkyReels V1 is an open-source, human-centric video foundation model focused on high-quality, cinematic video generation.

KET-RAG — KET-RAG is a retrieval-augmented generation framework that incorporates knowledge graphs for efficient document indexing and answer generation.

Goedel-Prover — Goedel-Prover is an open-source automated theorem proving model focused on the formal verification of mathematical problems.

Hacker News Daily Broadcast — An AI-powered project that delivers a daily Chinese podcast summarizing Hacker News. It automatically crawls popular articles from Hacker News and generates Chinese summaries.

VisionAgent — VisionAgent is a library for generating code to solve vision tasks, supporting multiple LLM providers.

Prototype — A template for quickly starting a Django project with OpenAI integration.

OmniParser-v2.0 — OmniParser is a versatile screen parsing tool that converts UI screenshots into a structured format, improving the performance of LLM-based UI agents.