Popular GitHub Repositories for Webcrawler

A Web Crawler based on LLMs implemented with Ray and Huggingface. The embeddings are saved into a vector database for fast clustering and retrieval. Use it for your RAG.

:sparkles: :dna: Turing ES - Enterprise Search, Semantic Navigation, Chatbot using Search Engine and Generative AI.

CodeBRT is an AI program generation plugin for VSCode. It helps you quickly generate code through AI, thus improving development efficiency.

Open Source Deep Research

Predict the BitCoin's currency by using Altcoin currency and Web News.

RAG and Webcrawler in a single package

An ai agent that retrieves data from an islamic website [in Turkish language] -> processes the data -> utilizes an open ai NLP to answer the user

DevCrawler - An LLM Friendly Web Crawler & Data Scraper