Popular GitHub Repositories for Scraping

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Lightweight library for scraping web-sites with LLMs

AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.

Tools to build web AI agents that can authenticate, interact with and extract data from any website.

Face recognition system using MTCNN, FACENET, SVM and FAST API to track participants of Big Brother Brasil in real time.

Turn Webpage to LLM friendly input text. Similar to Firecrawl and Jina Reader API. Makes RAG, AI web scraping, image & webpage links extraction easy.

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels.

Automate your LinkedIn job applications with AI! This bot utilizes GPT models such as GPT-4, GPT-3.5, and Google's Gemini Pro for Easy Apply form filling, customizable to your preferences and job search criteria.

AI tool for automating Upwork job applications using AI agents to find and qualify jobs, write personalized cover letters, and prepare for interviews based on your skills and experience.

Web scraping API for building AI applications.

Image scraping library for creating deep learning datasets

Automated web scraping spider generation using Browser Use and LLMs. Streamline the creation of Playwright-based spiders with minimal manual coding. Ideal for large enterprises with recurring data extraction needs.

Download Midjourney Showcase images automatically

Model Context Protocol server that integrates AgentQL's data extraction capabilities.

An automatically updated feed with summaries of the best Habr.com articles generated by the YandexGPT neural network.

A user-friendly python-based GUI which provides sentiment analysis of users' reviews toward a specific TripAdvisor facility

A simple library designed for rapid bot and ai scraper development using Selenium, the POM (Page Object Model) design and llm data extraction.

scraping and querying documents for LLMs

ScrapeGraph client langchain integration

Analyzing Google Play Store reviews of the Indonesian streaming platform Vidio through topic modeling with the assistance of GPT.