webpalm

Scrapy, a fast high-level web crawling & scraping framework for Python.

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

LlamaIndex is the leading framework for building LLM-powered agents over your data.

? SheetJS Spreadsheet Data Toolkit -- New home https://git.sheetjs.com/SheetJS/sheetjs

Elegant Scraper and Crawler Framework for Golang

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

Chat with your database or your datalake (SQL, CSV, parquet). PandasAI makes data analysis conversational using LLMs and RAG.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

A curated list of awesome big data frameworks, ressources and other awesomeness.

A scalable web crawler framework for Java.