AutoCrawler

Scrapy, a fast high-level web crawling & scraping framework for Python.

Elegant Scraper and Crawler Framework for Golang

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

大数据入门指南 :star:

A curated list of awesome big data frameworks, ressources and other awesomeness.

A scalable web crawler framework for Java.

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...