Popular GitHub Repositories for Data Pipeline

A Clojure machine learning library

Tools for ASR Corpus Generation from Online Video

Backtesting an algorithmic trading strategy using Machine Learning and Sentiment Analysis.

Splicing: Gen-AI Copilot for Data Engineering

Time Series Anomaly detection. The monitored signal is made-up of machinery vibration sensor measurements.

An Airflow-based pipeline that fetches data from a free API, cleans and transforms it, and saves it to a database—ready for downstream machine learning.

Sync your team's data to your LLM applications in real-time

End-To-End Data Engineering Project. Made to learn some common data engineering practices.

基于Docker Compose的Apache Airflow + Spark ETL流程：OpenWeather数据导入PostgreSQL数据库

Pipeline-Genie is an intelligent data pipeline that processes CSV datasets, identifies their schema, and leverages LLaMA 2.0 to extract business insights. Users can select relevant business needs, triggering automated ETL transformations using Apache Spark. The final transformed dataset is stored in AWS S3 and made available for download.

Generate lyrics with TensorFlow and the Genius API

Churn Prediction with Kedro, Kedro-Viz, and Kedro-Mlflow ?? ?. PowerBI Dashboard ? also included. Kedro? https://kedro.org/

Cryptocurrency prediction using LSTM (Long Short Term Memory) [ Hugging Face: https://huggingface.co/spaces/qywok/cryptocurrency_prediction ]

Latency Estimation for Neural Network Architecture

Vector Database testing for semantic search capabilities for use in LLM applications

A pipeline that collects, preprocesses and labels raw user-item interaction data to build a recsys with collaborative filtering (SVD, ALS) and LTR (XGBoost ranking) methods

An end‑to‑end real-time analytics & anomaly detection with PySpark Structured Streaming on user activity logs from Kafka

数据工程项目 - 自动化批量数据处理

A cloud-based deployment and scaling for ML services (Docker, k8s, AWS S3/Lambda, Flask, GitHub Actions)

This repository contains the code, documentation, and datasets for a comprehensive exploration of machine learning techniques to address class imbalance. The project investigates the impact of various methods, like ADASYN, KMeansSMOTE, and Deep Learning Generator, on classification performance while effectively demonstrating benefits of pipelining.