Data-SUITE

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..

A Doctor for your data

Resources for Data Centric AI

Deita: Data-Efficient Instruction Tuning for Alignment [ICLR2024]

Lab assignments for Introduction to Data-Centric AI, MIT IAP 2024 ????

[NeurIPS 2023] This is the code for the paper `Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias`.

DataCLUE: 数据为中心的NLP基准和工具包

OpenDataVal: a Unified Benchmark for Data Valuation in Python (NeurIPS 2023)

Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning

nbsynthetic is simple and robust tabular synthetic data generation library for small and medium size datasets