Popular GitHub Repositories for Benchmarking

An adversarial example library for constructing attacks, building defenses, and benchmarking both

Merlion: A Machine Learning Framework for Time Series Intelligence

AutoRAG: An Open-Source Framework for Retrieval-Augmented Generation (RAG) Evaluation & Optimization with AutoML-Style Automation

Unifying Variational Autoencoder (VAE) implementations in Pytorch (NeurIPS 2022)

A Toolbox for Adversarial Robustness Research

Unsupervised time series anomaly detection library

[NeurIPS'21 Outstanding Paper] Library for reliable evaluation on RL and ML benchmarks, even with only a handful of seeds.

A Pytorch Knowledge Distillation library for benchmarking and extending works in the domains of Knowledge Distillation, Pruning, and Quantization.

LogAI - An open-source library for log analytics and intelligence

[PVLDB 2024 Best Paper Nomination] TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

SpeechIO Leaderboard: a large, robust, comprehensive, benchmarking platform for Automatic Speech Recognition.

A library for ML benchmarking. It's powerful.

A PyTorch reinforcement learning library for generalizable and reproducible algorithm implementations with an aim to improve accessibility in RL

Bayesian Optimization as a Coverage Tool for Evaluating LLMs. Accurate evaluation (benchmarking) that's 10 times faster with just a few lines of modular code.

?? Flex those feathers!

A mini-framework for evaluating LLM performance on the Bulls and Cows number guessing game, supporting multiple LLM providers.

GERBIL - General Entity annotatoR Benchmark

A python package for benchmarking interpretability techniques on Transformers.

GraphXAI: Resource to support the development and evaluation of GNN explainers

Time series anomaly detection algorithm implementations for TimeEval (Docker-based)