Popular GitHub Repositories for Evaluation Framework

A framework for few-shot evaluation of language models.

The LLM Evaluation Framework

Data-Driven Evaluation for LLM-Powered Applications

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

Python SDK for running evaluations on LLM generated responses

DevQualityEval: An evaluation benchmark ? and framework to compare and evolve the quality of code generation of LLMs.

Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application

Evaluation suite for large-scale language models.

Multilingual Large Language Models Evaluation Benchmark

Official repository for the paper “The Dawn of Natural Language to SQL: Are We Fully Ready?” (VLDB'24)

This repository allows you to evaluate a trained computer vision model and get general information and evaluation metrics with little configuration.

The implementation of the paper "Evaluating Coherence in Dialogue Systems using Entailment"

The official implementation of ECCV'24 paper "To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now". This work introduces one fast and effective attack method to evaluate the harmful-content generation ability of safety-driven unlearned diffusion models.

RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024

WritingBench: A Comprehensive Benchmark for Generative Writing

OD-test: A Less Biased Evaluation of Out-of-Distribution (Outlier) Detectors (PyTorch)

Python client for Kolena's machine learning testing platform

A browser-based 3D multiplayer flying game with arcade-style mechanics, created using the Gemini 2.5 Pro through a technique called "vibe coding"

Scalable Meta-Evaluation of LLMs as Evaluators