Popular GitHub Repositories for Evaluation Framework

A framework for few-shot evaluation of language models.

The LLM Evaluation Framework

Data-Driven Evaluation for LLM-Powered Applications

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

Python SDK for running evaluations on LLM generated responses

DevQualityEval: An evaluation benchmark ? and framework to compare and evolve the quality of code generation of LLMs.

Test and evaluate LLMs and model configurations, across all the scenarios that matter for your application

Evaluation suite for large-scale language models.

Multilingual Large Language Models Evaluation Benchmark

Official repository for the paper “The Dawn of Natural Language to SQL: Are We Fully Ready?” (VLDB'24)

This repository allows you to evaluate a trained computer vision model and get general information and evaluation metrics with little configuration.

The implementation of the paper "Evaluating Coherence in Dialogue Systems using Entailment"

The official implementation of ECCV'24 paper "To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now". This work introduces one fast and effective attack method to evaluate the harmful-content generation ability of safety-driven unlearned diffusion models.

RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models. NeurIPS 2024

OD-test: A Less Biased Evaluation of Out-of-Distribution (Outlier) Detectors (PyTorch)

WritingBench: A Comprehensive Benchmark for Generative Writing

Python client for Kolena's machine learning testing platform

Scalable Meta-Evaluation of LLMs as Evaluators

Official Implementation of ACL2024 paper "Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs"(https://aclanthology.org/2024.findings-acl.168)