Popular GitHub Repositories for Eval

A curated list of Human Preference Datasets for LLM fine-tuning, RLHF, and eval.

Open Source Embeddings Optimisation and Eval Framework for RAG/LLM Applications. Documentations at https://docs.vectorboard.ai/introduction

Multi-Agent Step Race Benchmark: Assessing LLM Collaboration and Deception Under Pressure. A multi-player “step-race” that challenges LLMs to engage in public conversation before secretly picking a move (1, 3, or 5 steps). Whenever two or more players choose the same number, all colliding players fail to advance.

Deep learning based speech and pronunciation assessment API for 8 languages.

A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).

Public Goods Game (PGG) Benchmark: Contribute & Punish is a multi-agent benchmark that tests cooperative and self-interested strategies among Large Language Models (LLMs) in a resource-sharing economic scenario. Our experiment extends the classic PGG with a punishment phase, allowing players to penalize free-riders or retaliate against others.

Automation Framework using LLM-as-a-judge to Scale Eval of Gen AI solutions (RAG, Multi-turn, Query Rewrite, Text2SQL etc.); that is a good proxy for human judgement.

AgentTrace is a lightweight observability library to trace and evaluate agentic systems.

Go Artificial Intelligence (GAI) helps you work with foundational models, large language models, and other AI models.

EvalBench is a flexible framework designed to measure the quality of generative AI (GenAI) workflows around database specific tasks.

Evaluation framework for LLMs and prompts on real world coding tasks in JavaScript, Python and SQL

An open-source guide to LLMOps, blending theory and practice for deploying Large Language Models effectively.

Benchmarking modern neural networks for scRNA sequencing analysis