Popular GitHub Repositories for Evals

AI Observability & Evaluation

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including OpenAI Agents SDK, CrewAI, Langchain, Autogen, AG2, and CamelAI

The easiest tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets.

A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).

Go Artificial Intelligence (GAI) helps you work with foundational models, large language models, and other AI models.

An implementation of the Anthropic's paper and essay on "A statistical approach to model evaluations"

AI Agents play The Mind card game

Develop better LLM apps by testing different models and prompts in bulk.

The Modelmetry Python SDK allows developers to easily integrate Modelmetry’s advanced guardrails and monitoring capabilities into their LLM-powered applications.

dspy with ollama and llamacpp on google colab

Get started with LLMs, FTS and vector search, RAG, and more, in Go!