Popular GitHub Repositories for Llm Evaluation Framework

The LLM Evaluation Framework

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

Develop reliable AI apps

[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts

Realign is a testing and simulation framework for AI applications.

Code for "Prediction-Powered Ranking of Large Language Models", NeurIPS 2024.

Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.

TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

Community Plugin for Genkit to use Promptfoo

Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.

Sample project demonstrates how to use Promptfoo, a test framework for evaluating the output of generative AI models

Sample implementation demonstrating how to use Firebase Genkit with Promptfoo