Popular GitHub Repositories for Llm Benchmarking

Examining how large language models (LLMs) perform across various synthetic regression tasks when given (input, output) examples in their context, without any parameter update

Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

A comprehensive guide to LLM evaluation methods designed to assist in identifying the most suitable evaluation techniques for various use cases, promote the adoption of best practices in LLM assessment, and critically assess the effectiveness of these evaluation methods.

A benchmark for prompt injection detection systems.

A collection of LLM related papers, thesis, tools, datasets, courses, open source models, benchmarks

LLM-KG-Bench is a Framework and task collection for automated benchmarking of Large Language Models (LLMs) on Knowledge Graph (KG) related tasks.

Program synthesis for 3D spatial reasoning

Benchmark evaluating LLMs on their ability to create and resist disinformation. Includes comprehensive testing across major models (Claude, GPT-4, Gemini, Llama, etc.) with standardized evaluation metrics.

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts

A comprehensive code domain benchmark review of LLM researches.

This is a series of Python scripts for zero-shot and chain-of-thought LLM scripting

PARROT (Performance Assessment of Reasoning and Responses On Trivia) is a novel benchmarking framework designed to evaluate Large Language Models (LLMs) on real-world, complex, and ambiguous QA tasks.

Framework to benchmark LLMs performance on domain categorization, done as part of my internship at iQ Global.

LLM Serving Libs Benchmark

Benchmark LLMs' abilities to plan, strategize, and reason by making them play chess against each other.