Timo

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place.

Evaluate your LLM's response with Prometheus and GPT4 ?

? Agent-as-a-Judge and DevAI dataset

CodeUltraFeedback: aligning large language models to coding preferences

This is the repo for the survey of Bias and Fairness in IR with LLMs.

Code and data for Koo et al's ACL 2024 paper "Benchmarking Cognitive Biases in Large Language Models as Evaluators"

xVerify: Efficient Answer Verifier for Large Language Model Evaluations

A set of examples demonstrating how to evaluate Generative AI augmented systems using traditional information retrieval and LLM-As-A-Judge validation techniques

LLM-as-judge evals as Semantic Kernel Plugins

Controversial Questions for Argumentation and Retrieval