modelmetry-sdk-python

AI Observability & Evaluation

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including OpenAI Agents SDK, CrewAI, Langchain, Autogen, AG2, and CamelAI

The easiest tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets.

The AI framework that adds the engineering to prompt engineering (Python/TS/Ruby/Java/C#/Rust/Go compatible)

PAIG (Pronounced similar to paige or payj) is an open-source project designed to protect Generative AI (GenAI) applications by ensuring security, safety, and observability.

Fiddler Auditor is a tool to evaluate language models.

ChatGPT API Usage using LangChain, LlamaIndex, Guardrails, AutoGPT and more

Framework for LLM evaluation, guardrails and security

Building blocks for rapid development of GenAI applications

A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).