Popular GitHub Repositories for Observability

Stunning status pages, batteries included!

?️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.

Easy token price estimates for 400+ LLMs. TokenOps.

The ultimate LLM Ops platform - Monitoring, Analytics, Evaluations, Datasets and Prompt Optimization ?

The production toolkit for LLMs. Observability, prompt management and evaluations.

Distribute and run AI workloads magically in Python, like PyTorch for ML infra.

? LangKit: An open-source toolkit for monitoring Large Language Models (LLMs). ? Extracts signals from prompts & responses, ensuring safety & security. ?? Features include text quality, relevance metrics, & sentiment analysis. ? A comprehensive tool for LLM observability. ?

Langtrace ? is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. ???

A collection of production-ready Generative AI Agent templates built for Google Cloud. It accelerates development by providing a holistic, production-ready solution, addressing common challenges (Deployment & Operations, Evaluation, Customization, Observability) in building and deploying GenAI agents.

LangSmith Client SDK Implementations

Sister project to OpenLLMetry, but in Typescript. Open-source observability for your LLM application, based on OpenTelemetry

Markdown for the AI era

Foremast adds application resiliency to Kubernetes by leveraging machine learnt patterns of application health to keep applications healthy and stable

? Langfuse documentation -- Langfuse is the open source LLM Engineering Platform. Observability, evals, prompt management, playground and metrics to debug and improve LLM apps

Create custom DevOps AI agents that understand and manage your infrastructure.

Create, List, Update, Delete Amazon EKS clusters. Deploy and manage software on EKS. Run distributed model training and inference examples.

Jupyter notebooks for course Building and Evaluating Advanced RAG Applications, taught by Jerry Liu (Co-founder and CEO of LlamaIndex) and Anupam Datta (Co-founder and chief scientist of TruEra).

? Langfuse JS/TS SDKs - Instrument your LLM app and get detailed tracing/observability. Works with any LLM or framework

Analyzing and scoring reasoning traces of LLMs

Mixpanel for conversational AI