70 Best Open-Source Observability & Evaluation in 2026 (Ranked by GitHub Activity)

OpenAI-compatible gateway for multi-provider routing, retries, fallbacks, caching, and observability

Open-source Python SDK and AI gateway for calling 100+ LLMs through a unified OpenAI-compatible interface

Open-source RAG engine combining knowledge retrieval and agent capabilities for LLMs

Open-source LLM engineering platform for observability, evaluation, prompt and dataset management

From the team behind Gatsby, Mastra is a framework for building AI-powered applications and agents with a modern TypeScript stack.

Transforms complex documents like PDFs into LLM-ready markdown/JSON for your Agentic workflows.

Fastest enterprise AI gateway (50x faster than LiteLLM) with adaptive load balancer, cluster mode, guardrails, 1000+ models support & <100 µs overhead at 5k RPS.

Open-source CLI and library for evaluating and red-teaming prompts, agents, RAG systems, and LLM apps

AI-powered dashboard for real-time news aggregation, geopolitical monitoring, and infrastructure tracking

Scans AI agent skills for prompt injection, data exfiltration, supply-chain risks, and vulnerabilities

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

The LLM Evaluation Framework

AI Observability & Evaluation

Smart LLM Routing for OpenClaw. Cut Costs up to 70% 🦞🦚

The platform for LLM evaluations and AI agent testing

Audits AI agents against business KPIs and organizational requirements with A–F scorecards

Open-source AI engineering platform for agents, LLMs, and ML models

Open-source cloud-native vector database for semantic search, filtering, RAG, and reranking

Open-source AI orchestration framework for modular RAG pipelines and agent workflows

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

The fastest path to AI-powered full stack observability, even for lean teams.

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

AI Agent Engineering Platform built on an Open Source TypeScript AI Agent Framework

Private AI platform for agents, assistants and enterprise search. Built-in Agent Builder, Deep research, Document analysis, Multi-model support, and API connectivity for agents.

Open-source platform for AI agent tracing, evaluations, guardrails, prompts, and GPU monitoring

Open-source ETL for converting documents into structured data for language models

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Python scraper based on AI

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

Easily fine-tune, evaluate and deploy gpt-oss, Qwen3, DeepSeek-R1, or any open source LLM / VLM!

WFGY is an open-source AI Troubleshooting Atlas for RAG, agents, and real-world AI workflows. Includes the 16-problem map, Global Debug Card, and WFGY 3.0. ⭐ Star to help more builders find this repo.

Harness LLMs with Multi-Agent Programming

the LLM vulnerability scanner

UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection

An open-source visual programming environment for battle-testing prompts to LLMs.

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

An LLM-powered knowledge curation system that researches a topic and generates a full-length report with citations.

Unified framework for building enterprise RAG pipelines with small, specialized models

Supercharge Your LLM Application Evaluations 🚀

The official GitHub page for the survey paper "A Survey on Evaluation of Large Language Models".

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation.

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

Replace 'hub' with 'ingest' in any GitHub URL to get a prompt-friendly extract of a codebase

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

🕹️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.

Everything you need to know to build your own RAG application

Open-source Python library for agentic RAG across public GitHub code, issues, and repository information

Offline-deployable Chinese knowledge base Q&A with RAG and agents using LangChain and open-source LLMs

🤖 Chat with your SQL database 📊. Accurate Text-to-SQL Generation via LLMs using Agentic Retrieval 🔄.

Python SDK for monitoring, cost tracking, benchmarking, and debugging AI agents

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)

Evaluation tool for LLM QA chains

Gorilla: Training and Evaluating LLMs for Function Calls (Tool Calls)

SoTA production-ready AI retrieval system. Agentic Retrieval-Augmented Generation (RAG) with a RESTful API.

Retrieval Augmented Generation (RAG) chatbot powered by Weaviate

Local FastAPI for OCR extraction and PII removal from images, PDFs and Office files to Markdown or JSON

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

Easily compute clip embeddings and build a clip retrieval system with them

This tool has been deprecated. Use Agentic Document Extraction instead.

The open source platform for AI-native application development.

Open-source platform to evaluate and improve generative AI applications with 20+ preconfigured evaluations

Ready-to-deploy templates for RAG and enterprise search that sync with live data sources

Open-source text metrics toolkit for monitoring language models through input and output signals

LLM Comparator is an interactive data visualization tool for evaluating and analyzing LLM responses side-by-side, developed by the PAIR team.

A FastAPI service for semantic text search using precomputed embeddings and advanced similarity measures, with built-in support for various file types through textract.

Retrieval Augmented Generation (RAG) framework and context engine powered by Pinecone

Open source AI Agent evaluation framework for web tasks 🐒🍌

A robust, all-in-one GPT interface for Discord. ChatGPT-style conversations, image generation, AI-moderation, custom indexes/knowledgebase, youtube summarizer, and more!

Chatbot assistant enabling GitHub repository interaction using LLMs with Retrieval Augmented Generation