Evaluation and Monitoring

AlpacaEval is an open-source tool designed to automatically evaluate the performance of instruction-following langu…

ANN-Benchmarks is a standardized benchmarking framework for evaluating and comparing the performance of approximate…

ARES is a framework designed for automated evaluation of Retrieval-Augmented Generation (RAG) models, focusing on p…

BEIR is a benchmark and evaluation framework for information retrieval (IR) tasks, designed to assess the performan…

C-Eval is a benchmarking tool designed to evaluate the performance of foundation models on Chinese language tasks. …

The Code Generation LM Evaluation Harness is a framework designed to systematically evaluate the performance of cod…

COMET is an open-source framework designed for evaluating machine translation models by comparing predictions again…

Deepchecks is an open-source Python library designed for validating machine learning models and data integrity thro…

DeepEval is an open-source framework designed for evaluating large language model (LLM) applications, offering tool…

DomainBed is a benchmark suite designed for evaluating domain generalization algorithms in machine learning. It pro…

EvalAI is an open-source platform designed to facilitate large-scale evaluation of AI models by providing a standar…

Evalchemy provides a standardized framework for evaluating the performance of post-trained language models, offerin…

EvalPlus is a specialized evaluation framework designed for assessing large language models (LLMs) in code generati…

Evals is a framework developed by OpenAI for systematically evaluating the performance of AI models, particularly t…

EvalScope is an open-source framework designed for evaluating and benchmarking large AI models. It provides customi…

Evaluate is an open-source library designed to streamline the evaluation and comparison of machine learning models …

Future AGI is an open-source platform designed for engineering and optimizing AI agents, integrating tools for trac…

GAOKAO-Bench is an open-source evaluation framework that leverages questions from the Chinese National College Entr…

guidellm is a benchmarking tool designed to evaluate the performance of large language model (LLM) inference system…

Helicone is an open-source platform designed to help developers monitor, evaluate, and optimize large language mode…

HumanEval is a benchmark tool designed to assess the functional correctness of AI-generated code, specifically for …

Inspect is an open-source framework designed for evaluating large language models (LLMs), providing tools to assess…

JiWER is a Python library designed to compute Word Error Rate (WER) and other speech recognition accuracy metrics e…

Laminar is an open-source platform designed to trace, evaluate, label, and analyze large language model (LLM) data …

LangTest is an open-source toolkit designed for evaluating and monitoring the performance of natural language proce…

The Language Model Evaluation Harness is a standardized framework for evaluating the performance of generative lang…

LLMPerf is a benchmarking tool designed to measure and compare the performance of large language model (LLM) APIs, …

lmms-eval is an open-source evaluation framework designed for consistent and efficient assessment of Large Multimod…

Massive Text Embedding Benchmark (MTEB) is a standardized evaluation framework that measures the performance of tex…

Melting Pot is a suite of customizable test scenarios designed for evaluating multi-agent reinforcement learning (M…

Meta-World is an open-source benchmark environment designed for evaluating meta-reinforcement learning and multi-ta…

mir_eval is a Python library designed for evaluating Music Information Retrieval (MIR) systems by providing standar…

MLPerf Inference is a standardized benchmark suite designed to evaluate the performance of AI systems when running …

NannyML is an open-source Python library that helps monitor machine learning models in production by estimating the…

OGB (Open Graph Benchmark) provides standardized datasets, data loaders, and evaluation tools for graph machine lea…

Ollama Grid Search automates the evaluation of AI models, prompts, and inference parameters by systematically itera…

OpenCompass is an open-source platform for evaluating large language models (LLMs) across multiple datasets and mod…

Overcooked-AI is a benchmark environment designed to evaluate and monitor cooperative task performance between huma…

Prometheus-Eval is an open-source AI evaluation and monitoring tool designed to assess and optimize large language …

PromptBench is a unified evaluation framework designed to assess and benchmark the performance of large language mo…

RagaAI Catalyst provides tools for training, evaluating, and deploying language models specialized in assessing the…

RewardBench is a benchmarking tool developed to assess the performance and safety of reward models in AI systems. I…

RLBench is a large-scale benchmark and learning environment for vision-guided manipulation research, supporting tas…

SimplerEnv provides simulated environments for evaluating manipulation policies in robotics, enabling testing and r…

Speech-to-Text Benchmark is an open-source framework designed to evaluate and compare the performance of different …

SwanLab is an open-source tool designed for tracking and visualizing AI model training metrics and experiments. It …

TorchBench is a suite of benchmarks designed to evaluate the performance of PyTorch models across different hardwar…

TruLens is an open-source framework for evaluating and monitoring large language model (LLM) experiments. It provid…

TrustLLM is an open-source framework designed to assess the trustworthiness of large language models (LLMs) through…

VBench is an open-source benchmark suite designed for evaluating video generative models. It provides standardized …

VLMEvalKit is an open-source toolkit designed for evaluating large vision-language models (LVLMs), providing standa…