Deployment and Serving

AirLLM is an open-source tool designed to optimize memory usage during large language model (LLM) inference, enabli…

AITemplate is a Python framework that converts deep neural networks into optimized CUDA or HIP C++ code for high-pe…

BentoML is an open-source framework designed for packaging, deploying, and serving machine learning models in produ…

BISHENG is an open-source LLM application devops platform designed to streamline the deployment and management of l…

DeepDetect is a machine learning server designed for deploying and serving models in production environments. It su…

NVIDIA Dynamo is a high-performance inference framework optimized for deploying and serving generative AI and reaso…

exo is an open-source tool designed to simplify the deployment and management of AI clusters using everyday consume…

Genkit is an open-source framework designed to streamline the development and deployment of AI-powered applications…

Inference is a high-performance inference server designed for deploying computer vision models, supporting popular …

IPEX-LLM is a PyTorch-based library optimized for running large language models (LLMs) efficiently on Intel CPUs an…

Jina-serve is an open-source framework for building and deploying AI services with support for multiple communicati…

KServe is a Kubernetes Custom Resource Definition for serving predictive and generative machine learning models. It…

KTransformers is an open-source framework designed to optimize and streamline large language model (LLM) inference,…

Langtrace is an open-source observability tool designed for LLM applications, leveraging Open Telemetry to provide …

Lepton AI is a Python library designed to simplify the deployment and serving of AI models by converting Python cod…

LightLLM is a Python-based framework designed for efficient inference and serving of large language models (LLMs), …

LiteLLM is a Python SDK and proxy server that standardizes interactions with over 100 LLM APIs by converting them i…

LiteRT is a lightweight runtime optimized for deploying machine learning models on resource-constrained devices lik…

LiteRT-LM is a lightweight, cross-platform inference framework designed for efficiently running Large Language Mode…

LitServe is an open-source AI model serving engine built on FastAPI, designed to deploy and manage custom inference…

Llmfit is a terminal tool that optimizes LLM models for a system's hardware configuration. It detects the system's …

LMCache is a high-performance key-value (KV) cache layer designed to optimize and accelerate large language model (…

LocalAI is a self-hosted, open-source alternative to the OpenAI API, providing a REST API for running AI models loc…

MindsDB is an open-source platform that integrates machine learning models directly into databases, enabling users …

mini-sglang is a lightweight framework designed for efficient deployment and serving of large language models (LLMs…

MLRun is an open-source MLOps framework designed to streamline the deployment and management of machine learning wo…

MLServer is an open-source inference server designed for deploying and serving machine learning models, supporting …

Model Runner is a Docker-based tool for deploying and serving AI models, including LLMs, by containerizing them for…

Mosec is a Rust-powered, multi-stage pipelined model server designed for efficient AI model deployment. It supports…

nano-vllm is a lightweight, optimized implementation of vLLM designed for fast offline inference. It leverages tech…

nndeploy is an open-source framework designed to simplify the deployment of AI models by providing a high-performan…

Nuclio is a high-performance serverless platform designed for deploying and managing data, I/O, and compute-intensi…

OpenVINO (Open Visual Inference and Neural Network Optimization) is an open-source toolkit designed to optimize and…

OptiLLM is an open-source, OpenAI API-compatible inference proxy that optimizes LLM performance using 20+ advanced …

PowerInfer is a high-efficiency inference engine for large language models (LLMs) that optimizes CPU/GPU resource u…

Prompt2Model is an open-source system that converts natural language task descriptions into small, specialized AI m…

RamaLama is an open-source tool designed to streamline the deployment and serving of AI models locally using OCI co…

RunAnywhere provides SDKs for deploying AI models (LLMs, speech-to-text, text-to-speech) on mobile devices, support…

Seldon Core is an open-source platform designed to deploy, serve, and scale machine learning models in Kubernetes e…

SIE is an open-source inference server and production cluster designed for deploying AI models focused on embedding…

text-generation-inference is an open-source tool designed for efficient deployment and serving of large language mo…

TorchServe is a model serving framework designed specifically for deploying PyTorch models at scale. It provides a …

torchtune is a PyTorch library designed to simplify the authoring, fine-tuning, and experimentation of large langua…

Transformer Lab is an open-source workspace designed for locally fine-tuning, evaluating, exporting, and testing la…

Triton Inference Server is a scalable AI model serving platform that supports multiple frameworks (TensorFlow, PyTo…

Vercel AI is a TypeScript toolkit that simplifies the integration of AI models into web applications, supporting fr…

Vespa is an open-source platform for building scalable, real-time applications that perform search, ranking, and in…