Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ncnn is a high-performance neural network inference framework optimized for the mobile platform

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models