ONNX Runtime

Overview

ONNX Runtime is a cross-platform inference and training accelerator. Its inference runtime runs models exported from PyTorch, TensorFlow/Keras, scikit-learn, LightGBM and XGBoost, applies graph optimizations and uses hardware accelerators through execution providers including CUDA, TensorRT, OpenVINO, DirectML and CoreML. For model efficiency the project documents Python quantization APIs for 8-bit dynamic and static quantization in QOperator or QDQ format, calibration methods, quantization debugging, and block-wise weight-only Int4/UInt4 quantization for MatMul and Gather operators with RTN, HQQ and GPTQ algorithms. The project is MIT licensed.

Editorial verdict

Fits teams that need one inference runtime across servers, desktops and browsers, with execution providers spanning CUDA, TensorRT, OpenVINO, DirectML and CoreML and documented 8-bit and 4-bit quantization APIs. The trade-off is that it optimises execution rather than measuring it: no energy or carbon figures are produced, quantization accuracy needs validating per model, and the provider and hardware pairing determines how much efficiency is actually gained. Converting models to ONNX adds a build step. Teams wanting a serving stack with batching should look at a dedicated inference server.

Written by the GreenAIX editors from the recorded fields above. Vendors do not review or approve verdicts.

Use cases

• Run inference on models exported from PyTorch or TensorFlow
• Quantise models to Int8 or Int4 to cut memory and compute
• Accelerate inference with execution providers
• Optimise computation graphs for deployment

Sources

• ONNX Runtime homepage
• ONNX Runtime quantization documentation
• ONNX Runtime README
• ONNX Runtime MIT licence