LLM Compressor

Overview

LLM Compressor is an open-source library for optimizing language models for deployment with vLLM. It provides quantization algorithms for weights, activations, KV cache and attention, integrates with Hugging Face models, and saves checkpoints in the compressed-tensors format. It supports distributed data parallel and disk offloading for compressing very large models. The project is maintained by Red Hat AI and the vLLM project.

Use cases

• Quantize LLMs for lower memory use
• Compress KV cache and activations
• Produce checkpoints for vLLM
• Reduce inference hardware requirements

Sources

• LLM Compressor README
• LLM Compressor Apache 2.0 licence