bitsandbytes

Overview

bitsandbytes enables accessible large language models through k-bit quantization for PyTorch, with three main features for reducing memory consumption. Its 8-bit optimizers use block-wise quantization to maintain 32-bit performance at a small fraction of the memory cost; LLM.int8() enables 8-bit language model inference with only half the required memory and without performance degradation by quantizing most features to 8 bits and treating outliers with 16-bit matrix multiplication; and QLoRA enables 4-bit model training by inserting a small set of trainable low-rank adaptation weights. The library provides quantization primitives through Linear8bitLt and Linear4bit and 8-bit optimizers through its optim module, supports Linux, Windows and macOS on CPUs and NVIDIA, AMD and Intel accelerators, and is MIT licensed.

Use cases

• Halve memory use for LLM inference with LLM.int8()
• Fine-tune models in 4-bit with QLoRA
• Cut training memory with 8-bit optimizers
• Add 4-bit and 8-bit quantization to PyTorch models

Sources

• bitsandbytes repository