AQLM

Additive Quantization of Language Models (AQLM) quantizes multiple weights together and takes advantage of interdependencies between them. AQLM represents groups of 8-16 weights as a sum of multiple vector codes.

AQLM also supports fine-tuning with LoRA with the PEFT library, and is fully compatible with torch.compile for even faster inference and training.

Run the command below to install the AQLM library with kernel support for both GPU and CPU inference and training. AQLM only works with Python 3.10+.

pip install aqlm[gpu,cpu]

Load an AQLM-quantized model with from_pretrained().

from transformers import AutoTokenizer, AutoModelForCausalLM

quantized_model = AutoModelForCausalLM.from_pretrained(
    "ISTA-DASLab/Mixtral-8x7b-AQLM-2Bit-1x16-hf",
    dtype="auto", 
    device_map="auto"
)

Configurations

AQLM quantization setups vary mainly in the number of codebooks used, as well as codebook sizes in bits. The most popular setups and supported inference kernels are shown below.

Resources

Run the AQLM demo notebook for more examples of how to quantize a model, push a quantized model to the Hub, and more.

For more example demo notebooks, visit the AQLM repository.