mit-han-lab/llm-awq

This is stars and forks stats for /mit-han-lab/llm-awq repository. As of 10 May, 2024 this repository has 893 stars and 66 forks.

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [Paper] Efficient and accurate low-bit weight quantization (INT3/4) for LLMs, supporting instruction-tuned models and multi-modal LMs. The current release supports: AWQ search for accurate quantization. Pre-computed AWQ model zoo for LLMs (LLaMA-1&2, OPT, Vicuna, LLaVA; load to generate quantized weights). Memory-efficient 4-bit Linear in PyTorch. Efficient CUDA kernel implementation for fast inference (support context...