FMInference/FlexGen

This is stars and forks stats for /FMInference/FlexGen repository. As of 08 May, 2024 this repository has 8582 stars and 490 forks.

FlexGen: High-throughput Generative Inference of Large Language Models with a Single GPU [paper] FlexGen is a high-throughput generation engine for running large language models with limited GPU memory. FlexGen allows high-throughput generation by IO-efficient offloading, compression, and large effective batch sizes. Motivation In recent years, large language models (LLMs) have shown great performance across a wide range of tasks. Increasingly, LLMs have been applied not only to interactive applications...