mit-han-lab/streaming-llm

This is stars and forks stats for /mit-han-lab/streaming-llm repository. As of 27 Apr, 2024 this repository has 4561 stars and 234 forks.

Efficient Streaming Language Models with Attention Sinks [paper] streamingllm_demo.mp4 TL;DR We deploy LLMs for infinite-length inputs without sacrificing efficiency and performance. Abstract Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes...