huggingface/text-generation-inference

This is stars and forks stats for /huggingface/text-generation-inference repository. As of 30 Apr, 2024 this repository has 5391 stars and 565 forks.

Text Generation Inference A Rust, Python and gRPC server for text generation inference. Used in production at HuggingFace to power Hugging Chat, the Inference API and Inference Endpoint. Table of contents Get Started API Documentation Using a private or gated model A note on Shared Memory Distributed Tracing Local Install CUDA Kernels Optimized architectures Run Falcon Run Quantization Develop Testing Text Generation Inference (TGI) is a toolkit for deploying and serving Large Language Models...