Noeda/rllama

This is stars and forks stats for /Noeda/rllama repository. As of 26 Apr, 2024 this repository has 464 stars and 25 forks.

RLLaMA RLLaMA is a pure Rust implementation of LLaMA large language model inference.. Supported features Uses either f16 and f32 weights. LLaMA-7B, LLaMA-13B, LLaMA-30B, LLaMA-65B all confirmed working Hand-optimized AVX2 implementation OpenCL support for GPU inference. Load model only partially to GPU with --percentage-to-gpu command line switch to run hybrid-GPU-CPU inference. Simple HTTP API support, with the possibility of doing token sampling on client side It can load Vicuna-13B instruct-finetuned...