Guitaricet/peft_pretraining

This is stars and forks stats for /Guitaricet/peft_pretraining repository. As of 03 May, 2024 this repository has 279 stars and 23 forks.

ReLoRA -- PEFT Pretraining Official code for Stack More Layers Differently: High-Rank Training Through Low-Rank Updates https://arxiv.org/abs/2307.05695 Setup All requirements are listed in requirements.txt and kept up-to-date. cd peft_pretraining pip install -r requirements.txt Usage To train a model using ReLoRA, first, perform a warmup through regular training. Train language model with PEFT torchrun --nproc-per-node <N_GPUS> torchrun_main.py \ --model_config configs/llama_250m.json...