JonasGeiping/cramming

This is stars and forks stats for /JonasGeiping/cramming repository. As of 07 May, 2024 this repository has 1144 stars and 87 forks.

Cramming Language Model (Pretraining) This repository contains code to replicate our research described in "Cramming: Training a Language Model on a Single GPU in One Day". We experiment with language model pretraining a BERT-type model with limited compute, wondering "how bad can it really be"? You can find our paper here: https://arxiv.org/abs/2212.14034, and the abstract below: Recent trends in language modeling have focused on increasing performance through scaling, and have resulted in an environment...