NVIDIA/Megatron-LM

This is stars and forks stats for /NVIDIA/Megatron-LM repository. As of 29 Apr, 2024 this repository has 6500 stars and 1432 forks.

Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA. This repository is for ongoing research on training large transformer language models at scale. We developed efficient, model-parallel (tensor, sequence, and pipeline), and multi-node pre-training of transformer based models such as GPT, BERT, and T5 using mixed precision. Below are some of the projects where we have directly used Megatron: BERT and GPT Studies Using Megatron BioMegatron:...