kyegomez/LongNet

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"

Jupyter NotebookPythonmachine-learningartificial-intelligencetransformerattentionattention-is-all-you-needattention-mechanismsgpt3gpt4chatgptcontext-length
This is stars and forks stats for /kyegomez/LongNet repository. As of 06 May, 2024 this repository has 586 stars and 77 forks.

LongNet: Scaling Transformers to 1,000,000,000 Tokens This is an open source implementation for the paper LongNet: Scaling Transformers to 1,000,000,000 Tokens by Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei. The LongNet is a Transformer variant designed to scale sequence length up to more than 1 billion tokens without sacrificing performance on shorter sequences. News 📰 This implementation of LongNet is brought to you by Agora, we're an all-new open source...
Read on GithubGithub Stats Page
repotechsstarsweeklyforksweekly
khoj-ai/khojPythonHTMLEmacs Lisp3.8k01620
Paulescu/hands-on-train-and-deploy-mlPythonMakefileShell4800760
tonquer/JMComic-qtPython9270610
andrewmcgr/klipper_tmc_autotunePythonShell1740410
cucapra/filamentVerilogRustPython56040
CoretechR/OMOTECPythonC++7260880
facebook/redexC++JavaPython5.9k+36770
SWMFsoftware/utilFortranTeXIDL0000
SWMFsoftware/shareFortranPythonIDL0000
cbc02009/k8s-home-opsHCLShellPython6020