nikopj/FlashAttention.jl

This is stars and forks stats for /nikopj/FlashAttention.jl repository. As of 10 May, 2024 this repository has 8 stars and 1 forks.

FlashAttention.jl Julia implementation of Flash-Attention, a fused kernel for dot-product attention operation in transformer neural networks. Recommended Julia Install Method Install Julia via the juliaup manager: curl -fsSL https://install.julialang.org | sh Todo Julia implementation Naive algorithms: The following are naive in the sense that they all form the normalized adjacency matrix P. Note that GPU version come for free-ish from all of these naive algorithms by making use of NNlib.jl. naive...