lucidrains/flash-cosine-sim-attention

This is stars and forks stats for /lucidrains/flash-cosine-sim-attention repository. As of 06 May, 2024 this repository has 184 stars and 8 forks.

Dive into Deep Learning, redone by Quanta Magazine Flash Cosine Similarity Attention Implementation of fused cosine similarity attention in the same style as Flash Attention. The observation is that by adopting l2 normalized queries and keys, you no longer need to keep track of the row maximums for numerical stability. This greatly simplifies the flash attention algorithm, assuming cosine similarity attention comes at no generalization cost. In other words, stable, fast, memory efficient, and longer...