a_wild_dandanon Hacker News • Recommends
On “Jamba: Production-grade Mamba-based AI model”
To those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: link to this video
0:00 / 0:00
#statespacemodels#attentionmechanism#linearrnns
Do we need Attention? - Linear RNNs and State Space Models (SSMs) for NLP
Sasha Rush
14K subscribers • 1 video on Hallway
- Recommended by 3 people: 3 on Hacker News.
- Presented by Sasha Rush
- Linked in the descriptionsrush/do-we-need-attention
Advanced. Deep dive for people who know the field.
35K views3 years ago3 recommendations
AI and machine learningNeural networks and deep learningTransformers and LLMs
Sasha Rush surveys linear recurrent neural networks and state space models as potential alternatives to the attention mechanism in transformer architectures.
(More Recent version for Mamba: youtube.com/watch?v=dVH1dRoMPBc) A talk for MLSys surveying recent methods using linear RNNs and State Space models to replace attention in transformer-style models. Slides: github.com/srush/do-we-need-attention/blob/main/DoWeNeedAtte… This talk predates the work on Mamba, but covers foundational preliminaries. Mamba version coming soon.Why it's here
3 people recommend thisWhat people say. Recommended for providing excellent intuition and foundational explanations of state space models as alternatives to transformers.
anomaly72on Hacker News • Recommends
On “MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts”
Would recommend Sasha Rush's lecture link to this video , it provides the intuition for state space models via linear RNNs
srushon Hacker News • Recommends
On “The Annotated S4”
Hi! Blog author. This was an attempt a couple years ago to understand and write about this paper in a detailed way. Here is a video going through this topic as well: link to this video Section 2 of the blog post is no longer very relevant. A lot of advances (DSS, S4D) simplified that part of the process. Arguably also this all should be updated for Mamba (same authors).