tomashmon Hacker News • Recommends
On “The Illustrated Transformer (2018)”
This is good, but what bade me finally understand the transformer architecture 0 and attention 1, are 3Blue1Brown's videos. 0. link 1. link to this video
0:00 / 0:00
#transformer#attentionmechanism#self-attention
Attention in transformers, step-by-step | Deep Learning Chapter 6
3Blue1Brown
8.7M subscribers • 42 videos on Hallway
- Recommended by 5 people: 5 on Hacker News.
- Presented by Grant Sanderson, creator of 3Blue1Brown
- Linked in the description3b1b/manimManimCommunity/manim3b1b/videosarXiv 1301.3781
Intermediate. Assumes working knowledge.
4.8M views2 years ago5 recommendations
AI and machine learningTransformers and LLMsNeural networks and deep learningMath for machine learning
Grant Sanderson provides a visual, step-by-step explanation of the attention mechanism inside transformers and LLMs.
Demystifying attention, the key mechanism inside transformers and LLMs. Instead of sponsored ad reads, these lessons are funded directly by viewers: 3b1b.co/support Special thanks to these supporters: 3blue1brown.com/lessons/attention#thanks An equally valuable form of support is to simply share the videos. Demystifying self-attention, multiple heads, and cross-attention. Instead of sponsored ad reads, these lessons are funded directly by viewers: 3b1b.co/support The first pass for the translated subtitles here is machine-generated and, therefore, notably imperfect. To contribute edits or fixes, visit criblate.com Звуковая дорожка на русском языке: Влад Бурмистров. ------------------ Here are a few other relevant resources Build a GPT from scratch, by Andrej Karpathy youtu.be/kCc8FmEb1nY If you want a conceptual understanding of language models from the ground up, @vcubingx just started a short series of videos on the topic: youtu.be/1il-s4mgNdI?si=XaVxj6bsdy3VkgEX If you're interested in the herculean task of interpreting what these large networks might actually be doing, the Transformer Circuits posts by Anthropic are great. In particular, it was only after reading one of these that I started thinking of the combination of the value and output matrices as being a combined low-rank map from the embedding space to itself, which, at least in my mind, made things much clearer than other sources. transformer-circuits.pub/2021/framework/index.html Site with exercises related to ML programming and GPTs gptandchill.ai/codingproblems History of language models by Brit Cruise, @ArtOfTheProblem youtu.be/OFS90-FX6pg An early paper on how directions in embedding spaces have meaning: arxiv.org/pdf/1301.3781.pdf ------------------ Timestamps: - Recap on embeddings - Motivating examples - The attention pattern - Masking - Context size - Values - Counting parameters - Cross-attention - Multiple heads - The output matrix - Going deeper - Ending ------------------ These animations are largely made using a custom Python library, manim. See the FAQ comments here: 3b1b.co/faq#manim github.com/3b1b/manim github.com/ManimCommunity/manim/ All code for specific videos is visible here: github.com/3b1b/videos/ The music is by Vincent Rubinetti. vincentrubinetti.com vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown open.spotify.com/album/1dVyjwS8FBqXhRunaG5W5u ------------------ 3blue1brown is a channel about animating math, in all senses of the word animate. If you're reading the bottom of a video description, I'm guessing you're more interested than the average viewer in lessons here. It would mean a lot to me if you chose to stay up to date on new ones, either by subscribing here on YouTube or otherwise following on whichWhy it's here
5 people recommend this, 1 disputes itWhat people say. Highly recommended as one of the clearest and most intuitive visual explanations of transformer attention.
onlyrealcuzzoon Hacker News • Recommends
On “Claude Opus 4.8”
…a of what's happening (searching paths), and how it can potentially have huge orders of magnitude improvements (doing so without paying the full price of exploring the paths through the expensive and huge model). And also why it is so much harder to determine what it's "thinking". If you aren't familiar with hyper words, this is an amazing series: link to this video The general idea is that a token is a multidimensional vector to represent a word -> think like "man" is a noun, singular, English, pronoun, masculine, contemporary, .... Each time is sees a new word, it mutates this word to mean some new token (ofte…
MAXPOOLon Hacker News • Recommends
On “The Engineer’s Guide to Deep Learning: Understanding the Transformer Model”
There are many others that are better. 1/ The Annotated Transformer Attention is All You Need link 2/ Transformers from Scratch link 3/ Andrej Karpathy has really good series of intros: link Let's build GPT: from scratch, in code, spelled out. link GPT with Andrej Karpathy: Part 1 link 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning link Attention in transformers, visually explained | Chapter 6, Deep Learning link to this video Full 3Blue1Brown Neural Networks playlist link
superkuhon Hacker News • Recommends
On “LLMs Aren't Thinking, They're Just Counting Votes”
…] ). Though it is obvious that humans can do more than that too. And while "voting" that may be a close-enough metaphor for the multi-layer perceptron layers, it is not what is happening in the attention layers. I suggest checking out 3blue1brown's videos on gradient descent link and "How might LLMs store facts" link versus how attention works link to this video…
wren6991on Hacker News • Recommends
On “Can I Buy Your KV Cache?”
If you want the actual maths instead of handwaving, I recommend: link For something gentler, 3Blue1Brown: link to this video (this is part of a series)
lolinderon Hacker News • Disputes
On “YouTube seems to once again be rolling out its widely hated new web redesign”
Eh, insightful and thoughtful videos tend to have better comments than the low brow videos, but they're still not useful . The best of them read like a decent-but-not-great subreddit's comment thread. Taking one example, 3Blue1Brown's Deep Learning series, episode 6. Here are the top few comments 0: > I'm a university lecturer with a PhD in AI, and I cannot compete with the quality of this work. Videos like this put the entire higher education system to shame. Fantastic! This is a nice comment, but doesn't contribute anything to my day. I'm sure the creator is glad to see it and I'm glad someone said it, but I don't need to read it. > Are you kidding me? ONE WEEK FOR 2 MASTERPIECES?! > Tha…
rahimnathwanion Hacker News • Mentions
On “Visualizing Attention, a Transformer's Heart [video]”
He lists Ring Attention and half a dozen other techniques, but they're not within the scope of this video: link to this video
peterlaion Hacker News • Mentions
On “3Blue1Brown Follow-Up: From Hypothetical Examples to LLM Circuit Visualization”
About a year ago, I watched this 3Blue1Brown LLM tutorial ( link to this video ) on how a model’s self-attention mechanism is used to predict the next token in a sequence, and I was surprised by how little we know about what actually happens when processing the sentence “A fluffy blue creature roamed the verdant forest.” A year later, the field of mechanistic interpretability has seen significant advancements, and we're now able to "decompose" models into interpretable circuits that help explain how LLMs produce predictions. Using the second iteration of an LLM "debugger" I've been working on, I compare the hypothetical representations used in the tutorial to the actual representations I see when extra…