Transformers, the tech behind LLMs | Deep Learning Chapter 5

Name: Transformers, the tech behind LLMs | Deep Learning Chapter 5
Uploaded: 2024-04-01T19:13:57Z
Duration: 27 min 14 s
Channel: 3Blue1Brown
Description: Grant Sanderson provides a visual explanation of how transformers and large language models process data through embeddings and attention mechanisms.

3Blue1Brown

8.7M subscribers • 42 videos on Hallway

YouTube

Recommended by 14 people: 13 on Hacker News and 1 on Lobsters.
Presented by Grant Sanderson
Linked in the descriptionarXiv 1301.3781

Intermediate. Assumes working knowledge.

11M views2 years ago14 recommendations

AI and machine learning Computer science Transformers and LLMs Neural networks and deep learning Math for machine learning

Grant Sanderson provides a visual explanation of how transformers and large language models process data through embeddings and attention mechanisms.

Breaking down how Large Language Models work, visualizing how data flows through. Instead of sponsored ad reads, these lessons are funded directly by viewers: 3b1b.co/support --- Here are a few other relevant resources Build a GPT from scratch, by Andrej Karpathy youtu.be/kCc8FmEb1nY If you want a conceptual understanding of language models from the ground up, @vcubingx just started a short series of videos on the topic: youtu.be/1il-s4mgNdI?si=XaVxj6bsdy3VkgEX If you're interested in the herculean task of interpreting what these large networks might actually be doing, the Transformer Circuits posts by Anthropic are great. In particular, it was only after reading one of these that I started thinking of the combination of the value and output matrices as being a combined low-rank map from the embedding space to itself, which, at least in my mind, made things much clearer than other sources. transformer-circuits.pub/2021/framework/index.html History of language models by Brit Cruise, @ArtOfTheProblem youtu.be/OFS90-FX6pg An early paper on how directions in embedding spaces have meaning: arxiv.org/pdf/1301.3781.pdf Звуковая дорожка на русском языке: Влад Бурмистров. --- Timestamps - Predict, sample, repeat - Inside a transformer - Chapter layout - The premise of Deep Learning - Word embeddings - Embeddings beyond words - Unembedding - Softmax with temperature - Up next

Why it's here

14 people recommend this

What people say. Highly recommended as an exceptionally clear, visual explanation that makes the self-attention mechanism and transformer architecture intuitive.

BenoitPon Hacker News • 2 years agoRecommends

On “Visualizing Attention, a Transformer's Heart [video]”

3Blue1Brown is really a godsend. I had been trying and failed to get an intuition for the self-attention mechanisms, and also why you want to stack you transformer layers. I went through a lot of material (Key-Query-Value interpretations, SVD of these, etc.) but his comment in chapter 5 video 1 made it click for me: "But you should think of the primary goal of this network that it (the word embedding) flows through as being to enable each one of those vectors to soak up a meaning that's much more rich and specific than what mere individual words could represent". Here is my intuition: Transformers are merely a Convnet for token sequences. In the sense that information is first local and ea…