#backpropagation#neuralnetworks#micrograd
1.7M subscribers • 6 videos on Hallway
Beginner. No background in the topic needed.
4.3M views4 years ago17 recommendations
AI and machine learningComputer scienceNeural networks and deep learningMath for machine learningAlgorithms
Andrej Karpathy builds a tiny autograd engine in Python, explaining backpropagation and neural network training step by step.
This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school. Links: - micrograd on github: github.com/karpathy/micrograd - jupyter notebooks I built in this video: github.com/karpathy/nn-zero-to-hero/tree/master/lectures/mic… - my website: karpathy.ai - my twitter: twitter.com/karpathy - "discussion forum": nvm, use youtube comments below for now :) - (new) Neural Networks: Zero to Hero series Discord channel: discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments Exercises: you should now be able to complete the following google collab, good luck!: colab.research.google.com/drive/1FPTx1RXtBfc4MaTkf7viZZD4U2F… Chapters: intro micrograd overview derivative of a simple function with one input derivative of a function with multiple inputs starting the core Value object of micrograd and its visualization manual backpropagation example #1: simple expression preview of a single optimization step manual backpropagation example #2: a neuron implementing the backward function for each operation implementing the backward function for a whole expression graph fixing a backprop bug when one node is used multiple times breaking up a tanh, exercising with more operations doing the same thing but in PyTorch: comparison building out a neural net library (multi-layer perceptron) in micrograd creating a tiny dataset, writing the loss function collecting all of the parameters of the neural net doing gradient descent optimization manually, training the network summary of what we learned, how to go towards modern neural nets walkthrough of the full code of micrograd on github real stuff: diving into PyTorch, finding their backward pass for tanh conclusion outtakes :)What people say. Widely recommended as the absolute best, most intuitive introduction to backpropagation and the mathematical fundamentals of neural networks.
Geeeon Hacker News • Recommends
On “A step by step backpropagation example (2015)”
ian0on Hacker News • Recommends
On “Andrej Karpathy Is Joining OpenAI Again”
base698on Hacker News • Recommends
On “ChatGPT Explained: A Normie's Guide to How It Works”
gavion Hacker News • Recommends
On “Deep Learning as Computational Graphs (2018)”
radqon Hacker News • Recommends
On “GPT4 is 8 x 220B params = 1.7T params”
paulsmithon Hacker News • Recommends
On “Linear Algebra for Programmers”
throwaway7783on Hacker News • Recommends
On “How linear regression works intuitively and how it leads to gradient descent”
jackdoeon Hacker News • Recommends
On “The Illusion of Thinking: Strengths and limitations of reasoning models [pdf]”
omrishnon Hacker News • Recommends
On “Deep dive into the small details of micrograd”
jwitthuhnon Hacker News • Recommends
On “The Illustrated Transformer”
deracon Hacker News • Recommends
On “MinGPT: Minimal PyTorch re-implementation of GPT”
bialpioon Hacker News • Recommends
On “Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy”
tjvcon Hacker News • Recommends
On “Post-GPT Computing”
karpathyon Hacker News • Recommends
On “Llm.c – LLM training in simple, pure C/CUDA”
sprobertsonon Hacker News • Recommends
On “Ask HN: Calculus Refresher for AI/ML?”
strangecastson Hacker News • Recommends
On “Ray Tracing in One Weekend”