Deep Dive into LLMs like ChatGPT

Name: Deep Dive into LLMs like ChatGPT
Uploaded: 2025-02-05T18:23:47Z
Duration: 3 h 31 min 24 s
Channel: Andrej Karpathy
Description: Andrej Karpathy explains the full training stack of Large Language Models, from pretraining to post-training, and provides mental models for their behavior.

Andrej Karpathy

1.7M subscribers • 6 videos on Hallway

YouTube

Recommended by 10 people: 10 on Hacker News.
Presented by Andrej Karpathy, co-founder of OpenAI
Linked in the descriptionkarpathy/llm.c arXiv 2407.21783 arXiv 2203.02155 arXiv 2501.12948

Beginner. No background in the topic needed.

10M views1 year ago10 recommendations

AI and machine learning Transformers and LLMs Neural networks and deep learning Training and serving models

Andrej Karpathy explains the full training stack of Large Language Models, from pretraining to post-training, and provides mental models for their behavior.

This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their "psychology", and how to get the best use them in practical applications. I have one "Intro to LLMs" video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version. Instructor Andrej was a founding member at OpenAI (2015) and then Sr. Director of AI at Tesla (2017-2022), and is now a founder at Eureka Labs, which is building an AI-native school. His goal in this video is to raise knowledge and understanding of the state of the art in AI, and empower people to effectively use the latest and greatest in their work. Find more at karpathy.ai/ and x.com/karpathy Chapters introduction pretraining data (internet) tokenization neural network I/O neural network internals inference GPT-2: training and inference Llama 3.1 base model inference pretraining to post-training post-training data (conversations) hallucinations, tool use, knowledge/working memory knowledge of self models need tokens to think tokenization revisited: models struggle with spelling jagged intelligence supervised finetuning to reinforcement learning reinforcement learning DeepSeek-R1 AlphaGo reinforcement learning from human feedback (RLHF) preview of things to come keeping track of LLMs where to find LLMs grand summary Links - ChatGPT chatgpt.com/ - FineWeb (pretraining dataset): huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1 - Tiktokenizer: tiktokenizer.vercel.app/ - Transformer Neural Net 3D visualizer: bbycroft.net/llm - llm.c Let's Reproduce GPT-2 github.com/karpathy/llm.c/discussions/677 - Llama 3 paper from Meta: arxiv.org/abs/2407.21783 - Hyperbolic, for inference of base model: app.hyperbolic.xyz/ - InstructGPT paper on SFT: arxiv.org/abs/2203.02155 - HuggingFace inference playground: huggingface.co/spaces/huggingface/inference-playground - DeepSeek-R1 paper: arxiv.org/abs/2501.12948 - TogetherAI Playground for open model inference: api.together.xyz/playground - AlphaGo paper (PDF): discovery.ucl.ac.uk/id/eprint/10045895/1/agz_unformatted_nat… - AlphaGo Move 37 video: youtube.com/watch?v=HT-UZkiOLv8 - LM Arena for model rankings: lmarena.ai/ - AI News Newsletter: buttondown.com/ainews - LMStudio for local inference lmstudio.ai/ - The visualization UI I was using in the video: excalidraw.com/ - The specific file of Excalidraw we built up: drive.google.com/fi

Why it's here

10 people recommend this

What people say. Highly recommended as an exceptionally clear and approachable deep dive into how large language models work under the hood.

InvidFloweron Hacker News • 1 year agoRecommends

On “Ladder: Self-improving LLMs through recursive problem decomposition”

I think you'll get a lot of use from this video from Andrej Karpathy: link to this video It is long, but don't get scared off. He goes over a ton of different stuff related to model training, but makes it very easy to understand.