8.7M subscribers • 42 videos on Hallway
Intermediate. Assumes working knowledge.
2.2M views1 year ago1 recommendation
AI and machine learningComputer scienceNeural networks and deep learningMath for machine learningComputer vision
Welch Labs explains the mathematics and mechanics behind text-to-image models, covering CLIP, diffusion models, conditioning, and guidance.
Diffusion models, CLIP, and the math of turning text into images Welch Labs Book: welchlabs.com/resources/imaginary-numbers-book Sections - Intro - CLIP - Shared Embedding Space - Diffusion Models & DDPM - Learning Vector Fields - DDIM - Dall E 2 - Conditioning - Guidance - Negative Prompts - Outro - About guest videos Special Thanks to: Jonathan Ho - Jonathan is the Author of the DDPM paper and the Classifier Free Guidance Paper. arxiv.org/pdf/2006.11239 arxiv.org/pdf/2207.12598 Preetum Nakkiran - Preetum has an excellent introductory diffusion tutorial: arxiv.org/pdf/2406.08929 Chenyang Yuan - Many of the animations in this video were implemented using manim and Chenyang’s smalldiffusion library: github.com/yuanchenyang/smalldiffusion Cheyang also has a terrific tutorial and MIT course on diffusion models chenyang.co/diffusion.html practical-diffusion.org/ Other References All of Sander Dieleman’s diffusion blog posts are fantastic: sander.ai/ CLIP Paper: arxiv.org/pdf/2103.00020 DDIM Paper: arxiv.org/pdf/2010.02502 Score-Based Generative Modeling: arxiv.org/pdf/2011.13456 Wan2.1: github.com/Wan-Video/Wan2.1 Stable Diffusion: huggingface.co/stabilityai/stable-diffusion-2 Midjourney: midjourney.com/ Veo: deepmind.google/models/veo/ DallE 2 paper: cdn.openai.com/papers/dall-e-2.pdf Code for this video: github.com/stephencwelch/manim_videos/tree/master/_2025/sora Written by: Stephen Welch, with very helpful feedback from Grant Sanderson Produced by: Stephen Welch, Sam Baskin, and Pranav Gundu Technical Notes The noise videos in the opening have been passed through a VAE (actually, diffusion process happens in a compressed “latent” space), which acts very much like a video compressor - this is why the noise videos don’t look like pure salt and pepper. CLIP: Although directly minimizing cosine similarity would push our vectors 180 degrees apart on a single batch, overall in practice, we need CLIP to maximize the uniformity of concepts over the hypersphere it's operating on. For this reason, we animated these vectors as orthogonal-ish. See: proceedings.mlr.press/v119/wang20k/wang20k.pdf Per Chenyang Yuan: at , the blurry image that results when removing random noise in DDPM is probably due to a mismatch in noise levels when calling the denoiser. When the denoiser is called on x_{t-1} during DDPM sampling, it is expected to have a certain noise level (let's call it sigma_{t-1}). If you generate x_{t-1} from x_t without adding noise, then the noise present in x_{t-1} is always smaller than sigma_{t-1}. This causes the denoiser to remove too much noise, thus pointing towards the mean of the dataset. The text conditioning input to stable diffusion is not the 512-dim text embedding vector, but the output of thWhat people say. Recommended as an exceptionally clear visual explanation of the mathematics and mechanics behind diffusion models and image generation.
dilyevskyon Hacker News • Recommends
On “Playing with Vision Embeddings”
elliottoon Hacker News • Mentions
On “Training language models to be warm and empathetic makes them less reliable”
pornelon Hacker News • Mentions
On “The maths you need to start understanding LLMs”