AI can't cross this line and we don't know why.

Stephen Welch explains the neural scaling laws and the compute-efficient frontier that limit how far AI models can scale.

Have we discovered an ideal gas law for AI? Head to brilliant.org/WelchLabs/ to try Brilliant for free for 30 days and get 20% off an annual premium subscription. Welch Labs Imaginary Numbers Book! welchlabs.com/resources/imaginary-numbers-book Welch Labs Posters: welchlabs.com/resources Support Welch Labs on Patreon! patreon.com/welchlabs Special thanks to Patrons: Juan Benet, Ross Hanson, Yan Babitski, AJ Englehardt, Alvin Khaled, Eduardo Barraza, Hitoshi Yamauchi, Jaewon Jung, Mrgoodlight, Shinichi Hayashi, Sid Sarasvati, Dominic Beaumont, Shannon Prater, Ubiquity Ventures, Matias Forti, Brian Henry, Tim Palade, Petar Vecutin Learn more about WelchLabs! welchlabs.com TikTok: tiktok.com/@welchlabs Instagram: instagram.com/welchlabs REFERENCES A Neural Scaling Law from the Dimension of the Data Manifold: arxiv.org/pdf/2004.10802 First 2020 OpenAI Scaling Paper: arxiv.org/pdf/2001.08361 GPT-3 Paper: arxiv.org/pdf/2005.14165 Second 202 OpenAI Scaling Paper: arxiv.org/pdf/2010.14701 Google Deepmind “Chinchilla Scaling” Paper: arxiv.org/abs/2203.15556 Nice summary of Chinchilla Scaling: lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-impl… GPT-4 Technical Report: arxiv.org/pdf/2303.08774 Nice Neural Scaling Laws Summary: lesswrong.com/posts/Yt5wAXMc7D2zLpQqx/an-140-theoretical-mod… Explaining Neural Scaling Laws: arxiv.org/pdf/2102.06701 High Cost of Training GPT-4: wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-mo… Nvidia V100 FLOPs: lambdalabs.com/blog/demystifying-gpt-3 Nvidia V100 Original Price: [microway.com/hpc-tech-tips/nvidia-tesla-v100-price-analysis/… GPU model,Key Points](microway.com/hpc-tech-tips/nvidia-tesla-v100-price-analysis/…) Great paper on scaling up training infrastructure: arxiv.org/pdf/2104.04473 Eight Things to Know about LLMs: arxiv.org/abs/2304.00612 Emergent Properties of LLMs: arxiv.org/abs/2206.07682 Theoretical Motivation for Cross Entropy (Section 6.2): deeplearningbook.org/ Some papers that appear to pass the compute efficient frontier arxiv.org/pdf/2206.14486 arxiv.org/abs/2210.11399 CFAQJOTYQHT7JYIT Leaked GPT-4 training info patmcguinness.substack.com/p/gpt-4-details-revealed semianalysis.com/p/gpt-4-architecture-infrastructure epochai.org/blog/tracking-large-scale-ai-models