Horace He discusses the evolution of machine learning systems, optimizing single-GPU performance with compilers, and scaling training to tens of thousands of GPUs.
Over the last 10 years we've seen Machine Learning consume everything, from the tech industry to the Nobel Prize, and yes, even the ML acronym. This rise in ML has also come along with an unprecedented buildout of infra, with Llama 3 now reaching 4e25 floating point operations, or 40 yottaflops, or 40 trillion trillion floating point operations.
To build these ML models, you need ML systems, like PyTorch. In this talk, Horace will (attempt to) answer:
- How have ML systems evolved over time to meet the training needs of ML models? How does building ML systems differ from regular systems?
- How do we get the most out of a single GPU? What's the point of compilers if we're just training a single model?
- And what is the right way to think about scaling to 10s of thousands of GPUs?
Why it's here
2 people recommend this
What people say. Recommended for learning about low-level machine learning systems, GPU optimization, and large-scale training infrastructure.
t
u/thisis_a_cipherin r/compsci • Recommends
How do I get to the next level in low-level programming and ML?
I am currently a year 2 CS student. I've been coding for 8 years now, but I'm realising that despite all that time, my general ability and knowledge level don't actually amount to much beyond being able to use libraries, APIs, frameworks etc.
Specifically, I'm really interested in low-level stuff and machine learning but I have no idea ***how to become good enough at it to actually start making meaningful contributions***. It has become clear to me that my coursework is not going to be sufficient. What I mean by this is that if I take a compilers class or maybe a parallel computing class, that does not bring me up to a suffici…
