Bill Dally explains how co-designing hardware and software, specialized instructions, and new numeric formats have driven a 1000x improvement in GPU efficiency.
Allen School Distinguished Lecture Series
Title: Trends in Deep Learning Hardware
Speaker: Bill Dally, NVIDIA
Date: Thursday, November 9, 2023
Abstract: The current resurgence of artificial intelligence, including generative AI like ChatGPT, is due to advances in deep learning. Systems based on deep learning now exceed human capability in speech recognition, object classification, and playing games like Go. Deep learning has been enabled by powerful, efficient computing hardware. The algorithms used have been around since the 1980s, but it has only been in the last decade -- when powerful GPUs became available to train networks -- that the technology has become practical. Advances in DL are now gated by hardware performance. In the last decade, the efficiency of DL inference on GPUs had improved by 1000x. Much of this gain was due to improvements in data representation starting with FP32 in the Kepler generation of GPUs and scaling to Int8 and FP8 in the Hopper generation. This talk will review this history and discuss further improvements in number representation including logarithmic representation, optimal clipping, and per-vector quantization.
Bio: Bill Dally joined NVIDIA in January 2009 as chief scientist, after spending 12 years at Stanford University, where he was chairman of the computer science department. Dally and his Stanford team developed the system architecture, network architecture, signaling, routing and synchronization technology that is found in most large parallel computers today. Dally was previously at the Massachusetts Institute of Technology from 1986 to 1997, where he and his team built the J-Machine and the M-Machine, experimental parallel computer systems that pioneered the separation of mechanism from programming models and demonstrated very low overhead synchronization and communication mechanisms. From 1983 to 1986, he was at California Institute of Technology (CalTech), where he designed the MOSSIM Simulation Engine and the Torus Routing chip, which pioneered "wormhole" routing and virtual-channel flow control. He is a member of the National Academy of Engineering, a Fellow of the American Academy of Arts & Sciences, a Fellow of the IEEE and the ACM, and has received the ACM Eckert-Mauchly Award, the IEEE Seymour Cray Award, and the ACM Maurice Wilkes award. He has published over 250 papers, holds over 120 issued patents, and is an author of four textbooks. Dally received a bachelor's degree in Electrical Engineering from Virginia Tech, a master's in Electrical Engineering from Stanford University and a Ph.D. in Computer Science from CalTech. He was a cofounder of Velio Communications and Stream Processors.
This video is closed captioned.
Why it's here
3 people recommend this
What people say. Highly recommended for its deep dive into the hardware and software co-design that enabled the modern AI revolution.
a
aleccoon Hacker News • Recommends
On “Intel CEO: 'The entire industry is motivated to eliminate the CUDA market'”
It looks like both Pat Gelsinger and Lisa Su have no clue about software and can't even delegate this. They expect "the community" to do the software for their extremely complicated hardware. If you see the presentations by Nvidia's Bill Daly 1 it shows they've been evolving hardware and software together. The CUDA programming model was a massive bet a long time ago. And they deservedly won ML/AI. Unless Intel and AMD do a radical change it's game over for them. They will lose to ARM and Nvidia. 1link to this video
…ding to the video have been different numeric types (fp16 for example) and more specialised hardware together with more complex instructions. Note that process improvements have only caused a 2.5x from that 1000x. He also mentions that Google‘s TPU has no fundamental benefit over dedicated instructions in GPUs. Quite an interesting video IMO. 1: link to this video…
On “Why does integer representation of a floating point approximate its logarithm?”
In a recent talk1, Bill Dally of Nvidia suggests using actual logarithms, as they're more consistent in relative error than floats, which is especially relevant at low precision as increasingly used in AI. Multiplications then become simple additions, but then addition becomes more complicated, as in general it needs lookup tables. This is the same mechanism as a slide rule, and also reminds me of the way the Yamaha DX7 works. The lookup tables can be expensive, and the talk contains an idea how to batch them when adding up the results of many many multiplications (as is also often the case for AI workloads). 1: link to this video
On “Training for one trillion parameter model backed by Intel and US govt has begun”
> Let's Fermi estimate that. I couldn't agree more. Fermi estimates are underused. > Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data. And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens 1. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte. As a side note, I would guess that GPT-4 knows more "things" if you would be able to count them all up. For example, it knows more languages, more facts about cities, more facts about persons. However, people know way more inside their specialization. 1: link to this video
On “Measuring energy usage: regular code vs. SIMD code”
There's a talk by a Nvidia scientist I just saw where he talks about compute efficiency in terms of flops/joule and explicitly mentions vectorized instructions as one of the energy savers. I'll see if I can find it. Edit: found it (I think) I can't remember where in the presentation but he does mention energy efficiency as a proxy for performance. link to this video Edit 2: about 26:40 he starts talking about energy use in the context of performance.
