toastalon Lobsters • Recommends
GradIEEEnt half decent: The hidden power of imprecise lines
0:00 / 0:00
#gradientdescent#floating-point#half-precision
GradIEEEnt half decent: The hidden power of imprecise lines
suckerpinch
202K subscribers • 15 videos on Hallway
- Recommended by 2 people: 1 on Hacker News and 1 on Lobsters.
- Presented by Tom Murphy VII, computer scientist
Advanced. Deep dive for people who know the field.
390K views3 years ago2 recommendations
AI and machine learningSystems programmingNeural networks and deep learningMath for machine learningPerformance and optimization
Tom 7 demonstrates how the rounding errors in half-precision floating-point math can provide the non-linearity needed to train a neural network.
Before the invention of YouTube comments, most people could make remarks that were slightly technically incorrect without fear of immediate public rebuke. The one exception was professors, especially if the classroom included an annoying student such as “Tom 7.” The invention of YouTube was doubly revolutionary: Now anyone can experience being a professor being corrected by an annoying student, but also, corrections can be made years after the fact, and at significant length, as the student cannot be told to “take this offline.” This video is such a lengthy correction and digression, and perhaps itself a fount of mistakes. Due to the esoteric detail and prodigious whiteboarding, it is almost a “Tom Academy” video. But since I grew a moustache to film it, it qualifies for the Main Sequence. Keywords: Gradient descent, half-precision floating point, linear operations, F=MA, rounding, exotic transfer functions, machine learning, MNIST, CIFAR-10, chess, fractals, Frobenius, cryptography, fluint8, Motorola 6502. For SIGBOVIK 2023. Paper, impenetrable code, etc.: tom7.org/grad/Why it's here
2 people recommend thisWhat people say. Recommended as a brilliant and entertaining exploration of using floating-point rounding errors as activation functions.
jcparkynon Hacker News • Recommends
On “Your Transformer is Secretly Linear”
A fun example of that (making a neural network using floating point error as a source of non-linearity): link to this video
superkuhon Hacker News • Mentions
On “Numbers every LLM Developer should know”
Cool stuff. I looked at link and I noticed that that the fp8 support is only for the tensor cores and not the CUDA side. Does that mean training with H100 GPU in fp8 mode would use some software ecosystem that's not the existing vast existing CUDA one? Or am I just misunderstanding CUDA cores vs tensor cores? PS, as a joke, they should implement GPU fluint8 and get baked in non-linearity for the activation function without even using a non-linear function, link to this video ("GradIEEEnt half decent: The hidden power of imprecise lines" by suckerpinch)
Uehrekaon Hacker News • Mentions
On “"We'll call it AI to sell it, Machine Learning to Build it"”
I don’t get what people are trying to say when they say these kinds of things about AI. That human-level writing is as simple as a linear regression? That we could’ve had computer programs capable of human-level writing decades ago? Have they not used these AIs enough to see how powerful they are? Are they seeing the bad outits and thinking that AIs are always doing that poorly? Like seriously, if you’re telling me that it was obvious that a “linear regression” could pass the LSAT I’ve got a macvlan to sell you. Edit: they’re also literally not linear regression! link to this video
jcparkynon Hacker News • Mentions
On “Were RNNs all we needed?”
> squash an output into a range This isn't the primary purpose of the activation function, and in fact it's not even necessary. For example see ReLU (probably the most common activation function), leaky ReLU, or for a sillier example: link to this video