u/currentscurrentsin r/compsci • Recommends
On “Computation theoretical reason for why memory is a bottle neck in CPU/GPUs?”
Formally, this is called the Vonn Neumann bottleneck. Compute and memory are infinitely parallelizable, but the pipe between them is not. Moving information around is more expensive than computing on it.
So far, we have fought this by making memory faster and putting it closer to the CPU. Larger caches, higher memory bandwidth, etc. But this is expensive (both in power and die space) and doesn't really solve the problem.
The real solution is to abandon the idea of separate compute and memory, called compute-in-memory. But this would require fundamentally changing how we write programs, which is why we haven't done it already.
Neural networks are well-suited to it - you could build a ne…