paulmiston Hacker News • Recommends
On “Noam Shazeer Joins OpenAI”
I first saw Noam on Dwarkesh’s podcast together with Jeff Dean. Recommend if you want a taste of what’s Google’s folks take on things. link to this video
0:00 / 0:00
#transformer#mixtureofexperts#gemini
Jeff Dean & Noam Shazeer — 25 years at Google: from PageRank to AGI
Dwarkesh Patel
1.4M subscribers • 2 videos on Hallway
- Recommended by 1 person: 1 on Hacker News.
- Presented by Jeff Dean, Chief Scientist at Google, and Noam Shazeer, co-inventor of the Transformer
Intermediate. Assumes working knowledge.
349K views1 year ago1 recommendation
AI and machine learningDistributed systemsTransformers and LLMsScaling and large-scale architectureHistory of computing
Jeff Dean and Noam Shazeer discuss the evolution of Google's infrastructure and AI, from MapReduce and PageRank to the Transformer and Gemini.
This week I welcome two of the most important technologists in any field. Jeff Dean is Google's Chief Scientist, and through 25 years at the company, has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip, to Gemini. Noam Shazeer invented or co-invented all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to Gemini and many other things. We talk about their 25 years at Google, going from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and soon to ASI. 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒 * Transcript: dwarkesh.com/p/jeff-dean-and-noam-shazeer * Apple Podcasts: podcasts.apple.com/us/podcast/jeff-dean-noam-shazeer-25-year… * Spotify: open.spotify.com/episode/4atx1POpKIL8WGvdVfdnbb?si=DLn5uQYMQ… 𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 * Meter wants to radically improve the digital world we take for granted. They’re developing a foundation model that automates network management end-to-end. To do this, they just announced a long-term partnership with Microsoft for tens of thousands of GPUs, and they’re recruiting a world class AI research team. To learn more, go to meter.com/dwarkesh * Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh * Curious how Jane Street teaches their new traders? They use Figgie, a rapid-fire card game that simulates the most exciting parts of markets and trading. It’s become so popular that Jane Street hosts an inter-office Figgie championship every year. Download from the app store or play on your desktop at figgie.com/ To sponsor a future episode, visit dwarkesh.com/p/advertise 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 - Intro - Joining Google in 1999 - Future of Moore's Law - Future TPUs - Jeff’s undergrad thesis: parallel backprop - LLMs in 2007 - “Holy shit” moments - AI fulfills Google’s original mission - Doing Search in-context - The internal coding model - What will 2027 models do? - A new architecture every day? - Automated chips and intelligence explosion - Future of inference scaling - Already doing multi-datacenter runs - Debugging at scale - Fast takeoff and superalignment - A million evil Jeff Deans - Fun times at Google - World compute demand in 2030 - Getting back to modularity - Keeping a giga-MoE in-memory - All of Google in onWhy it's here
1 person recommends this, 1 disputes itWhat people say. Recommended for its deep historical and technical insights into the development of modern distributed systems and LLMs at Google.
DarkCrusader2on Hacker News • Disputes
On “Dario Amodei – "We are near the end of the exponential"”
Yeah I also don't understand how he is able to get such high profile guests. His interview with Jeff Dean and Noam Shazeer last year1 is so hilariously bad. Jeff and Noam kept trying to give really insightful answers on how they see AI development shaping in coming years and he was just steering the conversation to shallow and silly tabloid gossip (why don't you "just" let AI improve the next version in a loop so we can quickly have singularity, Jeff Dean AI running in a DC, evil Jeff Dean AI escaping containment and on and on). It was just embarrassing. The interview would have been so much better with just Jeff and Noam without him. 1 link to this video
itchyjunkon Hacker News • Mentions
On “Google is using AI to censor independent websites like mine”
Noam Shazeer says it at some point in the Dwarkesh Patel podcast. Edit : Saw multiple people comment by the time I hit sent. Was it also memorable for you guys? I think it was just the way he said it or something. link to this video
gjm11on Hacker News • Mentions
On “Andrej Karpathy – It will take a decade to work through the issues with agents”
…inks AI systems might be solving some of the Millennium Prize Problems in mathematics by then; from about 17:10 he's asked about how scientists will work ~5 years in the future and he says AI systems will be apprentices or collaborators; at any rate he's not not saying that human-level AI is likely to come in the near future. From about 1:12:40 in link to this video Noam Shazeer (second author on that paper), in response to a question about "fast takeoff", says that he does expect a very rapid improvement in AI capabilities; he's not explicit about when he expects that to happen or how far he expects it to go, but my impression from the other bits of that discussion I watched is that he too is not not…
sosodevon Hacker News • Mentions
On “Gemini 3”
Here's some insight from Jeff Dean and Noam Shazeer's interview with Dwarkesh Patel link to this video > When you think about divulging this information that has been helpful to your competitors, in retrospect is it like, "Yeah, we'd still do it," or would you be like, "Ah, we didn't realize how big a deal transformer was. We should have kept it indoors." How do you think about that? > Some things we think are super critical we might not publish. Some things we think are really interesting but important for improving our products; We'll get them out into our products and then make a decision.