Skip to content
0:00 / 0:00

#freemandyson#enricofermi#overfitting

Freeman Dyson - Fermi's rejection of our work (94/157)

Web of Stories - Life Stories of Remarkable People

65K subscribers • 2 videos on Hallway

YouTube
  • Recommended by 2 people: 2 on Hacker News.
  • Presented by Freeman Dyson, theoretical physicist and mathematician
  • May be out of date. Uploaded 10 years ago, in an area where tools change fast.

Beginner. No background in the topic needed.

257K views10 years ago2 recommendations

AI and machine learningComputer scienceClassic machine learningMath for programmers

Freeman Dyson recounts how Enrico Fermi rejected his team's complex physical model, famously quoting John von Neumann on overfitting with too many parameters.

To listen to more of Freeman Dyson’s stories, go to the playlist: youtube.com/playlist?list=PLVV0r6CmEsFzDA6mtmKQEgWfcIu49J4nN Freeman Dyson (1923-2020), who was born in England, moved to Cornell University after graduating from Cambridge University with a BA in Mathematics. He subsequently became a professor and worked on nuclear reactors, solid state physics, ferromagnetism, astrophysics and biology. He published several books and, among other honours, was awarded the Heineman Prize and the Royal Society's Hughes Medal. [Listener: Sam Schweber; date recorded: 1998] TRANSCRIPT: So I was in a high state of enthusiasm and I decided I would go to Chicago and show these results to Fermi and tell him how well we were doing. We wanted to have Fermi's blessing on our efforts as he was really the prime mover in this whole subject, and it was a good opportunity for me to get to know Fermi. Anyway, I arranged through Hans Bethe to go to Chicago and tell him about what we were doing. So I arrived at Chicago and knocked on Fermi's door, and he was very polite. I came in, and he said, 'Yes?' and I showed him the graphs on which our experiment, our theoretical numbers were plotted and Fermi's experimental numbers were plotted, and the agreement was on the whole pretty good. And Fermi hardly looked at these graphs, he just put them on the desk, just glanced at them very briefly and he said, 'I am not very impressed with what you've been doing.' And he said, 'When one does a theoretical calculation, you know, there are two ways of doing it. Either you should have a clear physical model in mind, or you should have a rigorous mathematical basis. You have neither.' So that was it - in about two sentences he disposed of the whole subject. Well then I asked him, well what does he think about the numerical agreement, and he said, 'How many parameters did you use for the fitting? How many free parameters are there in your method?' So I counted up. It turned out there were four. And he said, 'You know, Johnny von Neumann always used to say, "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." So I don't find the numerical agreement very impressive either.' So I said, 'Thank you very much for you help,' and I said goodbye. There was nothing more to be said. The whole discussion took maybe 10 or 15 minutes. And I came back to Cornell to tell the team the bad news. So that was another watershed in my life, and I think it was profoundly useful what Fermi did. He had this amazing intuition. He could spot what was good and what was bad right away. I mean, we might have worked on these calculations for five years if Fermi hadn't given us the red light and, as it was, Fermi was absolutely right because in the end of course it turned out that the theory on which we based the whole calculation was an illusion. There is really no such thing as a pseudo-scalar theory of pions. In reality 10 years later, or whenever it was, quar

Why it's here

2 people recommend this

What people say. Frequently cited by engineers and data scientists as the origin of the famous quote about fitting an elephant with four parameters.

EdwardCoffinon Hacker News • Recommends

On “Fitting an elephant with four non-zero parameters”

Freeman Dyson recounts the episode 1 that inspired this paper in his web of life interviews (prepositioned to the fitting an elephant bit) 2 1 link to this video 2 link to this video

Calavaron Hacker News • Recommends

On “Fitting an elephant with four non-zero parameters”

> > In 1953, Enrico Fermi criticized Dyson’s model by quoting Johnny von Neumann: “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” For those who are interested, you can watch Freeman Dyson recount this conversation in his own words in an interview: link to this video

magicalhippoon Hacker News • Mentions

On “Study: Dark matter does not exist and the universe is 27B years old”

Full paper here1, and here's2 the earlier paper about the model itself. They overloaded the CCC acronym, which has also been used for Penrose's Conformal cyclic cosmology3, though can't say which is more speculative. I'm also reminded of Von Neumann's elephant4, given that they vary the constants it doesn't seem that surprising I suppose that they can get a fit. Freeman Dyson learned this the hard way5. This is just one test, the article mentions a few others that they'll need to pass to be a serious contender. Will be interesting to see how it goes. 1: link 2: link 3: link 4: link 5: link to this video

kayo_20211030on Hacker News • Mentions

On “What Is a Particle? (2020)”

I found the reference. It was an interview with Freeman Dyson where he said "quarks were invented", and I don't think it's a slip of the tongue. It's at 3:17 in this video of an interview with him link to this video It's representative of a view that there's a thing (a particle) that explains another thing (a force) that was consistent with both theory and experiment. Thus, a quark could be a particle, and it was whatever the experimentalists and theorists said it was, however it was measured or contemplated. For a deeper meaning it becomes an exercise in hermeneutics i.e what does it mean when we say "particle"? That was the point of the original piece - there is no uncontested view of a particle's fo…

godelskion Hacker News • Mentions

On “A statistical approach to model evaluations”

…ty of problems that need to be solved in which there is no algorithmic procedure. Where there is no explicit density function. Intractable, doubly intractable, and more problems. Maybe we're just too dumb, but it's clear there are plateaus where luck is needed to advance. I do not believe our current machines would be capable of closing a gap. 0 link to this video…

godelskion Hacker News • Mentions

On “The impact of competition and DeepSeek on Nvidia”

Interestingly a lot of the math and physics people in the ML community are considered "grumpy researchers." A joke apparent with this starter pack0. From my personal experience (undergrad physics, worked as engineer, came to CS & ML because I liked the math), there's a lot of pushback. - I've been told that the math doesn't matter/you don't need math. - I've heard very prominent researchers say "fuck theorists" - I've seen papers routinely rejected for improving training techniques with reviewers say "just tune a large model" - I see papers that show improvements when conditioning comparisons on compute restraints because "not enough datasets" or "but does it scale" (these questions can al…

godelskion Hacker News • Mentions

On “Understanding Reasoning LLMs”

I don't think this makes sense and I'm not quite sure why you went to ML, but that's okay. I am a machine learning researcher, but also frustrated with the state of machine learning, in part because, well... you can probably see how "proof by empirical evidence" is dialed up to 11. Sorry, long answer incoming. It is far from complete too but I think it will help build strong intuition around your questions. Will knowledge transfer? That entirely depends on the new problem. It also entirely depends on how related the problem is. But also, what information was used to solve the pre-transfer state. Take LLMs for example. There's lots of works that have shown them being difficult to train for so…

godelskion Hacker News • Mentions

On “ARC-AGI without pretraining”

> I feel like extensive pretraining goes against the spirit of generality. What do you mean by generality? Pretraining is fine. It is even fine for pursuit to AGI. Humans and every animal has "baked in" memory. You're born knowing how to breath and have latent fears (chickens and hawks). Generalization is the ability to learn on a subset of something and then adapt to the entire (or a much larger portion) of the superset. It's always been that way. Humans do this, right? You learn some addition, subtraction, multiplication, and division and then you can do novel problems you've never seen before that are extremely different. We are extremely general here because we've learned the causal rule…

godelskion Hacker News • Mentions

On “The cultural divide between mathematics and AI”

> the primary aim isn't really to find out whether a result is true but why it's true. I'm honestly surprised that there are mathematicians that think differently (my background0). There are so many famous mathematicians stating this through the years. Some more subtle like Poincare stating that math is not the study of numbers but the relationship between them, while others far more explicit. This sounds more like what I hear from the common public who think mathematics is discovered and not invented (how does anyone think anything different after taking Abstract Algebra?). But being over in the AI/ML world now, this is my NUMBER ONE gripe. Very few are trying to understand why things are…

godelskion Hacker News • Mentions

On “Pen and Paper Exercises in Machine Learning (2022)”

Just curious, how "deep" have you gone into the theory? What resources have you used? How strong is your math background? Unfortunately a lot of the theory does require some heavy mathematics, the type you won't see in a typical undergraduate degree even for more math heavy subjects like physics. Topics such as differential geometry, metric theory, set theory, abstract algebra, and high dimensional statistics. But I do promise that the theory helps and can build some very strong intuition. It is also extremely important that you have a deep understanding of what these mathematical operations are doing . It does look like this exercise book is trying to build that intuition, but I haven't rea…

godelskion Hacker News • Mentions

On “The "AI 2027" Scenario: How realistic is it?”

I do say the same think about JKR, btw. And for the same things, because the content she writes. I think you focused on the fanfic part and not the part where I'm criticizing where they say their stuff is the most important to keep humanity alive and they're charging money for it. Meanwhile, you may notice in academia we publish papers to make them freely available, like on arXiv. If it is that important that people need to know, you make it available. The second person, Hinton, is not as good of an authority as you'd expect. Though I do understand why people take him seriously. Fwiw, his Nobel was wildly controversial. Be careful, prizes often have political components. I have degrees in bo…

godelskion Hacker News • Mentions

On “Chomsky on what ChatGPT is good for (2023)”

> The fact that we have figured out how to translate language into something a computer can "understand" should thrill linguists. I think they are really excited by this. There seems no deficiency of linguists using these machines. But I think it is important to distinguish the ability to understand language and translate it. Enough that you yourself put quotes around "understanding". This can often be a challenge for many translators, not knowing how to properly translate something because of underlying context. Our communication runs far deeper than the words we speak or write on a page. This is much of what linguistics is about, this depth. (Or at least that's what they've told me, since…

godelskion Hacker News • Mentions

On “Chomsky on what ChatGPT is good for (2023)”

I think many people are missing the core of what Chomsky is saying. It is often easy to miscommunicate and I think this is primarily what is happening. I think the analogy he gives here really helps emphasize what he's trying to say. If you're only going to read one part, I think it is this: | I mentioned insect navigation, which is an astonishing achievement. Insect scientists have made much progress in studying how it is achieved, though the neurophysiology, a very difficult matter, remains elusive, along with evolution of the systems. The same is true of the amazing feats of birds and sea turtles that travel thousands of miles and unerringly return to the place of origin. | Suppose Tom Jo…

godelskion Hacker News • Mentions

On “Persona vectors: Monitoring and controlling character traits in language models”

> only a small subset will contain actual understanding, a property that is independent of the model's input-output behavior? I think this is close enough. I'd say "a model's ability to make accurate predictions is not necessarily related to the model's ability to generate counterfactual predictions." I'm saying, you can make extremely accurate predictions with an incorrect world model. This isn't conjecture either, this is something we're extremely confident about in science. > I don't think I can agree. In your 2+a+b-2 example, that is an unnecessarily convoluted, but entirely correct model of addition. I gave it as a trivial example, not as a complete one (as stated). So be careful with e…

godelskion Hacker News • Mentions

On “Andrej Karpathy – It will take a decade to work through the issues with agents”

> that does not have a technical meaning I don't think the definition is very refined, but I think we should be careful to differentiate that from useless or meaningless. I would say most definitions are accurate, but not precise. It's a hard problem, but we are making progress on it. We will probably get there, but it's going to end up being very nuanced and already it is important to recognize that the word means different things in vernacular and in even differing research domains. Words are overloaded and I think we need to recognize this divergence and that we are gravely miscommunicating by assuming the definitions are obvious. I'm not sure why we don't do more to work together on this…

godelskion Hacker News • Mentions

On “Spatial intelligence is AI’s next frontier”

…t you'll look like you're progressing because you're better fitting it. But in the end the models will need to create some compact formulation representing concepts such as motion. Or in other words, a physics. And it's not like physicists aren't know for being detail oriented and nitpicky over nuances. That is breed into then with good reason 0 link to this video…