Skip to content
Hallway
0:00 / 0:00

#instrumentalconvergence#aisafety#alignment

Why Would AI Want to do Bad Things? Instrumental Convergence

Robert Miles AI Safety

174K subscribers • 9 videos on Hallway

YouTube
  • Recommended by 6 people: 6 on Hacker News.
  • A classic. People kept recommending it from 2019 to 2023.
  • Presented by Robert Miles, AI safety communicator
  • May be out of date. Uploaded 8 years ago, in an area where tools change fast.

Beginner. No background in the topic needed.

274K views8 years ago6 recommendations

AI and machine learningAI safety and impact

Robert Miles explains the concept of instrumental convergence and why an AGI with almost any goal would pursue self-preservation and resource acquisition.

How can we predict that AGI with unknown goals would behave badly by default? The Orthogonality Thesis video: youtube.com/watch?v=hEUO6pjwFOo Instrumental Convergence: arbital.com/p/instrumental_convergence/ Omohundro 2008, Basic AI Drives: selfawaresystems.files.wordpress.com/2008/01/ai_drives_final… With thanks to my excellent Patrons at patreon.com/robertskmiles : Jason Hise Steef Jason Strack Chad Jones Stefan Skiles Jordan Medina Manuel Weichselbaum 1RV34 Scott Worley JJ Hepboin Alex Flint James McCuen Richárd Nagyfi Ville Ahlgren Alec Johnson Simon Strandgaard Joshua Richardson Jonatan R Michael Greve The Guru Of Vision Fabrizio Pisani Alexander Hartvig Nielsen Volodymyr David Tjäder Paul Mason Ben Scanlon Julius Brash Mike Bird Tom O'Connor Gunnar Guðvarðarson Shevis Johnson Erik de Bruijn Robin Green Alexei Vasilkov Maksym Taran Laura Olds Jon Halliday Robert Werner Paul Hobbs Jeroen De Dauw Konsta William Hendley DGJono robertvanduursen Scott Stevens Michael Ore Dmitri Afanasjev Brian Sandberg Einar Ueland Marcel Ward Andrew Weir Taylor Smith Ben Archer Scott McCarthy Kabs Kabs Phil Tendayi Mawushe Gabriel Behm Anne Kohlbrenner Jake Fish Bjorn Nyblad Jussi Männistö Mr Fantastic Matanya Loewenthal Wr4thon Dave Tapley Archy de Berker Kevin Vincent Sanders Marc Pauly Andy Kobre Brian Gillespie Martin Wind Peggy Youell Poker Chen Kees Darko Sperac Paul Moffat Noel Kocheril Jelle Langen Lars Scholz

Why it's here

6 people recommend this

What people say. Recommended as one of the clearest and most accessible introductions to the core arguments of AI alignment and safety.

ludwigschuberton Hacker News • Recommends

On “Emergent Tool Use from Multi-Agent Interaction”

...and many would say that’s because us humans are bad at imagining optimizing agents without anthropomorphizing them. This is a reasonable, even typical suspicion that many people share! The best explanation I know of why it’s unfortunately wrong is by Robert Miles in a video, but if you prefer a more thorough treatment, you could also read about “instrumental convergence” directly. If you find a flaw in this idea, I’d be interested to hear about it! :) Robert Miles’ video: link to this video Instrumental Convergence: link Now afaik nothing in this argument says that we can’t find a way to control this in a more complex formalism-but we clearly haven’t done so yet.
View on Hacker News

Veedracon Hacker News • Recommends

On “Statement on AI Risk”

…ss intentionally preserved; we don't know how to make AI systems encode any complex preference robustly. Robert Miles' videos are among the best presented arguments about specific points in this list, primary on the alignment side rather than the capabilities side, that I have seen for casual introduction. Eg. this one on instrumental convergence: link to this video Eg. this introduction to the topic: link He also has the community-led AI Safety FAQ, link , which gives brief answers to common questions. If you have specific questions I might be able to point to a more specific argument at a higher level of depth.…
View on Hacker News

DirkHon Hacker News • Recommends

On “The greatest risk of AI is from the people who control it, not the tech itself”

…discussion, but nobody actually takes the thought experiment seriously as a "real-world" thing. If you actually want to engage with the argument in good faith for why an AI might indeed be smart enough to wipe us out but "dumb enough" to just pursue some other goal ("make paperclips as an extreme example") there is a great video by Rob Miles here: link to this video Point out the flaws in the reasoning. Just saying "this is nonsense" does nothing but prove you've never actually taken the time to understand the best arguments. Also... Alan Turing, Geoffrey Hinton... Extremely influential and intelligent people take/took this seriously. These are not SciFi fanboys. AI doom only became sci-fi after smart…
View on Hacker News

sabellitoon Hacker News • Recommends

On “John Carmack's new AGI company, Keen Technologies, has raised a $20M round”

There's a wondeful youtube channel from a researcher who focusses exactly on this topic, I think you should check it out: link to this video
View on Hacker News

jbay808on Hacker News • Recommends

On “Chalmers: Is there a canonical source for “the argument for AGI ruin” somewhere”

…t to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this1, as well as a paper by Deepmind presenting one possible solution2. 1 link 2 link Good explainer video by Robert Miles: link to this video…
View on Hacker News

pixl97on Hacker News • Recommends

On “Nvidia Trains LLM on Chip Design”

Robert Miles answers your question link to this video The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
View on Hacker News

pixl97on Hacker News • Mentions

On “Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario”

…ht in loops pretty easily and stops when using tools. But lets imagine a super intelligence that can only communicate over the internet, what can it accomplish? Are there places you can sign up for banking accounts without a physical presence? If so, then it could open accounts and move money. Money is the ultimate form of instrumental convergence link to this video (Robert Miles) Once you have money you can buy things, like more time on AWS. Or machine parts from some factory in China, or assembly in Mexico. None of these require an in person presence and yet real items exist because of the digital actions. At the end of the day money is all you need, and being super intelligent seems like a good path…
View on Hacker News

digbybkon Hacker News • Mentions

On “Sam Altman: Why You Should Fear Machine Intelligence (2015)”

link to this video Basically, any sufficiently advanced agent will have “prevent anyone from switching me off” as an instrumental goal. And any sufficiently advanced intelligence will come up with ways of achieving its goals that lesser intelligences can’t predict. Taking steps to prevent itself from being turned off could be its first order of business, not necessarily only as the human is about to hit the off switch. While we can’t predict what a greater intelligence would come up with, the obvious thing is social engineering. Remember that one of the factors that lead to the Jan 6th attack on the capital was that a random text generating agent named “Q” (we know it was a human because it was before…
View on Hacker News