Skip to content
Hallway
0:00 / 0:00

#rlhf#rewardmodeling#reinforcementlearning

Training AI Without Writing A Reward Function, with Reward Modelling

Robert Miles AI Safety

174K subscribers • 9 videos on Hallway

YouTube
  • Recommended by 3 people: 3 on Hacker News.
  • A classic. People kept recommending it from 2019 to 2023.
  • Presented by Robert Miles, AI safety communicator
  • Linked in the descriptionarXiv 1706.03741
  • May be out of date. Uploaded 6 years ago, in an area where tools change fast.

Intermediate. Assumes working knowledge.

259K views6 years ago3 recommendations

AI and machine learningReinforcement learningAI safety and impactNeural networks and deep learning

Robert Miles explains how reinforcement learning agents can learn from human preferences instead of hand-written reward functions.

How do you get a reinforcement learning agent to do what you want, when you can't actually write a reward function that specifies what that is? The paper: arxiv.org/pdf/1706.03741.pdf The blogpost: openai.com/blog/deep-reinforcement-learning-from-human-prefe… Thanks to my wonderful patrons: patreon.com/robertskmiles James Gladamas Steef Scott Worley Jordan Medina Simon Strandgaard JJ Hepboin Pedro A Ortega Said Polat Chris Canal Jake Ehrlich Kellen lask Francisco Tolmasky Michael Andregg David Reid Robert Daniel Pickard Peter Rolf Chad Jones Richárd Nagyfi Jason Hise Phil Moyer Shevis Johnson Erik de Bruijn Alec Johnson Clemens Arbesser Ludwig Schubert Bryce Daifuku Allen Faure Eric James Qeith Wreid Jonatan R Ingvi Gautsson Michael Greve Julius Brash Tom O'Connor Robin Green Laura Olds Jon Halliday Paul Hobbs Jeroen De Dauw Lupuleasa Ionuț Tim Neilson Eric Scammell Igor Keller Ben Glanton anul kumar sinha Sean Gibat Cooper Lawton Will Glynn Tyler Herrmann Tomas Sayder Ian Munro Jérôme Beaulieu Nathan Fish Taras Bobrovytsky Anne Buit Vaskó Richárd Sebastian Birjoveanu Euclidean Plane Andrew Harcourt DGJono robertvanduursen Dmitri Afanasjev Marcel Ward Andrew Weir Ben Archer Kabs Miłosz Wierzbicki Tendayi Mawushe Jannik Olbrich Anne Kohlbrenner Jussi Männistö Wr4thon Martin Ottosen Archy de Berker Marc Pauly Andy Kobre Brian Gillespie Poker Chen Kees Darko Sperac Truls Paul Moffat Anders Öhrt Marco Tiraboschi Michael Kuhinica Fraser Cain Robin Scharf Seth Brothwell Kasper Schnack Klemen Slavic Patrick Henderson Oct todo22 Melisa Kostrzewski Hendrik Daniel Munter Graham Henry Duncan Orr Bryan Egan Robert Hildebrandt James Fowkes Alan Bandurka Ben H Tatiana Ponomareva Michael Bates Simon Pilkington Dion Gerald Bridger Petr Smital Daniel Kokotajlo Fionn Yuchong Li Diagon Parker Lund Paul Emmerich Russell schoen Andreas Blomqvist Bertalan Bodor David Morgan Jeremy Ben Schultz Zannheim Daniel Eickhardt lyon549 HD Ihor Mukha 14zRobot Ivan Arne Strasser Jason Cherry Igor (Kerogi) Kostenko Isaac Boates Thomas Dingemanse Davy Ker Alexander Brown Devon Bernard Ted Stokes James Helms Matheson Bayley patreon.com/robertskmiles

Why it's here

3 people recommend this

What people say. Recommended as the clearest explanation of reinforcement learning from human feedback and reward modeling.

sillysaurusxon Hacker News • Recommends

On “Autonomous DeLorean drives sideways to move forward”

I would highly recommend link to this video as a way of framing the problem. It seems like our options for ML are currently: self-play, dataset fitting, or human feedback. It's possible to solve any problem that can fit into one of those molds. (Unfortunately the opposite is true: if a problem doesn't fit into those categories, it seems very hard to solve with any form of ML.) But, now that I think about it a bit more, the sentence "use a trained net as a tool for building a controller with known continuity properties" seems neither obvious nor straightforward to do. I'm decently well-versed in AI techniques at this point, and I can't really think how I'd sit down and write a program that would generat…
View on Hacker News

cubefoxon Hacker News • Recommends

On “What We Know About LLMs (A Primer)”

Specifically about RLHF, I find this video by Rob Miles still the best presentation of the ingenious original 2017(!) paper: link to this video RLHF is actually older than GPT-1, which came out in 2018. It didn't get applied to language models until 2022 with InstructGPT, an approach which combined supervised instruction fine-tuning with RLHF.
View on Hacker News

afarrellon Hacker News • Recommends

On “Ask HN: How do I get out of the procrastination abyss?”

…hould talk with a human. 2. Make sure you're taking care of your body and brain: exercise, good food, good sleep (this is easier if you're getting exercise), meditation, etc. 3. Understand what your "reward model" is. Under what conditions do you recognize "Aha! I've made progress!"? To really understand what I'm getting at here, watch this video: link to this video 4. Find internal things to strengthen your reward model's signals. Go back to when you originally started learning ML and Re-connect to your "Why". Do you still find those reasons compelling? If so, take some time to bring some vividness to your picture. -- Start With Why. 5. Find external punishments to remove. Now that you've spent 8 mont…
View on Hacker News

Edoardo Guerrieroon Artificial Intelligence Stack Exchange • Mentions

On “Why are reinforcement learning methods sample inefficient?”

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model of the environment. In general I would say that there are two issues related to sample efficiency: 1 the size of the 'action'+'environment states' space 2 the exploration strategy used. Regarding the first point, in reinforcement learning is really easy to encounter situations in which the number of combinations of possible actions and possible environment states explode, becoming intractable. Lets for example consider the Atari game from the Rainbow paper you linked: the environment in which…
5View the answer

afarrellon Hacker News • Mentions

On “Persisting as a solo founder”

as an ADHDer, one thing I've found key is figuring out what things in my environment and mind are the things I want to signal "you're doing the right thing, keep going." Essentially, what is your 'reward model'? -- link to this video Test-Driven-Development is one example of this.
View on Hacker News

afarrellon Hacker News • Mentions

On “Setting goals for 2021 – A brief guide about personal goal setting”

…brain, he'll say it is not a signal of reward, but of anticipated reward. It does spike when you recognise a surprisingly strong reward, (like when you hit a jackpot or an unexpectedly interesting HN link) to help you learn to repeat the path to that reward2, but mostly it signals "Yep, you're on the right path. Now go make that test green". 1 link to this video 2 link…
View on Hacker News

afarrellon Hacker News • Mentions

On “Ask HN: How To Overcome Information Addiction?”

A. Meet your basic biological needs. Your brain is a part of your body. A person with an iron deficiency can feel compelled to chew ice1. Ice does not cures anemia, brains are weird parts of the body and do weird things when the body is not getting what it needs. So, get better sleep, food, and exercise. Prof Andrew Huberman has some good tips2. B. Decide what few things you really value and allocate your time to those on purpose. If you find yourself repeatedly drawn to the news, maybe you are genuinely give a fuck about civic engagement. Your personality has a need for it. Trying to meet that need by scrolling twitter is like eating 3 bags of tortilla chips for dinner -- quick-access,…
View on Hacker News