Skip to content
Hallway
0:00 / 0:00

#chatgpt#rlhf#aialignment

ChatGPT with Rob Miles - Computerphile

Computerphile

2.6M subscribers • 70 videos on Hallway

YouTube
  • Recommended by 4 people: 4 on Hacker News.
  • Presented by Rob Miles, AI safety researcher

Intermediate. Assumes working knowledge.

517K views3 years ago4 recommendations

AI and machine learningTransformers and LLMsAI safety and impact

Rob Miles explains how ChatGPT was trained using reinforcement learning from human feedback and why this leads to confident nonsense.

A massive topic deserves a massive video. Rob Miles discusses ChatGPT and how it may not be dangerous, yet. More from Rob Miles: bit.ly/Rob_Miles_YouTube The 'Danish' Reddit post: reddit.com/r/GPT3/comments/zb4msc/speaking_to_chatgpt_in_per… Some of Rob's own videos which are relevant: Reward Modelling: youtube.com/watch?v=PYylPRX6z4Q Instrumental Convergence: youtube.com/watch?v=ZeecOKBus3Q Problems with Language Models: youtube.com/watch?v=w65p_IIp6JY facebook.com/computerphile twitter.com/computer_phile This video was filmed and edited by Sean Riley. Computer Science at the University of Nottingham: bit.ly/nottscomputer Computerphile is a sister project to Brady Haran's Numberphile. More at bradyharan.com

Why it's here

4 people recommend this

What people say. Recommended as an excellent explanation of how ChatGPT was trained with human feedback and the alignment issues that cause it to hallucinate.

mikewaroton Hacker News • Recommends

On “Ask HN: Are ChatGPT answers getting worse for anyone else?”

…removes all the other dimensions of consideration, and quashes it into a scalar. The optimum strategy for the rating system is to try to BS its way through things, which then teaches ChatGPT to BS its way through things as well. Much like rating systems on HN, and all social media, this destroys information and tends to dis-incentivize nuance. 1 link to this video 2 link…
View on Hacker News

mikewaroton Hacker News • Recommends

On “Princeton CS Prof: ChatGPT Is a Bullshit Generator (2022)”

It might help quite a bit to watch this overview of how ChatGPT works, and especially how it was trained .1 Human attention was too expensive for the training, so they made another AI to simulate the trainers. Unfortunately, the trainer simulator training didn't have an expressive enough set of options for the actual humans, which kept a lot of quite valuable feedback out of the whole system by reducing things to single ranking. If the person didn't know the correct answer, they'd likely just go based on gut. This in turn lead the training simulator to do the same. This lead ChatGPT to just try to bullshit an answer rather than stating it didn't know, and getting downvoted. 1 link to this video
View on Hacker News

f_devdon Hacker News • Recommends

On “Open Assistant: Conversational AI for Everyone”

It depends on your definition of cheating, but this is definitely not "using humans to answer all questions one could ask". Rather it's a way to tune language models to be more like assistants rather than "most likely continuation machines". For context I recommend watching the discussion on what chatgpt is/does0, and what the open assistant project's aim is1. 0: link to this video 1: link
View on Hacker News

simonhon Hacker News • Recommends

On “The new Bing will happily give you citations for a pile of nonsense”

I’ve read that before, it is fascinating. GPT3 was just trained on source text, but ChatGPT has undergone an an additional very rigorous programme of intensive training bootstrapped by human feedback. This has evolved it in order to improve its performance as an information assistant, especially on specific tasks that have proved problematic in the past. This video explains the approach and some of its limitations. link to this video
View on Hacker News

fxjon Hacker News • Recommends

On “What GPT-4 Does Is Less Like “Figuring Out” and More Like “Already Knowing””

…rence between a sonnet and a limmerick. But what happens is, that there is a miss-alignment between writing a poem and making the user happy. GPT is trained that its highest goal is to make the user happy. GPT believes that users dont know what sonnets are anyway so i can get away with a rhyme that is not a sonnet. Here is a good video about that: link to this video…
View on Hacker News

pixl97on Hacker News • Mentions

On “Open Assistant – project meant to give everyone access to a great chat based LLM”

link to this video Computerphile did an interview with Rob Miles a few days ago talking about model training, model size, and bulllshittery which he sums up in the last few moments of the video. Numerous problems exist in training that enhance bad behaviors. For example it appears that the people giving input on the responses may have a (Yes|No) voting system, but not a (Yes | No | I actually have no idea on this question) which appears it can create some interesting alignment issues.
View on Hacker News

pixl97on Hacker News • Mentions

On “Open Assistant – project meant to give everyone access to a great chat based LLM”

…typically provided by human input judging the output of the model. And we still see where this goes wrong in ChatGPT where the bot turns into a 'Yes Man' because it's aligned with giving an answer rather than saying I don't know even when it's confidence in the answer is low. Computerphile did a video on this in the last few days on this subject. link to this video…
View on Hacker News

spuzon Hacker News • Mentions

On “GPT-4”

…ls and using this abstract, pared down information, extrapolate the correct answer to the puzzle. I see the failure in the example with the goat the lion and the cabbage as simply a matter of overfitting. Edit: I see a lot of people saying "it doesn't understand logic; it's just predicting the next word." I'm basing my understanding on this video: link to this video The claim is that it would be impossible to feed enough input into a system such that it could produce anything as useful as ChatGPT unless it was able to abstract the underlying logic from the information provided. If you consider the he number of permutations of the animal crossing puzzle this quickly becomes clear. In fact it would be im…
View on Hacker News

fxjon Hacker News • Mentions

On “What GPT-4 Does Is Less Like “Figuring Out” and More Like “Already Knowing””

Also it is important to note that GPT shows large miss-alignments. The problem comes from the fact, that it is hard or impossible to give an objective what GPT should be optimized to (nobody knows the truth), so proxys are used. One proxy is that it should make the user happy and the user should give many thumbs up. But this does not mean that it has to give the "correct" answer, which the user himself might not know in the first place. So it invents things because during the reinforcement learning users were happy with these answers. A funny example is the github co-pilot which writes buggy code, because it thinks this is what the user wants. Here is a video about that: link to this video
View on Hacker News