Skip to content
0:00 / 0:00

#teslaautopilot#computervision#neuralnetworks

[CVPR'21 WAD] Keynote - Andrej Karpathy, Tesla

WAD at CVPR

4.8K subscribers • 1 video on Hallway

YouTube
  • Recommended by 4 people: 4 on Hacker News.
  • Presented by Andrej Karpathy, Senior Director of AI at Tesla
  • An opinion. It argues one side of a question engineers disagree on.

Intermediate. Assumes working knowledge.

51K views5 years ago4 recommendations

AI and machine learningSystems programmingComputer visionNeural networks and deep learningTraining and serving models

Andrej Karpathy explains how Tesla transitioned Autopilot to a vision-only neural network stack and why they removed radar.

Talk given on 2021/06/20. Andrej is the Senior Director of AI at Tesla, where he leads the team responsible for all neural networks on the Autopilot. Previously, Andrej was a Research Scientist at OpenAI working onDeep Learning in Computer Vision, Generative Modeling and Reinforcement Learning. Andrej received his PhD from Stanford, where he worked with Fei-Fei Li on Convolutional/Recurrent Neural Network architectures and their applications in Computer Vision, Natural Language Processing and their intersection. Over the course of his PhD, Andrej squeezed in two internships at Google where he worked on large-scale feature learning over YouTube videos, and in 2015 he interned at DeepMind and worked on Deep Reinforcement Learning. Together with Fei-Fei, Andrej designed and taught a new Stanford class on Convolutional Neural Networks for Visual Recognition (CS231n). The class was the first Deep Learning course offering at Stanford and has grown from 150 enrolled in 2015 to 330 students in 2016, and 750 students in 2017.

Why it's here

4 people recommend this

What people say. Engineers recommend this talk as an excellent explanation of Tesla's computer vision architecture and the engineering reasons for dropping radar.

fossuseron Hacker News • Recommends

On “Waymo doubles service area in Phoenix Metro”

This talk was worth watching (any Karpathy or Carmack talk is imo): link to this video Specifically this is the relevant part: link to this video They removed radar because it was making things worse.

simondotauon Hacker News • Recommends

On “Tesla’s ‘Full Self-Driving’ Beta Software Used on Public Roads Lacks Safeguards”

If you're interested, Tesla's head of AI recently did a public presentation of their new vision based depth system. It's worth a watch if you find this stuff interesting and enjoy learning about the forefront of technology. link to this video A key takeaway is that they've been running this stack in shadow mode (validating output but not controlling the car) on everyone's Tesla for quite some time. Equivalent to 1000 YEARS worth of real world driving. And from this data they've proven it is now superior to radar in all circumstances.

ggreeron Hacker News • Recommends

On “Tesla FSD Beta almost causes a head-on collision [video]”

…"no obstacles detected" when the car in front of you is slamming on the brakes. Again, you have to use the vision cameras to figure out what's actually going on. The people at Tesla decided that the radar was creating more noise than it was helping with, so they removed radar last year. Andrej Karpathy explains it in far more detail in this talk: link to this video…

JoshColeon Hacker News • Recommends

On “Tesla FSD Beta almost causes a head-on collision [video]”

…on. If it wasn't possible to do better than human with only human senses, we wouldn't be able to point to many different situations in which driving with an AI clearly outperforms humans, but we can. For at least those subsets of AI driving, we can say that an AI driver will outperform humans. > I can't understand why they removed radar. Check out link to this video for a very detailed breakdown…

fossuseron Hacker News • Mentions

On “Grand jury indicts Trevor Milton, Nikola founder, on three counts of fraud”

FSD was advertised as paying for something that would come in the future - I bought it under that assumption (and so did everyone I know that has it). That was and remains true. They've continued to make progress towards it: link to this video Autopilot is different and isn't advertised as level 5 - 'autopilot' implies fancy lane keeping not turn by turn point to point autonomy. Elon's FSD timelines were wrong - that isn't fraud. People say everything he's trying to do won't work constantly, just because they're occasionally right doesn't validate all of the times they're wrong (e.g. Model 3 is vaporware will never ship, the company is bankrupt and will close in two weeks, the Model S will never ship,…

fossuseron Hacker News • Mentions

On “Grand jury indicts Trevor Milton, Nikola founder, on three counts of fraud”

They're not just 'tweaking some settings': link to this video What they have today isn't FSD (which they state explicitly), but I'm happy to give them extra money early given the work they're doing on the chance that it works in the future. I think they have the best chance of success of actually shipping something real.

DeBraidon Hacker News • Mentions

On “Why Teslas Keep Striking Parked Firetrucks and Police Cars”

IIRC, in this1 video Andrej Karpathy mentions radar was mostly redundant, and in addition was occasionally harmful (sending conflicting signals vs. video), so they got rid of it to reduce cost and complexity. 1: link to this video

ggreeron Hacker News • Mentions

On “Tesla is outrunning the supply chain crunch”

No. Newer versions of Tesla's software disable the radar on cars that shipped with it.1 Vision has gotten so much better (thanks in part to training from radar data) that radar adds noise to their model of the outside world. Andrej Karpathy explained the issues in a talk this summer.2 1. link 2. link to this video

simondotauon Hacker News • Mentions

On “Tesla is outrunning the supply chain crunch”

Given that radar is functionally blind1 to static objects when travelling at high speeds, I'd argue it's not just a good thing, it's a necessary thing. Tesla have demonstrated the superiority of their vision approach and validated it with 1,000 years of real-world driving, running it in shadow mode on customer vehicles.2 1 Technically radar isn't blind to static objects, but since it cannot disambiguate them from tunnel entrances and highway overpasses, the data is useless. Therefore it's functionally blind. 2 link to this video

JoshColeon Hacker News • Mentions

On “Tell HN: Tesla rear-ended me on autopilot, no one will investigate”

From the keynote of CVPR. CVPR is the premier annual computer vision event comprising the main conference and several co-located workshops and short courses. link link to this video

simondotauon Hacker News • Mentions

On “Tesla to recall vehicles that may disobey stop signs”

> why not use binocular forward facing vision? Because Tesla have demonstrated that it's unnecessary. The depth information they are getting from the forward-facing camera is exceptional. Their vision stack now produces depth information that is dramatically superior to that from a forward-facing radar. link to this video (It's also worth noting that depth information can be validated when the vehicle is in motion, because a camera in motion has the ability to see the scene from multiple angles, just like a binocular configuration. This is how Tesla trains the neural networks to determine depth from the camera data.)

aeternumon Hacker News • Mentions

On “Tesla Vision Update: Replacing Ultrasonic Sensors with Tesla Vision”

Karpathy covered it during a conference keynote here: link to this video It also is pretty common with other vehicles, they just don't get as much press as Tesla: link link

onethoughton Hacker News • Mentions

On “Even after $100B, self-driving cars are going nowhere”

Appreciate the downvote/disagreement. But here is Karpathy explaining how vision can be used to measure distance to objects accurately0 Here is the fact that LiDAR doesn't work in the rain1: "... In heavy rain, for example, the light pulses emitted from the lidar system are partially reflected off of rain droplets which adds noise to the data, called 'echoes'." Which logically implies you need to revert to vision, as see 0 also for why Radar is unreliable. 0 link to this video 1 link

JoshColeon Hacker News • Mentions

On “Tesla removes parking sensors, the results are predictably terrible”

He is talking about the long technical talk at CVPR. link to this video For example look at this section: link to this video Also if you look into it you will find Google's self-driving car project ran into similar issues. They upgraded the radar in response to the issues rather than removing it, but there are actual issues. Vision only was better than radar plus vision in their stack. For real world situations that frequently come up in driving radar wasn't worth it. The theoretical accuracy of the sensor fusion didn't happen in practice because in practice you needed to have vision tell you when sensor fusion wasn't going to be accurate. It isn't naive sensor fusion as in toy math models of sensor fusion. So wh…