Andrej Karpathy (Tesla): CVPR 2021 (see description for better quality version)

I would really like to hear Tesla to defend their decision to not use them. Andrej Karpathy talks some about it in this (it's quite long, but the whole thing is quite interesting): link to this video

For others: Andrej Karpathy is the director of artificial intelligence and Autopilot Vision at Tesla. Was on front page yesterday for a presentation on Tesla's Autopilot / Autonomous features: link to this video

…tty much said autonomy 2.0 is all about end-to-end. I've spoken to a few people who actually do commercial autonomy, and they seemed more skeptical on whether end2end is the answer in the near-term. One idea we toy with is to use existing frozen architectures (OpenAI releases some and so do other big players) and do a small bit of fine-tuning. 1 link to this video 2 link…

link to this video In the above video Andrej Karpathy claims that LIDAR required previously compiled maps, the creation of which is not scalable. Do you disagree with that point of view?

Stereo vision is oversold in humans. You can easily drive with one eye. We use a lot of information to estimate depth including relative size of things, as well as “structure from motion” which is the relative movement of things over time. Tesla does estimate depth using structure from motion (and probably other cues). They call it virtual LIDAR in presentations. link to this video

IIRC there was a presentation from Karpathy talking about the challenges with sensor fusion, particular in resolving divergence between e.g. the vision and the radar stack: link to this video My background is in physics, but I find myself having a growing appreciate for the vision-only stack. It's really challenging building a formal understanding of the world that is robust to outliers that are so numerous as navigating in an urban environment. With vision, you have multiple kinds of information that are highly correlated (colour, spatial distribution, depth, etc) that are self-consistent. Whereas, fusing radar with vision, where object responses to radar are highly geometry & material dependent, is a…