Skip to content
0:00 / 0:00

#python#hashtable#raymondhettinger

Modern Dictionaries by Raymond Hettinger

SF Python

19K subscribers • 5 videos on Hallway

YouTube
  • Recommended by 9 people: 9 on Hacker News.
  • A classic. People kept recommending it from 2016 to 2023.
  • Presented by Raymond Hettinger, Python core developer

Intermediate. Assumes working knowledge.

149K views9 years ago9 recommendations

Programming languagesComputer sciencePythonData structuresPerformance and optimization

Raymond Hettinger explains the evolution of Python's dictionary implementation into the compact and ordered version introduced in Python 3.6.

Abstract Python's dictionaries are stunningly good. Over the years, many great ideas have combined together to produce the modern implementation in Python 3.6. This fun talk is given by Raymond Hettinger, the Python core developer responsible for the set implementation and who designed the compact-and-ordered dict implemented in CPython for Python 3.6 and in PyPy for Python 2.7. He will use pictures and little bits of pure python code to explain all of the key ideas and how they evolved over time. He will also include newer features such as key-sharing, compaction, and versioning. This talk is important because it is the only public discussion of the state of the art as of Python 3.6. Even experienced Python users are unlikely to know the most recent innovations. Who and Why (Audience): This talk is for all Python programmers. It is designed to be fully understandable for a beginner (it starts from first principles) but to have new information even for Python experts (how key-sharing works, how the compact-ordered patch works, how dict versioning works). At the end of this talk, you can confidently say that you know how modern Python dictionaries work and what it means for your code. Bio Raymond Hettinger has also served as a director of the Python Software Foundation, and has mentored many people over the years on their contributions to the python-dev community. He is also well known for his contributions to the Python Cookbook, and shares many pieces of Python wisdom on Twitter. He is a frequent keynote speaker at Python Conferences around the world and has received the Distinguished Service Award at PyCon 2014 for his exceptional contributions to the python community. Other info: This talk is delivered at SF Python's 2nd Annual Holiday Party for Python Devs in SF Bay Area, CA. In you are in San Francisco area looking to meet other python devs, please check our schedule for meetups on sfpython.org

Why it's here

9 people recommend this

What people say. Highly recommended as a deep and insightful explanation of hash table optimization and Python dictionary internals.

tomnipotenton Hacker News • Recommends

On “Hash tables with open addressing”

Raymond Hettinger gave a presentation a few weeks ago on the evolution of the Python dict implementation - one of the most insightful videos I've seen in a long time. link to this video

svieiraon Hacker News • Recommends

On “Ask HN: What is your favourite tech talk?”

Python dictionaries: A Confluence of Great Ideas (also known as Modern Dictionaries) by Raymond Hettinger link to this video

AndrewOMartinon Hacker News • Recommends

On “Pre-Allocated Lists in Python”

link to this video Modern Dictionaries by Raymond Hettinger The official description of this talk is below, but personally, I'd watch any video with "Raymond Hettinger" in the title. > Python's dictionaries are stunningly good. Over the years, many great ideas have combined together to produce the modern implementation in Python 3.6. This fun talk is given by Raymond Hettinger, the Python core developer responsible for the set implementation and who designed the compact-and-ordered dict implemented in CPython for Python 3.6 and in PyPy for Python 2.7. He will use pictures and little bits of pure python code to explain all of the key ideas and how they evolved over time. He will also include newer featu…

Dowwieon Hacker News • Recommends

On “I Wrote My Fastest Hashtable”

Raymond Hettinger made an interesting talk about Python's dict (hash table) and how Python's core development team evolved it over time. He's giving the polished version of talk at PyCon this May but if you can't wait for it, here's a rough version: link to this video

StreakyCobraon Hacker News • Recommends

On “Migrating to Python 3 with pleasure”

This has been changed in Python 3.6 only, due to a change in dictionary implementation to make them more efficient. "Modern Dictionaries by Raymond Hettinger" 1 is a quite interesting and technical talk about these changes, explaining also the change in iteration order IIRC, worth a watch in my opinion! 1 link to this video

maxnoeon Hacker News • Recommends

On “Python 3 at Facebook”

Actually they optimized the hashmap, and it turned out a side effect of the optimization was insertion order. First implemented in pypy, then ported to cpython 3.6. Talk by Raimond Hettinger about the new implementation with several other implementations: link to this video

StreakyCobraon Hacker News • Recommends

On “Ask HN: What are some of the best technical talks you've heard?”

I don't know if it is part of the bests, but very interesting anyway: Modern Dictionaries by Raymond Hettinger 1 It's about Python 3.6 dictionaries improvements. 1 link to this video

bbrikon Hacker News • Recommends

On “Python dicts are now ordered”

As far as I know, it was implemented by Raymond Hettinger. This is a very interesting talk about the new tech: link to this video

crashbunnyon Hacker News • Recommends

On “Python dicts are now ordered”

Raymond Hettinger does or did a few talks on dictionaries too. He covers the history of dictionaries in python and how they evolved. It was mentioned in the article's comment section. link to this video

samwhoon Hacker News • Recommends

On “Hashing”

You're welcome! I deliberately didn't go in to that for a few reasons. 1. It would have made this article very, very long. 2. It's a bit out of scope for an article on hashing. 3. I think I might give hash maps their own article in future. Hash maps are fantastically deep. So many different ways to do it. You'll find a lot of material online but I'd recommend Raymond Hettinger's talk on how Pythons hash map data structure has evolved over time: link to this video .

StreakyCobraon Hacker News • Mentions

On “Comprehensive Python Cheatsheet (2018)”

I think this behavior will only be working starting from Python 3.6, as the dictionary obtained by calling `dict.fromkeys()` before this would not keep the ordering 1 1 link to this video

pansa2on Hacker News • Mentions

On “Python Hash Tables: understanding dictionaries”

The talks by Brandon Rhodes are these two: * The Mighty Dictionary (PyCon 2010): link * The Dictionary Even Mightier (PyCon 2017): link There's also one by Raymond Hettinger, which seems to have been given first at a meetup and then, in shorter form, at PyCon 2017: * Modern Dictionaries: link to this video * Modern Python Dictionaries - A confluence of a dozen great ideas (PyCon 2017): link

yuy910616on Hacker News • Mentions

On “Flask 2.0”

3.6 had a couple of big improvement - f strings, annotation, and if I'm not mistaking, the devs did something to improve the dictionaries; from 3.6 on dicts are ordered. I don't know too much about it but raymond hettinger gave a talk about it. link to this video

quelltexton Hacker News • Mentions

On “Hash Ordering and Hyrum's Law”

…s the behavior happens since then because the implementation only tracks indices in the hash table itself and relies on maintaining entries separately in a second array that gets expanded in insertion order. This would also seem straightforward but it raises a few questions such as how deletion is implemented (efficiently). EDIT2: Okay, the talk ( link to this video ) mentions they just leave holes until the next resize (at around 42:00). Raymond also mentions there that his original idea didn't preserve ordering but happened due to an additional compacting optimization? Should probably watch the whole thing some time to get the history. Sounds like a fun talk.…

Rendelloon Hacker News • Mentions

On “Every change breaks someone's workflow (2013)”

I was going to make a tongue-in-cheek suggestion about randomizing undefined behaviour, but I suppose that's common in debugging and even seen in some APIs. At one time, in Python, dictionary (hashmap) ordering was randomized so users wouldn't rely on non-specified ordering behaviour. Eventually though, dictionaries were guaranteed insertion ordering. link to this video

Spivakon Hacker News • Mentions

On “Are Python Dictionaries Ordered Data Structures?”

The python dict() is already high-quality and high-performance. You're probably not going to be able to do much better than the current state. If you want faster you end up moving your data into an environment that lets you make more assumptions and be more restrictive with your data and operate on it there. It's how numpy, polaris, and pandas work. Everything in Python is a dict, there's no data structure that's been given more attention. link to this video