Skip to content
Hallway
0:00 / 0:00

#simdjson#simd#jsonparsing

Parsing JSON Really Quickly: Lessons Learned

InfoQ

234K subscribers • 58 videos on Hallway • Conference channel

YouTube
  • Recommended by 4 people: 3 on Hacker News and 1 on Lobsters.
  • A classic. People kept recommending it from 2020 to 2024.
  • Presented by Daniel Lemire

Advanced. Deep dive for people who know the field.

85K views6 years ago4 recommendations

Systems programmingComputer sciencePerformance and optimizationAlgorithmsData structures

Daniel Lemire shares lessons from building simdjson, explaining how to use SIMD instructions and avoid branch mispredictions for ultra-fast parsing.

QCon San Francisco, the international software conference, returns November 17-21, 2025. Join senior software practitioners from early adopter companies as they share real-world insights and actionable advice to help you adopt the right technologies and practices. Gain exposure to innovative approaches and fresh ideas in software development and engineering, designed to inspire and challenge you. Don’t miss this chance to deepen your knowledge, sharpen your skills, and stay ahead in the ever-evolving world of technology. Register now: bit.ly/3BYzfbe --------------------------------------------------------------------------------------------------------------------------------------------------- Video with transcript included: bit.ly/2QSfyHQ Daniel Lemire talks about the lessons learned while writing the fast JSON parser, simdjson. One of the most important lessons is the importance of a nearly obsessive focus on performance metrics - to constantly measure the impact of the choices. This presentation was recorded at QCon San Francisco 2019: bit.ly/38sivWf The next QCon is QCon London 2020 – March 2-4, 2020: bit.ly/2VfRldq For more awesome presentations on innovator and early adopter topics check InfoQ’s selection of talks from conferences worldwide bit.ly/2tm9loz #JSON #Performance #Parsing

Why it's here

4 people recommend this

What people say. Recommended for its fascinating insights into low-level code optimization, vectorization, and parsing gigabytes of JSON per second.

dehesaon Hacker News • Recommends

On “So you want to write your own CSV code (2014)”

…hole of edge cases and configurability. Looking back (and only thinking on productivity) my time would have been better spent doing something else. I did, however, learn quite a lot along the way: - how text is represented in a computer (and all its caveats), - how to identify and tune slow code, - how to vectorize code (thanks to Daniel Lemire's link to this video ), - how to encapsulate code better, - how to use some great CSV tools out there (thanks to Leon's link ) - how to better manage and communicate an open source project. All in all, I am still not sure it was a mistake. Moreover when Swift didn't really had, at the time, a good Codable interface to CSV. Next time I encounter a similar prob…
View on Hacker News

Rendelloon Hacker News • Recommends

On “Word-Aligned Bloom Filters”

I find these optimizations fascinating. Anyone familiar with Lemire likely knows about this, but I listened to a podcast episode1 with him a few days ago and learned about `simdjson`, the tool he authored that parses JSON at 25x the speed of the standard C++ library.23 It's worth looking at if you're into this sort of thing. 1. link 2. link to this video 3. link
View on Hacker News

D4ckardon Hacker News • Recommends

On “The One Billion Row Challenge”

Daniel Lemire has an interesting talk on this: link to this video . They key takeaway is that the disk is rarely the bottleneck.
View on Hacker News

dbremneron Lobsters • Recommends

Parsing JSON Really Quickly: Lessons Learned (2020)
3View on Lobsters

avianon Hacker News • Mentions

On “Simdjson: Parsing gigabytes of JSON per second”

…equence of past 2000 decisions on the same branch instruction? Also, why would the performance increase incrementally like that? I would imagine that it would remember the loop history on the first run and achieve maximum performance on the second run. I doubt that there's really a neural net in the silicon doing this as the author speculates. 1 link to this video…
View on Hacker News

ubedanon Hacker News • Mentions

On “GigaToken: ~1000x faster Language model tokenization”

Github: link It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json. Daniel Lemire - One of the authors at QCon 2019: link to this video Another Youtube video that explains it: link
View on Hacker News

u/levodelellisin r/ProgrammingLanguages • Mentions

On “Lexing html very efficiently”

IMO lexing is usually the fastest part. Have you seen this? link to this video In the video he talks about using shuffle to classify characters and to remove quotes with a backslash before them. In a throwaway project try classifying text as variables (like `a`, `href` and `custom-attr`) and important tokens (like `<>'\"'=`). Your code may become faster just because you need less if statements to figure out what a character is
1View on Reddit