Skip to content
0:00 / 0:00

#lucene#elasticsearch#invertedindex

What is in a Lucene index? Adrien Grand, Software Engineer, Elasticsearch

LuceneSolrRevolution

3.7K subscribers • 1 video on Hallway

YouTube
  • Recommended by 3 people: 3 on Hacker News.
  • Presented by Adrien Grand, Software Engineer at Elasticsearch

Advanced. Deep dive for people who know the field.

80K views12 years ago3 recommendations

Databases and dataComputer scienceSearch and indexingDatabase internalsData structures

Adrien Grand explains the internal design decisions of Apache Lucene, detailing how it stores data on disk for fast search and analytics.

Presented by Adrien Grand, Software Engineer, Elasticsearch Although people usually come to Lucene and related solutions in order to make data searchable, they often realize that it can do much more for them. Indeed, its ability to handle high loads of complex queries make Lucene a perfect fit for analytics applications and, for some use-cases, even a credible replacement for a primary data-store. It is important to understand the design decisions behind Lucene in order to better understand the problems it can solve and the problems it cannot solve. This talk will explain the design decisions behind Lucene, give insights into how Lucene stores data on disk and how it differs from traditional databases. Finally, there will be highlights of recent and future changes in Lucene index file formats.

Why it's here

3 people recommend this

What people say. Recommended as an instructive and easy-to-follow deep dive into Lucene internals and index structures.

drejon Hacker News • Recommends

On “Elasticsearch from the Bottom Up (2013)”

I can super highly recommend talks by Adrien Grand on Lucene and Elasticsearch. They are super instructive yet easy to follow. link to this video link

ddorian43on Hacker News • Recommends

On “Challenges in Implementing a Full-Text-Search Engine”

This is extremely high level. A nice view of how lucene internals work by core committer Adrien Grand: link link to this video

stereosteveon Hacker News • Recommends

On “Reducing search indexing latency to one second”

This is excellent. I was recently reviewing Lucene concepts and found this video really good: link to this video Also this site has a series of Lucene articles that are pretty nice. The one on Term Vectors in particular: link Based on some quick research it seems like Lucene is already using a sorted skip data structure for the posting list, so I wonder why they had to do a custom implementation? Perhaps it has to do with their custom Document ID scheme and how they want to preserve order in the Posting List being different from the default behavior. It also sounds like searchers are searching on indexes as they're being written, and there is some custom coordination around visibility, which might re…