The Design and Implementation of a Bitemporal DBMS – Håkan Råberg

Håkan Råberg discusses the architecture of Crux, a schemaless bitemporal database that uses Kafka for its commit log and serves Datalog queries.

Bitemporality allows you to modify, enhance and correct the view of the past while also keeping all versions for audit and repeatable reads. We see this as a missing piece of the puzzle to scale up functional programming, as time and time again we have encountered similar problems in projects. Problems concerning competing timelines in different systems, a desire for immutability and purity, and a messy reality of delays and information that matures over time. While popular as a research topic in the 90s, bitemporality is only now starting to become practical due to cheap cloud storage. There's a close kinship to modern event sourcing architectures and functional programming, and we can now start rephrasing the earlier ideas in this newer light. At JUXT we decided to build Crux, a schemaless bitemporal database which uses Kafka for its commit log. It provides Datalog queries served by disposable index nodes powered by RocksDB or LMDB, each being a function of the log. This talk will look at the approach we took to build Crux, what design decisions we made and how we reached them. Crux aims to be an "unbundled" database of several pluggable components and layers adhering to "The UNIX philosophy", a bit akin to how the Clojure community favours composing smaller libraries. At the same time, there are significant technical challenges involved in making bitemporality practical, often in tension with the idea of neatly composing components.