Escape from the Heap: Low-Level Programming in Common Lisp

Ahmon Dancy explains how Franz Inc. uses off-heap data structures in Common Lisp to manage memory outside the garbage collector.

by Ahmon Dancy, software engineer, Franz Inc. Bay Area Lisp and Scheme Meetup balisp.org/ Sat 7 Nov 2015 Franz HQ Oakland, CA Slides goo.gl/WIYWzl (Google Drive) Abstract Franz Inc has been selling Franz Lisp and Allegro Common Lisp for the last thirty years. In 2005, Franz entered the market for Semantic Graph Databases by introducing AllegroGraph. Our graph database is used by intelligence agencies, the army, and recently by pharmaceutical companies, financial institutions and hospitals. The visualization tool Gruff and the database AllegroGraph are written in Allegro Common Lisp. In the process of building the database we had to invent several new, mostly lower level technologies, one of them being out of the heap data structures. Sometimes data structures need to be persistent or shareable amongst separate processes. In these cases, alternatives to using the heap must be considered. In this talk we will explore the motivations for out-of-heap data structures. We will discuss some of the out-of-heap data structures that we've created in the course of developing our database product, such as lists, hash tables, and arrays. We will describe the tools and mechanisms that we used to implement them, including memory-mapped files, foreign structs, aligned pointers and direct memory accesses. Finally we will discuss the downsides of out-of-heap data structures and the constant struggle between abstractions and performance.