Chris Nuernberger explains the design of tech.ml.dataset and dtype-next for high-performance columnar data processing in Clojure.
The London Clojurians are happy to present:
Title: **High Performance Data With Clojure**
Speaker: **Chris Nuernberger**
Chris Nuernberger (github.com/cnuernber/) will be presenting
a talk about his work with `dtype-next` (github.com/cnuernber/dtype-next)
and `tech.ml.dataset` (github.com/techascent/tech.ml.dataset)
the new high performance data processing system for Clojure. His talk
will cover the axiomatic basis of array programming, some interesting
high performance primitives, and how representing your problem with a
column-major map-of-arrays design gives you some interesting
properties as compared to your standard Clojure row-major
sequence-of-maps design.
Chris got his start programming high performance 3D graphics in the
early 2000's. He has worked extensively in GIS, 3D graphics, and
machine learning building such things as graphical playback-style
debuggers for physics simulations systems, Cinema 4d-style graphics
editors, GIS tools including research into (at the time) new lossless
compression technologies and national scale facial recognition
systems. He started playing with Clojure in 2008 and has been working
in Clojure full time since 2015.
---
Please, consider supporting the London Clojurians with a small donation:
opencollective.com/london-clojurians/
Your contributions will enable the sustainability of the London
Clojurians community and support our varied set of online and
in-person events:
- ClojureBridge London: supports under-represented groups discover Clojure
- re:Clojure: our **free** to attend annual community conference
- monthly meetup events with speakers from all over the world
- subscription and admin cost such as domain name & Zoom plan for
larger online meetups
Why it's here
2 people recommend this
What people say. Recommended as a deep dive into high-performance JVM data processing and the tech.ml.dataset library.
c
u/chrisnuernbergerin r/Clojure • Recommends
TMD 7.000 - The Long Beta Is Over
We have released version 7.000 of tech.ml.dataset - our in-memory column-major data processing engine!
This release contains a lot of bugfixes, performance improvements, API improvements, and the results of quite a lot of painstaking research into how the JVM works and fast reduction pathways for complex systems.
Specifically the engine was very carefully built from a perspective of making reductions as fast as possible leading to improvements in several areas. Ideally this version is a drop-in replacement with better performance than version 6 (or pandas or dplyr).
An example of one specific improvement out of many that we now to kernel combining in th…
