RustConf 2019 - Rust for Weld, a High Performance Parallell JIT Compiler by Shoumik Palkar

Shoumik Palkar discusses using Rust to build Weld, a high-performance parallel JIT compiler that optimizes data-intensive libraries.

RustConf 2019 - Rust for Weld, a High Performance Parallell JIT Compiler by Shoumik Palkar Weld is an open source Rust project that accelerates data-intensive libraries and frameworks by as much as 100x. It does so by JIT-compiling a custom parallel intermediate representation and optimizing across functions within a single library as well as across different libraries, so developers can write modular code and still get close to bare metal performance without incurring expensive data movement costs. We first describe our use of Rust to build an extensible, performance sensitive compiler for Weld. In particular, we discuss challenges in writing a native code generator with LLVM in Rust, challenges in reducing JIT compilation times within our compiler, and architecting the compiler for both extensibility using Rust's powerful trait system, but also portability of our library across other languages, such as Python and C. Finally, we discuss our experiences with building a parallel runtime in Rust, and the unique challenges associated with calling unsafe Rust from JIT'd code.