u/miki151zenon-lang.orgin r/ProgrammingLanguages • Recommends
This talk by Nicholas Matsakis is the best explanation of the "compiler as a service" architecture that I've found so far
0:00 / 0:00
#incrementalcompilation#salsa#rustc
Responsive compilers - Nicholas Matsakis - PLISS 2019
PLISS
908 subscribers • 3 videos on Hallway
- Recommended by 4 people: 1 on Lobsters and 3 in engineering communities.
- A classic. People kept recommending it from 2019 to 2024.
- Presented by Nicholas Matsakis
Advanced. Deep dive for people who know the field.
16K views7 years ago4 recommendations
Programming languagesSystems programmingCompilers and interpretersRustVim, Emacs and editors
Nicholas Matsakis explains how the Rust team restructured the compiler to support incremental compilation and IDE integration using a query-based architecture.
Many compiler textbooks and courses treat compilation as a "batch process", where the compiler takes in a bunch of input files, executes a suite of comiler passes, and ultimately produces object code as output. Increasingly, though, users expect integration with IDEs like VSCode, which requires a different structure. Moreover, many languages have recursive constructs where the correct processing order is difficult to determine statically. Nicholas will discuss some of the work the Rust team has been doing on restructuring the compiler to support incremental compilation and IDE integration. Slides: nikomatsakis.github.io/pliss-2019/responsive-compilers.html Part of PLISS'19: pliss2019.github.io/Why it's here
4 people recommend thisWhat people say. Recommended as the clearest explanation of the compiler-as-a-service architecture and incremental compilation.
u/bjzabaPikelet, Unnamed Binary DDLin r/ProgrammingLanguages • Recommends
Responsive compilers - Nicholas Matsakis - PLISS 2019
u/Thnikkaman14in r/ProgrammingLanguages • Recommends
"Responsive Compilers" was a great talk. Have there been any updates/innovations in this space since 2019?
Someone on reddit linked this 2019 talk about building a responsive, incremental IDE/LSP-compatible compiler. Highly recommend watching it.
5 years later, do people use this paradigm in practice? Better yet, are there popular frameworks/libraries people use for incremental compilation, or do most compilers just roll their own framework? I see that the speaker's salsa framework has some stars on github but I'm not very familiar with rust
The talk mentions a few not-quite-solved problems in the space, I wonder if 5 years later some of the best practices are better understood:
* ([1:…
inactive-useron Lobsters • Recommends
Responsive compilers - Nicholas Matsakis - PLISS 2019
u/pein_samain r/ProgrammingLanguages • Mentions
On “As a Vim enthusiast, had an idea for a language...”
In this presentation, a Rust dev talks about how thinking about IDE support transformed the very basic concepts of how a compiler should work. Responsive compilers - Nicholas Matsakis - PLISS 2019 - YouTube
u/oilshellin r/ProgrammingLanguages • Mentions
On “Language Server built into the compiler ?”
It's a good idea, but it's hard. You basically have to "invert" the whole compiler -- from a batch process to an incremental one. And it's even a bit more than that (errors are different, etc.).
90 minute video from a core Rust team member about this:
link to this video
If you're a beginner I would start out with the batch process, and then figure out how to do the language server later. It will clarify the thinking / algorithms at least.
u/cxzukin r/ProgrammingLanguages • Mentions
On “Why incremental parsing matters?”
Hi Null,
\> still need significant infrastructure to keep track of what you are editing right?
There is overlap to the task of building a text editor/IDE, and incremental parsing. E.g. Keeping a list of changes (An edit history) is useful outside of incremental parsing, such as undo and redo.
\> Is having this infrastructure + implementing the necessary modifications to the parser worth it? (from a latency and from a coding perspective)
Editing Composed Languages, 2019 - Lukas Diekmann is a wonderful read, providing an overview, and I highly recommend to anyone looking at getting into incremental parsing.
Regarding latency, Pages 48-51 compare against batch parsing, and also estimate t…
u/mttdin r/ProgrammingLanguages • Mentions
On “References about parsers used in IDE”
"Responsive compilers" by Nicholas Matsakis may be worth a watch: link to this video (PLISS 2019; slides: https://nikomatsakis.github.io/pliss-2019/responsive-compilers.html; salsa project: https://github.com/salsa-rs/salsa).
References:
- Practical Algorithms for Incremental Software Development Environments - 1998 PhD Dissertation; Tim A. Wagner - https://www2.eecs.berkeley.edu/Pubs/TechRpts/1997/5885.html - make sure to compare with improvements in "Editing composed languages" (right below)
- Editing composed languages - 2019 PhD Dissertation; Lukas Diekmann - https://diekmann.co.uk/diekmann_phd.pdf
- Parser Generation for Interactive Environments - 1992 PhD dissertation; Jan Rekers - http://cites…
u/scottmcmrust🦀in r/ProgrammingLanguages • Mentions
On “Unison: A Friendly Programming Language from the Future • Runar Bjarnason”
…)".
I'm also confused by the whole "my code references other things by hash". Does that mean that I need to update every caller if I fix a bug in my function implementation?
Caching stuff in hash tables is definitely great. But does it need a new language? For example, you can cache stuff by hash in your LSP implementation, like described in <link to this video For example, the Unison talk describes "it only needs to be typechecked once" around 16:23, but if you hash stuff, you don't need to typecheck it again in *any* language.
Or it talks about "names are taken", but you also don't need Unison to solve that. Rust today allows different versions of dependencies, because hashing is just one of…
u/bluehavanain r/ProgrammingLanguages • Mentions
On “Advice/best practice/arhitecture pattern for building language with LSP in mind?”
There's also a great presentation on the reasoning behind salsa: [video
u/scottmcmrust🦀in r/ProgrammingLanguages • Mentions
On “How we make the Futhark compiler crash”
I think they're particularly good because all the inputs are fixed, so there's never any user data or state that you need to save on a crash, like there might be for an image editor or similar.
Phases have some potential problems, though. Niko has an interesting talk about this from PLISS 2019 <link to this video
u/scottmcmrust🦀in r/ProgrammingLanguages • Mentions
On “Resources on writing a parallel compiler”
You might consider a query-oriented approach like with which Rust is currently experimenting. Then you could probably ask multiple (memoized) questions at the same time.
Niko has a talk about it at link to this video
u/bjzabaPikelet, Fathomin r/ProgrammingLanguages • Mentions
On “Should my compiler stop after the first file with errors?”
Rust uses it - it's mentioned in Niko Matsakis' talk, Responsive Compilers.
u/pein_samain r/ProgrammingLanguages • Mentions
On “A different / new way to write compilers?”
Yes. Check out this presentation on how Rust does it.
u/scottmcmrust🦀in r/ProgrammingLanguages • Mentions
On “How do you separate the different stages of your compilation pipeline?”
One thing you should do i make sure you have an `Error` node in the various stages. So you can "parse" an expression as `Error` after you already emitted a diagnostic about it, but keep going with the rest of the program. Then typechecking emits the type of `expr::Error` as `ty::Error`, *without emitting another diagnostic*, so it doesn't cause downstream problems in further checks.
Alternatively, consider whether stages are even a good model. Relevant conference talk: <link to this video
u/scottmcmrust🦀in r/ProgrammingLanguages • Mentions
On “Does anyone have issues with tightly coupled implementations?”
I don't know a single successful language that hasn't rewritten its compiler at least once, so congrats on reaching that stage 🙂
If you're curious about a *very* different way to do things from the traditional way, you might be interested in Niko's PLISS talk on responsive compilers: <link to this video
Forcing everything into a memoizable structure helps force you to not add hacks the way it can be easy to do in classic architectures.