How to Build a Modern Diffusion Language Model

Volodymyr Kuleshov introduces the building blocks and research advances behind modern diffusion-based large language models.

Volodymyr Kuleshov (Cornell University) simons.berkeley.edu/talks/volodymyr-kuleshov-cornell-univers… Diffusion Generative Modeling: Progress and Next Steps This talk introduces diffusion for language and the research advances that underlie modern diffusion LLMs. We describe the building blocks of today's leading open-source models, starting from simple masked diffusion language modeling (MDLM), and covering techniques for iterative refinement, controllable generation, and efficient inference with variable-length sequences. These techniques unlock capabilities autoregressive models lack: fast generation at ＞1000 tokens/second, new forms of inference-time scaling through repeated refinement, and fine-grained controllable generation that has proven especially valuable in the sciences. We ground these in the context of leading models for biological sequences (proteins and DNA) and natural language, including frontier diffusion LLMs like Nemotron Diffusion, Gemma Diffusion, and Mercury 2.