Learning to Reason with LLMs

Noam Brown describes OpenAI's o1 model and how training LLMs with reinforcement learning to generate a chain of thought improves reasoning.

Noam Brown (OpenAI) simons.berkeley.edu/talks/noam-brown-openai-2024-09-26 Transformers as a Computational Model Large language models (LLMs) have demonstrated remarkable capabilities in generating coherent text and completing various natural language tasks. Nevertheless, their ability to perform complex, general reasoning has remained limited. In this talk, I will describe OpenAI's new o1 model, an LLM trained via reinforcement learning to generate a hidden chain of thought before its response. We have found that the performance of o1 consistently improves with more reinforcement learning compute and with more inference compute. o1 surpasses previous state-of-the-art models in a variety of benchmarks that require reasoning, including mathematics competitions, programming contests, and advanced science question sets. I will discuss the implications of scaling this paradigm even further.