When Can We Trust What a Model Has Learned? Identifiability, Causality, and Reliable AI

Pradeep Ravikumar discusses how identifiability and causality can help determine what aspects of data a machine learning model has actually learned.

Pradeep Ravikumar (Carnegie Mellon University) simons.berkeley.edu/talks/pradeep-ravikumar-carnegie-mellon-… Trustworthy AI: From Hallucinations to Reliable Autonomy Modern AI systems can achieve remarkable predictive and generative performance while remaining unreliable under distribution shift, novel compositions, or intervention. A fundamental reason is non-identifiability: many different internal representations and mechanisms can explain the same observed data and training objective, while implying very different behavior outside the training distribution. Thus, before asking whether a learned representation is interpretable, truthful, or causally meaningful, we should ask a more basic question: what aspects of what the model has learned are actually determined by the available evidence? I will discuss a line of work that uses identifiability and causality as organizing principles for reliable representation learning. The central idea is that additional structure, such as sparsity, variation across environments, weak supervision, and assumptions about how causal mechanisms change, can turn otherwise underdetermined latent representations into recoverable, semantically meaningful abstractions. I will describe examples ranging from discovering latent causal variables and mechanisms to identifying interpretable concepts inside foundation models, where controlled contrasts and environmental variation can distinguish causally active concepts from merely correlated directions. This perspective suggests a route from prediction to reliable autonomy: rather than relying solely on increasingly capable models together with post-hoc checks, we can ask what latent variables, concepts, and mechanisms are identifiable; what interventions they support; and which aspects of their behavior can therefore be expected to remain stable as the world changes.