John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.
We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.
Topics:
Implicit vs. explicit world models, and the case against JEPA-style objectives
Compact belief states: why compression beats a growing KV cache
Does research still matter in the age of scale? The Kimi K3 argument
Agent-driven research: 2,000 pre-training experiments in 90 days
The invention of CAPTCHA
Optimizers from SGD and Vowpal Wabbit to Muon
Chapters
00:00 Why world models: the sample-complexity gap
09:48 The case against JEPA; a transformer-style implicit world model
15:52 Compact belief states: epicycles vs. heliocentrism
23:41 Does research still matter? The Kimi K3 argument
27:35 Open vs. closed models
35:57 Recursive self-improvement and agent-driven research
42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility
54:54 The invention of CAPTCHA
1:00:53 Optimizers: from Vowpal Wabbit to Muon
Music
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
