Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks.
We talk about why deep learning theory is finally catching up to practice, including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics.
The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire.
We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm.
key topics
Connections between physics, neuroscience, and AI
Emergent properties in complex systems
Scaling laws in language models
Data efficiency and pruning in AI
Neuroscience insights into consciousness and self
The future of AI and brain modeling
Chapters
00:00 Introduction to Surya Ganguli
00:57 Surya's Background: From String Theory to Neuroscience
02:22 Emergent Properties in Physics, Neuroscience, and AI
03:16 Energy Landscapes and Loss Landscapes in High Dimensions
04:07 Why Local Minima Don't Exist in High-Dimensional AI
05:22 Gradient-Based vs. Gradient-Free Learning Methods
08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges
13:48 Scaling Laws and Data Efficiency in Language Models
18:10 Properties of Data that Affect Scaling Laws
22:04 Constructing Non-Redundant Data Sets for Better Learning
24:32 Theory vs. Empirical Results in AI Research
32:19 Fundamental Components of Deep Learning: Are They Changing?
34:31 Future Paradigms in AI Beyond Current Models
37:22 Teaching AI and Humans: Paradigm Shifts in Learning
41:37 Consciousness, Self, and the Brain: Surya's Perspectives
49:49 Neuroscience and AI: Understanding the Brain and Consciousness
01:02:03 Understanding the Brain: Challenges and Opportunities
01:09:21 Brain-Computer Interfaces and AI in Neuroscience
Music
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
