We host Tal Linzen, Associate Professor at NYU and Research Scientist at Google, for a conversation on the intersection of cognitive science and large language models.
We discussed why children can learn language from around 100 million words while LLMs need trillions, and the surprising finding that as models get better at predicting the next word, they become worse models of how humans actually process language. Tal walked us through how his lab uses eye-tracking and reading-time data to compare model behavior to human behavior, and what that reveals about prediction, working memory, and the limits of current architectures.
We also got into nature versus nurture and how inductive biases can be instilled by pre-training on synthetic languages, world models and whether transformers actually use the geometric structure they encode, the BabyLM challenge and data-efficient language learning, and what mechanistic interpretability can offer cognitive science beyond just fixing model bugs. The conversation closed on academia versus industry, the role of PhDs in the current AI moment, and how AI coding tools are changing the way Tal teaches and evaluates students at NYU.
Timeline
00:13 — Intro and what cognitive science means
02:16 — Using computational simulations to understand how humans learn language
05:26 — How children learn language vs. how LLMs are pre-trained
07:53 — Why mainstream LLMs are not good models of humans
10:07 — Comparing humans and models with eye-tracking and reading behavior
13:52 — Sensory modalities, smell, and how much you can learn from language alone
16:03 — Animal cognition and decoding animal communication
17:00 — Nature vs. nurture, inductive biases, and what transformers can and can't learn
21:21 — Instilling inductive biases through synthetic languages
27:34 — The bouba/kiki effect and cross-linguistic sound symbolism
28:33 — Latent causal structure in language and whether models discover it
31:13 — Does knowing linguistics help build better models?
35:07 — World models: what they mean, and why transformers encode geometry but don't use it
39:13 — Tokenization, and why Tal doesn't like it
41:35 — Scaling laws and the inverse-U curve of model quality vs. human fit
44:34 — Where the human–model mismatch comes from: architecture, memory, and data
47:08 — Diffusion language models and sentence planning
48:21 — Data quality, synthetic data, and curriculum effects
50:54 — Comparing models at different training stages to human development; BabyLM
54:40 — What level of the model should we actually probe? Representations vs. behavior
1:01:04 — Mechanistic interpretability, Deep Dream, and human dreaming
1:02:11 — Cognitive neuroscience, intracranial recordings, and working memory
1:10:31 — Should you still do a PhD in 2026?
1:12:31 — Will software engineers lose their jobs to AI?
1:17:43 — Teaching in the age of coding agents: what changes in the classroom
1:20:54 — What's next: human-like LLMs as user simulators, and recruiting
Music:
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
"Palms Down" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
Changes: trimmed
About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
