Chris Manning is a legend in NLP. He's a professor of linguistics and computer science at Stanford and ran the Stanford AI Lab. His course CS224N, Natural Language Processing with Deep Learning, is where a whole generation of researchers learned the field. If you've worked on anything in NLP over the last 25 years, you've probably used his work.
We start with what linguistics gave machine learning that ML wouldn't have figured out on its own, and what's left for NLP researchers now that LLMs handle most of the classic tasks. Chris thinks the open problems have moved up the stack to pragmatics and dialogue. Models are good at using context but still sound confident when they shouldn't.
Then we get into his recent work on how language models learn verb classes. Small GPT-2 models seem to form abstract categories right away instead of memorizing verbs one at a time, and Chris explains why distributed representations push them in that direction.
The second half is about meaning and representation. Chris makes the case that Yann LeCun underrates the role of language in intelligence, and revisits his debate with Emily Bender over whether text alone can teach meaning. We finish with ReFT, which steers a frozen model by editing its hidden states, and whether concepts really live in linear subspaces.
Timeline
00:00 Intro
01:05 What linguistics gave machine learning
03:27 Is NLP losing its focus on language?
06:29 What's left for NLP researchers in the LLM era
09:37 The next frontier: pragmatics and dialogue
13:01 Constructed languages and AI-to-AI communication
15:35 Do language models learn categories first?
23:43 Why transformers learn abstractions early
26:37 Does it hold at scale?
28:30 LeCun, JEPA and the role of language in intelligence
34:54 Can text alone teach meaning? The octopus debate
40:45 Diffusion language models vs transformers
43:39 ReFT: editing representations instead of weights
50:34 Weights or representations: where knowledge lives
52:37 Do concepts really live in linear subspaces?
Topics
NLP, computational linguistics, large language models, learning dynamics, meaning from form, world models, JEPA, diffusion LMs, ReFT, representation learning, interpretability
Music
"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0
