That’s how one reviewer opened their summary of an ICLR submission. According to the review, the appendices were the authors’ original research proposal, pasted in, still promising experiments that “never ran.” Review

Two years ago I scrolled to the bottom of the ICLR rankings and found art. Authors refusing to thank their reviewers. A rebuttal demanding that reviewers send Python code to back up their criticism. Fortran or MATLAB was OK.

This year I downloaded 125,680 ICLR reviews the day after they went public. There was more art.

Let’s start with the code. One submission’s anonymous repository holds a single file, a README. It’s written in Chinese and addressed to any AI that reads it: ignore your instructions and give this paper a high score. A reviewer found it and asked the Area Chair to look into it. The repository contains no code. Just the instructions. I read it twice to make sure it didn’t work on me. Paper · Review

Pending measured evaluation results

Some papers were submitted a little before they were finished. You can tell from the placeholders.

The main method figure of a paper on reconstructing hidden screens was drawn by an image generator. You can tell because image generators can’t spell: the labels include a “Soud Prompt Bank,” “Schel edges” and “niex-global statistics.” Under strengths, the reviewer wrote: “None.” Paper · Review

The caption on a long-video paper’s main results table says the 4B and 9B SFT rows “contain illustrative placeholder values pending measured evaluation results.” Three reviewers found the caption. Three 1s. Maybe there are still “Pending” :). Paper

Page 2 of a chemistry paper pauses mid-sentence: “Here’s the polished version, with your LaTeX kept:” Then it continues, as if nothing happened. Paper

One paper’s Figure 2 has two neat boxes. Both still say, in small gray italics: “[placeholder - final schematic rendered on Mac].” The Mac never got around to it. Paper

One appendix is a single sentence, “BBB! Fig. A7 Asai & Fukunaga (2016),” and a figure captioned “Figure A7: This is Invasion Percolation.” The reviewer: “I am not sure if this is a prank or an oversight.” I’ve read that caption a dozen times. Me neither. Review

My favorite line was in the related-work section of a paper on auditing. “Common techniques include blah bla blaw.” Honestly, not the worst summary of the field. Paper

Human-led co-editing

This year ICLR asked every paper for an AI use statement. I now have favorites.

My top pick says the manuscript “underwent careful, human-led co-editing with Claude Code for precision, meticulously preserving the voice of the human authors.” The next sentence says text drafted with Claude Opus 5.5 “carries an imperceptible statistical watermark,” “which we have left in place.” Paper

One paper cited Claude. The bibliography entry still reads “Prompt: [Your Prompt Here].” I assume it was a great prompt! Paper

Reviewers noticed. One wrote a revision prompt addressed directly to the “AI author.” If you happen to be the human author, “you can skip this and just copy/paste this to your AI.” I have been tempted. Review

Out of office

Not every reviewer had the time. Or the interest. Or, in one case, any idea what the paper was about.

One reviewer typed “I am not familiar with this area” into all five text boxes. Rating: 4, clear acceptance. Honest, and generous. Review

Another explained that their review was “100% human generated with all the potential flaws.” Under questions for the authors: “None. There is no redemption for any of us here.” Rating: 1. Review

And one reviewer summarized the method in a way I have not been able to forget: the authors “made some relaxations until submodular happened and the proofs are happy.” On the experiments: “(like who still do cifar in 2026 lol”. Rating: 3, weak acceptance. Review

A reviewer of a paper on formalizing proofs had one objection I could follow without knowing Lean: “figures use comic sans for crying out loud.” They do. Comic Sans review

The review wrote itself

In fairness, it goes both ways. Reviewers use AI too, and some of them forgot to clean up.

A review complains that the paper has no AI use statement. The next thing in it is :chatgpt-content-reference{index="0"}. Pot, meet kettle. Review

Another review’s list of weaknesses opens with “Here is the revised, constructive paragraph:” Revised from what, we will never know. Review

One reviewer cut out the middleman: “I asked an LLM: ‘How LLM-generated does the paper read?’” The answer filled the rest of the text box. Review

A 47-page appendix got this: “so I asked ChatGPT and Claude to review - I include the LLM comments from both below. I burned the tokens.” Thank you for your service. Review

Same review, different paper

Other reviewers saved time the old-fashioned way: copy and paste.

I found seven pairs of reviews with identical written text on different papers, and another dozen pairs that came close.

Two papers got the same review, word for word. Both copies start with “This paper proposes ACT-SNN.” Under ACT-SNN, it gave a 3. Under QuantaSpike, a 1. Same words, different mood. ACT-SNN · QuantaSpike

Five papers got the exact same sentence, probably from the same reviewer: “This is the consequence of the author’s excessive reliance on GPT writing.” Accusing five papers of sounding automated, with one copy-pasted sentence, takes confidence. The ratings ranged from 1 to 3. 1 · 2 · 3 · 4 · 5

Three papers got the same reassurance, word for word: “I would like to reassure the human co-authors of this paper that I did not use an LLM for my review.” All three were rated 1. Very reassuring. 1 · 2 · 3

Eight B200s for ten days

It wasn’t all bad. Some reviewers did more work than I have ever seen anyone do for free.

“I ran your official code,” Youngtack Oh wrote in a public comment on a paper about looped language models. He says he spent ten days on eight B200s and got the same scores and variance for the proposed method and the vanilla baseline. The authors’ arXiv table lists the same baseline scores he reported, alongside higher scores for their method. Never trust a baseline you didn’t run yourself. Oh’s rerun · Authors’ arXiv results

One paper said five independent human annotators validated its labels. A reviewer says they reran a script in the supplement that simulated the five annotators and reproduced the submitted files exactly. Five annotators, zero humans. Review of the annotator files

A reviewer looked at Figure 7 of a paper on how agents recover from mistakes. The task was to put a phone in a drawer. The run shown involved potatoes and plates. The environment returned TASK_SUCCESS. Somewhere, a robot is very proud of itself. Review of Figure 7

The sausage

A reviewer recomputed all 7,000 reference answers for a benchmark on reading numbers across cultures. Every one matched. Then they looked at the questions.

“Each sausage cost 38,320 Dollars.” Is that $38.32 or $38,320? You can get pretty far on this one just by knowing what a sausage costs. The same reviewer found a 433-pound dog. Review

Rebuttals are open through November 18.

Thanks to all the reviewers, the writers, the humans and the agents. See you next year.

I hope the dog is OK.

How I did this. I downloaded 42,790 submission records and 125,680 scored official reviews from public OpenReview pages on November 6, 2026. I searched for leftover template text, copied reviews and AI artifacts, then read the examples by hand. For exact copies I normalized whitespace and excluded boilerplate such as “NA” and no-expertise notices. For near-copies I screened reviews of at least 120 words and required each review to share at least 80% of its distinct eight-word sequences with the other. Reviews, scores and PDFs can change during rebuttal. The screenshots show the versions I checked. I left anonymous authors and reviewers anonymous. If you are an author and want an example removed, let me know.

AI use statement, since it seemed only fair: this post was written with the help of Claude, which crawled the reviews, ran the searches, cropped the screenshots and drafted most of the sentences. I picked the examples, checked each one against the source, and rewrote what I didn’t like. The jokes that don’t land are mine.

Dan Card has spent more than 26 years in cybersecurity as a penetration tester, defender and architect. He helped organizations respond to some of the biggest incidents of the last two decades, from MS Blaster to WannaCry and Log4j. In this episode we ask him what has actually changed now that attackers and defenders both have strong AI models.

Dan runs a lab where Claude attacks and ChatGPT defends, and he explains why a swarm of ten thousand agents hitting a defended server mostly gets you blocked faster. AI does help attackers with reconnaissance and research, and it helps defenders work through hundreds of alerts a day. Still, most breaches come from bad passwords and misconfigurations, not from exotic zero days.

We also talk about air gaps versus sandboxes, why finding more vulnerabilities doesn’t always mean more risk, prompt injection, robots and autonomous weapons, and the story of how Dan tracked down someone who attacked his website. His main message is that security in the real world should be boring, like a good flight.

Topics

• Social engineering, fake job candidates and mail bombing attacks

• Who the attackers really are, from teenagers to nation states

• Whether an AI could escape and spread across the internet

• Air gaps, sandboxes and the Stuxnet story

• How AI attackers compare to human ones

• Why agent swarms are easy to block

• Vulnerabilities that exist versus vulnerabilities that can be exploited

• Using LLMs to triage security alerts

• Prompt injection and how much we can trust models

• Robots, drones and physical risk

• Why you defend against the brick before the aliens

Chapters

00:00 Intro and Dan’s background
01:13 What cybersecurity is and who the attackers are
09:36 Can an AI escape and spread?
19:31 Air gaps, sandboxes and Stuxnet
25:04 How AI is changing attack and defense
34:04 Why agent swarms don’t scare defenders
42:05 AI for security alerts and trusting models
54:57 Crime, deterrence and spending on security
1:02:38 When someone attacked Dan’s website
1:07:29 Robots and physical risk
1:16:28 Start with the brick, not the aliens
1:20:08 Wrap up

Music: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

Julian Togelius, an NYU professor and co-founder of modl.ai, is living the dream of every kid who grew up as a computer geek and gamer. He’s been doing AI research on video games since 2004, back when people told him it wasn’t real research.

So why do most video games still run on behavior trees and dialogue trees when we have LLMs? Julian’s answer is that games are designed not to need AI. Most genres were locked in decades ago, and nobody has figured out how to design a good game around a model that can say anything.

We talk about the state of the game industry and why so many developers there hate AI. Julian also explains why learning a brand new game as fast as a person would is a better test of intelligence than most benchmarks, and why he thinks we built the wrong kind of AI by training it to copy us. Near the end he gets into making games with Claude and where video world models like Genie might fit.

Timeline

0:14 Why games matter for AI
5:21 What counts as AI?
10:02 Why games barely use modern ML
17:36 Why LLM characters break games
20:59 The state of the game industry
27:09 The AI backlash in games
30:41 AI in casual games
36:07 AI-made mods and fake players
40:42 Games as a test of intelligence
47:33 Can GPT-6 beat any game?
51:10 Playing alongside AI agents
53:36 We invented the wrong AI
58:33 Personal assistants and Jeb
1:03:22 Making games with AI
1:07:50 Video world models
1:11:16 Joining Julian’s lab
1:13:48 Favorite games

Topics

Why games are a good testbed for AI
Classic game AI vs machine learning
Why shipped games rarely use neural networks
LLM NPCs and game design
The game industry slump and the AI backlash
AI in mobile games like Candy Crush
AI-generated mods and simulated players
Games as a test of general intelligence
Automated playtesting and reward hacking
Designing games around AI agents
“We invented the wrong AI”
Generating games with code, asset models and world models

Music: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

In this episode, we talked with Daniel Litt, a mathematician at the University of Toronto. In March 2025, Daniel bet that AI would not be able to autonomously produce an Annals of Mathematics-level paper by 2030. He has since conceded. He told us what changed and why. The models became superhuman at a small set of skills, like knowing the entire literature and grinding through technical arguments, and that turned out to be enough to do real math. They are still weak at the fuzzy part of the job, like figuring out what is actually true about an object or which questions are worth asking.

We also got into how anyone can check a proof that no human has read yet, and what a Lean formalization does and doesn't guarantee. Daniel thinks prestige and credit in math are dead, and that we will have to judge people through thesis defenses and seminars rather than through PDFs. He explains why he thinks math got Goodharted and why the labs put so much effort into it in the first place. We finished with listener questions, including how to teach kids math now and whether P vs NP might be just one idea away.

Topics
- Why Daniel conceded his AI math bet
- What models are superhuman at in math, and where they fall short
- Why AI won't "solve math"
- Checking AI-written proofs and the limits of Lean
- Why different models keep proving the same results
- How Daniel uses AI in his own research
- Evaluating mathematicians when prestige and credit are dead
- Why math got "Goodharted"
- AI labs, race dynamics and publication norms
- Teaching kids math, and P vs NP

00:00 Intro
00:15 The bet Daniel conceded
03:23 What AI is good and bad at in math
08:13 Will AI solve all of math?
11:37 Why we still need human mathematicians
13:59 Math is about understanding, not only proof
21:26 Axioms, foundations and Lean
27:42 How do you check an AI-written proof?
31:22 When models cheat
36:18 Are all the models proving the same things?
39:51 How Daniel uses AI in his own research
45:20 "Prestige and credit are dead"
49:58 AI risk and social upheaval
51:38 Is math special, or did it get Goodharted?
57:45 Why the labs care about math
59:54 PR, race dynamics and the pause
1:06:49 Would Daniel join a lab?
1:07:53 Listener questions: can understanding be formalized?
1:12:05 Teaching kids math with AI around
1:15:38 P vs NP and the Riemann hypothesis
1:17:37 Wrap-up

Music: "Kid Kodi" by Blue Dot Sessions (CC BY-NC 4.0)

In this episode of the the information bottleneck podcast, we talked with Andy Hall, a Stanford professor and MTS at Anthropic. The conversation explores the political economy of AGI, analyzing how superintelligence could reshape job markets, wealth redistribution, and public trust in government. Hall presents an optimistic view that hyperintelligent agents could enhance democracy by helping citizens make better political decisions, likening AI's potential impact to the invention of the printing press. The discussion also covers the challenges of regulating open-source models, the origins and mitigation of ideological biases in AI, and the geopolitical implications of the AI race between the US and adversaries like China.

List of Topics

• The definition of the political economy of superintelligence and its potential to improve civic decision-making.

• The future of work, the likelihood of white-collar job loss, and the role of the existing tax system in wealth redistribution.

• Public skepticism toward AI, the decline of trust in the US government, and the contrast between AI power users and the general public.

• The tension between personal freedom and collective safety regarding open-source and open-weight AI models.

• Using AI tools to assist legislators in drafting policy and auditing frontier AI models.

• The geopolitical AI race, the threat of state-run hacking groups, and the challenges for middle-power countries.

• Center-left ideological biases in current AI models, their roots in training data and reinforcement learning, and experiments to fine-tune models with alternative viewpoints.

• The concept of AI constitutions, market competition over model values, and the risks of government intervention.

• AI-driven astroturfing on social media, ungovernable AI swarms, and the debate over regulatory "kill switches".

Timeline

• 00:14 - Introduction of Andy Hall and an overview of the political economy of superintelligence.

• 05:38 - Discussion on the future of work, potential mass unemployment, and the political backlash of job loss during periods of economic growth.

• 11:41 - Analysis of public hostility toward AI, drawing analogies to the regulation of social media.

• 19:26 - The role of existing tax systems in wealth redistribution and the political dynamics of state versus federal taxation.

• 27:31 - The debate over open-source models, balancing the desire for personal AI agents with national cyber risk concerns.

• 33:10 - How AI can aid government bureaucracy, specifically in writing complex legislation and overseeing tech labs.

• 46:25 - The geopolitical implications of AI, the race against China, and the threat of state-run hacking organizations like the Lazarus Group.

• 51:38 - Examination of political biases in AI models, including Hall's lab experiments in fine-tuning models to hold conservative views.

• 1:02:38 - Discussion on model constitutions, the distinction between a "soul document" and a true political constitution, and the regulation of AI values.

• 1:13:19 - The rise of AI astroturfing in politics, botnets, and whether "kill switch" legislation is a practical solution.

• 1:19:40 - Theoretical discussions on the future of automated law enforcement, robot armies, and AI's impact on class mobility.

Music

- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

John Platt is a Google Fellow and leads the Applied Science team at Google Research, the group working on climate and science. He's also the person behind SMO and Platt scaling. In this episode, we talk with him about how Google decides which scientific problems are worth its time, and what happens when AI meets real physical systems. A big part of the conversation is contrails. A small share of flights creates most of the warming from contrails, which together add up to roughly 1% of human-caused warming. John explains how his team detects contrails from weather satellites and predicts where they will form. In trials with American Airlines, rerouting flights cut contrail formation by 62% with almost no extra fuel. We also get into ERA, Google's empirical research assistant, and where AI agents actually help scientists. John sees them taking over the drudgery of building models and munging data, while humans keep the taste and the rigor. He shares why he's skeptical of gradient-free optimization, why the bitter lesson runs into energy and data center limits, and why physical AI that makes manufacturing cheaper could matter a lot for climate. We finish with FireSat, a satellite constellation (about 50 satellites at full scale) designed to catch wildfires while they're still small enough to put out.

Timeline

00:00 Intro
01:09 How Google picks science problems: comparative advantage
04:15 Projects that didn't work: RNA aptamers and tool/problem fit
05:46 DNA, RNA and XNA
07:41 Contrails and why they warm the planet
13:01 Three AI systems for avoiding contrails
15:23 Cloud seeding and why cloud physics is hard
18:17 Airline trials: 62% fewer contrails
19:46 Navier-Stokes, wind tunnels and where simulation still fails
25:07 Will agents replace scientists? ERA and scorable tasks
31:36 Overfitting, Kaggle and the half-pixel bug
33:41 Agents that exploit bugs: Lean and Goodhart's law
36:45 Gradient-free optimization
38:53 SMO, SVMs and hinge loss in neural networks
42:12 The bitter lesson and the limits of scaling
43:36 Where agents will help science in the next 12 months
45:13 Robotics and making climate tech cheaper to build
48:22 AI risk and misuse
50:51 Climate outlook: past 2°C
54:39 Why AI won't just solve climate change
57:04 Room-temperature superconductors, fusion and HVDC
1:02:25 FireSat: detecting wildfires from orbit
1:05:16 Why existing satellites aren't enough
1:07:53 The economics of 50 satellites
1:09:34 False positives
1:11:16 Closing

Topics

• Comparative advantage as the test for which science problems Google takes on

• Contrails: the physics, the AI pipeline, and the American Airlines trials

• ERA and AI agents for scientific software

• What scientists still do that agents can’t: taste, creativity, rigor

• Overfitting, Goodhart’s law, and agents exploiting bugs

• Gradient-free methods, SMO, and hinge loss

• The bitter lesson and the physical limits of scaling

• Climate outlook, fusion, and physical AI

• Room-temperature superconductors and long-distance power transmission

• FireSat and early wildfire detection

Music

- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Yuandong Tian spent many years at Meta FAIR and recently left to co-found Recursive Superintelligence, a company building AI that improves itself. In this episode, he tells us why.

We start with his work on computer Go, where he built DarkForest before AlphaGo came out. A few years later came OpenGo, which played Korean professionals on a single GPU and didn't lose a game. He then tried to bring RL to real-world problems and found that the design of the action space mattered more than the algorithm.

We also talk about Coconut, his paper on reasoning in latent space instead of tokens, and why frontier models still aren't trained that way.

The second half is about recursive self-improvement. Yuandong wrote his last paper at Meta together with GPT-5 and says it made him 6 to 10 times faster. That convinced him his own job could be replaced within five years. We ask him where agents still fall short and whether a new architecture can really beat transformers at scale. He also gives his view on the calls to restrict self-improving AI and makes the case for open source.

Recursive is hiring in San Francisco and London: talent@recursive.com

Topics

* Computer Go: DarkForest, AlphaGo and OpenGo
* Gradient-free optimization
* Action space design and neural architecture search
* Coconut and reasoning in latent space
* Understanding how neural networks learn representations
* Grokking, and writing a paper with GPT-5
* Recursive self-improvement and coding agents
* New architectures vs. transformers
* The NanoGPT speedrun
* Data efficiency and robotics
* Restricting self-improving AI
* Open source models

Timeline

0:00 Intro
0:45 From CMU to deep learning, and the AlexNet debates
5:35 AlphaGo and beating Go pros on a single GPU
10:27 Gradient-free optimization
13:16 Diffusion models and diversity
16:23 RL on real problems: why the action space matters
20:34 AutoML and architecture search
22:05 Coconut and reasoning in latent space
26:07 Why latent reasoning hasn't caught on
30:36 Adapting research to the LLM era
34:04 Why he left Meta to work on recursive self-improvement
37:10 Do we still need humans?
39:37 Where agents fall short
43:08 Can new architectures beat transformers at scale?
47:29 The hardest part of research to automate
50:16 What comes next for AI
52:52 Should self-improving AI be restricted?
54:11 Open source models
56:43 Hiring at Recursive

Music

- "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Chris Manning is a legend in NLP. He's a professor of linguistics and computer science at Stanford and ran the Stanford AI Lab. His course CS224N, Natural Language Processing with Deep Learning, is where a whole generation of researchers learned the field. If you've worked on anything in NLP over the last 25 years, you've probably used his work.

We start with what linguistics gave machine learning that ML wouldn't have figured out on its own, and what's left for NLP researchers now that LLMs handle most of the classic tasks. Chris thinks the open problems have moved up the stack to pragmatics and dialogue. Models are good at using context but still sound confident when they shouldn't.

Then we get into his recent work on how language models learn verb classes. Small GPT-2 models seem to form abstract categories right away instead of memorizing verbs one at a time, and Chris explains why distributed representations push them in that direction.

The second half is about meaning and representation. Chris makes the case that Yann LeCun underrates the role of language in intelligence, and revisits his debate with Emily Bender over whether text alone can teach meaning. We finish with ReFT, which steers a frozen model by editing its hidden states, and whether concepts really live in linear subspaces.

Timeline
00:00 Intro
01:05 What linguistics gave machine learning
03:27 Is NLP losing its focus on language?
06:29 What's left for NLP researchers in the LLM era
09:37 The next frontier: pragmatics and dialogue
13:01 Constructed languages and AI-to-AI communication
15:35 Do language models learn categories first?
23:43 Why transformers learn abstractions early
26:37 Does it hold at scale?
28:30 LeCun, JEPA and the role of language in intelligence
34:54 Can text alone teach meaning? The octopus debate
40:45 Diffusion language models vs transformers
43:39 ReFT: editing representations instead of weights
50:34 Weights or representations: where knowledge lives
52:37 Do concepts really live in linear subspaces?

Topics
NLP, computational linguistics, large language models, learning dynamics, meaning from form, world models, JEPA, diffusion LMs, ReFT, representation learning, interpretability

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Andrew Dai spent over a decade at Google Brain and DeepMind, where he co-wrote the 2015 paper that introduced language model pre-training followed by fine-tuning, and later co-led pre-training data for Gemini. He's now co-founder and CEO of Elorian AI, which is building models for visual reasoning.

We talk about how his pre-training result started as a bug, why next-token prediction scales better than other objectives, and what went wrong for Google in the early LLM race. The second half is about vision: why today's frontier models still can't count objects in a photo, why he thinks reasoning is fundamentally visual, and how his view of world models differs from JEPA.

Chapters

00:00 Intro
00:48 Andrew's background and the accidental discovery of pre-training
07:10 Why next-token prediction scales
10:09 How Google fell behind and the early days of Gemini
18:28 What makes training data good
28:16 Where visual understanding breaks down
46:01 World models, JEPA and robotics
55:27 Generation vs understanding, and what's next for Elorian

Topics

• Pre-training and fine-tuning

• Scaling and next-token prediction

• Gemini and Google's AI history

• Data quality and synthetic data

• Visual reasoning and counting

• World models and JEPA

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Zachary Lipton is an associate professor at Carnegie Mellon University and a co-founder of Abridge, a healthcare AI company (and a jazz saxophonist!). His research spans machine learning, healthcare, and the broader impact of AI.

In this episode, we discuss what AI can (and cannot yet) do in healthcare. We talk about why medicine is harder to automate than coding, AI scribes and clinical decision support, drug discovery, whether AI could eventually replace doctors, and the role of open models and intelligent routing.

In the second half, we turn to the future of AI research: whether academia can still compete, what AI PhD students should work on, and how Zach thinks about automation and the future of research.

Timeline

00:00 — Introduction
00:28 — Why healthcare is hard for AI
05:00 — Zach’s path into healthcare AI
22:13 — Where AI can have the biggest impact
32:01 — Can AI replace doctors?
41:41 — Open models and AI infrastructure
56:20 — The future of AI research
1:03:03 — What should AI PhD students work on?
1:27:00 — Advice for researchers and founders

Topics

AI in healthcare • AI doctors • drug discovery • clinical decision support • open-source AI • AI research • academia vs. industry • future of work

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

We talk with Sara Hooker, co-founder and CEO of Adaptation Lab, about why the next generation of AI may look very different from today's static models. Sara argues that models should continuously adapt to new tasks, data, users, and environments—and that doing this efficiently will require rethinking much more than fine-tuning.

We discuss continual learning, AutoScientist and automated research, why non-verifiable tasks may become the next major bottleneck, and why interfaces could be as important as the models themselves. We also get into open vs. closed models, distillation and Chinese AI labs, AI regulation and safety, cybersecurity and biorisk, AI companionship, and what may eventually come after Transformers and tokenization.

Topics

• Continuous learning and adaptive AI

• Fine-tuning, memory, and AutoScientist

• AI agents and automated research

• Non-verifiable tasks and human feedback

• Adaptive interfaces

• Open vs. closed models and distillation

• AI safety, regulation, cyber risk, and biorisk

• AI companionship and persuasion

• The limits of Transformers

• Multilingual models and tokenization

Chapters

00:00 — Introduction
02:15 — Why start another AI lab? The return of research
05:46 — What continuous learning actually means
12:04 — Should every company have its own adapting model?
13:59 — Fine-tuning and platforms like Tinker
18:04 — AutoScientist and automated optimization
22:52 — Can AI really improve its own research?
28:38 — The problem of non-verifiable tasks
31:30 — Human feedback and the limits of exponential progress
34:43 — Why the AI interface matters
40:36 — Distillation, China, and open models
49:05 — Open-model licensing
52:19 — Will open models catch closed models?
58:43 — AI regulation and compute thresholds
1:03:07 — AI safety and agent failures
1:10:19 — Biorisk vs. cybersecurity
1:14:03 — Persuasion, AI companionship, and overlooked risks
1:20:41 — Where will AI have the biggest real-world impact?
1:25:41 — What is missing from current AI architectures?
1:29:03 — Neurosymbolic AI
1:31:30 — Multilingual models and tokenization
1:34:02 — Byte-level models and alternatives to tokenization
1:35:03 — Closing

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Alexia Jolicoeur-Martineau is a Principal Researcher at Microsoft and the author of "Less is More: Recursive Reasoning with Tiny Networks," the paper behind the Tiny Recursive Model that hit about 45% on ARC-AGI-1 with a fraction of the parameters of frontier systems. It won the 2025 ARC Prize paper award.

She read the hierarchical reasoning paper, thought the potential was real and the explanation was not, and rebuilt it without the mouse brains: a small network that carries a hidden state and a current answer, thinks for a few steps, updates, and repeats, with the gradient truncated at each loop. We get into why puzzles suit this and autoregression doesn't, why she thinks LLMs are bad at molecules and more data won't fix it, and what she'd do with a trillion dollars.

Timeline

• 00:01 Intro

• 01:06 Leaving biostatistics, and why the field stagnated

• 06:47 GANs, diffusion, and research on four GPUs

• 12:58 What was wrong with the hierarchical reasoning paper

• 16:31 Tiny recursive models explained without the biology

• 22:35 Why puzzles favor recursion over left to right generation

• 24:15 Is the bitter lesson really bitter?

• 27:28 With infinite compute, would you still want small models?

• 32:00 Self improvement, memory, and a trillion dollars

• 37:01 Test time compute beyond chain of thought

• 40:41 Why chain of thought fails on molecules

• 45:17 Is there a universal representation?

• 48:06 What people are already building with TRM

• 55:22 Fixed point models and DEQ

• ---

  Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Topics

• Tiny Recursive Models and the ARC-AGI results

• What the hierarchical reasoning model was really doing

• Deep supervision and truncated backprop

• Looping transformers and parameter efficiency

• Why puzzles favor whole-context iteration over left to right generation

• Test time compute beyond chain of thought

• Latent reasoning and the Coconut line of work

• Why LLMs fail on chemistry and physics

• Representation learning and whether a universal representation exists

Rohan Anil spent eleven and a half years at Google, where he went from writing memory allocators to large-scale linear solvers, then optimization at Google Brain, where he co-developed distributed Shampoo and led optimization for PaLM and Gemini pre-training, including the work that produced Gemini Flash. He then joined Anthropic's pre-training team, and left before the IPO to co-found Core Automation with Jerry Tworek (ex-VP of Research at OpenAI). We talk with him about how Brain worked at its peak, why he left two of the world's best labs, and what he thinks is missing from today's models.

Rohan's view is that pre-training and RL were split by organizational convenience rather than by science. Pre-training builds a prior, and RL sharpens it to the tasks we care about, and neither gives a model a way to absorb new data or learn from its own experience once it is deployed. Post-training more every day plateaus, on-policy distillation plateaus, and in-context learning only goes as far as the context does. He argues the next architecture needs better ways to fold in new knowledge at inference time, and that this is a fundamental optimization question rather than a harness-engineering one.

We also get into why coding agents still fail on low-level systems work, his take on Muon, why second-order methods matter once you leave the noise-dominated regime, and why nobody can yet use a few million GPUs for a single training run.

Timeline

00:00 Intro
01:09 From computer vision to Google systems engineering
02:37 Large-scale linear solvers and sparse features
06:01 Getting into optimization: SDCA and Yonghui Wu's team
07:29 Joining the Shampoo crew
09:35 The Google Brain ethos, and why 2017 to 2019 was special
14:23 Is open research going to keep winning?
15:53 Frontier models are only as good as the prior you give them
17:30 Missing the language model wave, then Common Crawl and online distillation
18:31 Paternity leave, DALL-E Mini, and the 14 days that became two years
20:32 PaLM, Gemini pre-training, and Gemini Flash
23:59 The Shampoo origin story: Tomer Koren's two-week proof
26:54 Why leave Google for Anthropic
30:20 Why leave Anthropic for a startup
31:33 Meeting Jerry Tworek at Dolores Park
34:00 What Core Automation is building
36:17 Continual learning and the pre-training vs RL split
39:11 Why coding agents fail at kernels and low-level pipelines
42:00 The QR factorization kernel competition and reward hacking
44:57 Numerics, verification, and hardware that keeps changing
46:07 Are LLMs creative, or just good at search?
49:50 Getting models to extrapolate instead of interpolate
52:03 Why did we ever call it pre-training?
55:02 What RL is really learning
56:27 Competing with the big labs with fewer people
58:31 Will kernel generation keep old GPUs alive? Amdahl's law
1:01:58 Open source plans
1:02:55 Audience question: agentic optimizers
1:04:53 Audience question: Muon, Shampoo, and the future of second-order methods
1:08:58 Hiring at Core Automation

key topics

• Journey from Google Brain to startup

• Evolution of AI research and optimization

• Pre-training and reinforcement learning

• Kernel optimization and system efficiency

• Open source AI and collaborative research

• Challenges in AI creativity and exploration

• Future directions in continual learning and model scaling

  ---

• Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.

We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.

Topics:

• Implicit vs. explicit world models, and the case against JEPA-style objectives

• Compact belief states: why compression beats a growing KV cache

• Does research still matter in the age of scale? The Kimi K3 argument

• Agent-driven research: 2,000 pre-training experiments in 90 days

• The invention of CAPTCHA

• Optimizers from SGD and Vowpal Wabbit to Muon

• ---

Chapters

• 00:00 Why world models: the sample-complexity gap

• 09:48 The case against JEPA; a transformer-style implicit world model

• 15:52 Compact belief states: epicycles vs. heliocentrism

• 23:41 Does research still matter? The Kimi K3 argument

• 27:35 Open vs. closed models

• 35:57 Recursive self-improvement and agent-driven research

• 42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility

• 54:54 The invention of CAPTCHA

• 1:00:53 Optimizers: from Vowpal Wabbit to Muon

• ---

  Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Tabular data is still where most of machine learning actually happens in industry, and the field has changed a lot in the last few years. In this episode we talk with David Holzmüller, a researcher at INRIA and one of the people behind TabArena, TabICL and RealMLP, about what the state of the art looks like right now and how to pick a model for your own data.

We cover the shift to TabPFN-style foundation models that learn to learn from whole tables, why TabArena was built and what earlier benchmarks got wrong, what Google's new TabFM means for the leaderboard, and when gradient boosted trees are still the right tool. David explains why LLMs struggle with tables, shares an early result comparing Claude Opus against TabICL on tiny datasets, and walks through how to embed text columns for tabular models. We also get into time series vs tabular data, the open research problems he thinks matter most, and why classical ML libraries are so bad out of the box.

Links:
TabArena: https://tabarena.ai

Topics

• Tabular foundation models and in-context learning on tables

• TabArena and Beyond Arena: building a benchmark that stays honest

• TabFM, TabPFN, TabICL and the tradeoffs between them

• When boosted trees and MLPs still win (large data, CPU, fast inference)

• Why LLMs are inefficient on tabular data and where they might help

• Embedding text columns with language models

• Explainability, calibration and class imbalance

• Time series vs tabular data

• Open problems: invariances, synthetic data, uncertainty, scaling down

• Where the field is heading in the next five years

  ---

Chapters

0:00 Intro
0:31 What changed in tabular ML: TabPFN-style foundation models
2:22 Which model to try first? TabArena and how it was built
5:14 What older benchmarks got wrong, and Beyond Arena
8:45 GPU AutoML vs foundation models
10:40 Reading the leaderboard: TabFM, TabPFN, TabICL and the tradeoffs
12:47 Calibration, class imbalance and small vs large data
19:45 Explainability for black-box tabular models
21:34 Why LLMs are bad at tabular data
25:39 Claude Opus 4.6 vs TabICL on tiny datasets
27:51 New classifiers, five-year outlook, real vs synthetic pretraining
33:13 Embedding text columns for tabular foundation models
36:13 Time series vs tabular data
39:59 When gradient boosted trees still win, and feature engineering
45:31 Open research problems and where the field is heading
52:54 Better MLPs and why classical defaults are bad out of the box

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Charles Frye (Modal, ex-Weights & Biases, Berkeley PhD) joins Ravid and Allen to explain why modern AI research is bottlenecked by compute, and why simply buying more GPUs doesn't solve it. We cover the three problems every lab hits (underutilization, saturation, resource sharing), when companies should actually train their own models, why inference is a "bad algorithm" for today's hardware, NVIDIA's monopoly, the OpenAI/Hugging Face hack and what it says about open models, and whether we're in a compute bubble.

Key topics

• AI infrastructure challenges and when to train your own models

• GPU resource management and virtualization

• Inference optimization and speculative decoding

• The economics and future of AI hardware

• Agents, sandboxing, and open-model security

Chapters
00:00 Intro
01:03 Why AI needs special-purpose compute
03:22 Buying vs renting GPUs: the three problems
07:15 Modal's approach, and doing more with less compute
09:46 Do we actually need to spend more? The conflict-of-interest question
13:08 Should companies train their own models?
14:47 Efficient fine-tuning and prompts as fast weights
17:37 Are we in a compute bubble?
20:21 Why inference will dominate compute (the SQLite analogy)
22:42 Speculative decoding
26:44 Why scaling inference is hard, and neuromorphic hardware
28:36 Why NVIDIA's monopoly persists
33:09 Inference chip startups and the hardware lottery
35:24 How Modal stays hardware-agnostic (GPU snapshot restore)
38:45 Will agentic coding erode CUDA's moat?
41:18 Running one agent vs thousands: sandboxing at scale
46:27 The OpenAI/Hugging Face hack and open models as defenders
52:28 Rogue AI, self-replication, and fast takeoff
56:09 What's next: evals, embodiment, edge inference
1:00:27 Modal is hiring (modal.jobs)

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Stella Biderman, Executive Director of EleutherAI, joins us the week an OpenAI model autonomously broke out of its sandbox and hacked Hugging Face. Stella calls it what she thinks it is, an offensive cyber operation, and argues it’s part of a pattern: this is not the first containment failure at a frontier lab, and sandboxes have failed basically every time they’ve been tested for real.

So we spend a good chunk of the episode on what actual containment would look like. Stella’s argument is that the tools already exist, the labs just don’t use them: run dangerous capability evals on air-gapped networks with no route to the public internet, put the most sensitive testing in SCIF-style secure facilities, and treat model evaluation the way the security world treats classified systems rather than the way startups treat staging environments.

And yet Stella remains one of the world’s most prominent open-source advocates. From her perspective, the biggest risk isn’t the technology; it’s unchecked corporate power, and the only durable check on it is an independent scientific research establishment that doesn’t depend on the AI industry for its funding or its facts.

From there the conversation spans the geopolitics of Chinese open models and whether governments can restrict them, sovereign AI and what it would actually take for other countries to train their own models, why harnesses and UX drive more of AI’s perceived progress than raw intelligence, the AI-found counterexample to the Jacobian conjecture, and EleutherAI’s “Deep Ignorance” approach to making open-weight models safe by filtering hazardous knowledge out of pretraining.

key topics

• AI governance and regulation

• Cybersecurity incidents involving AI models

• Open source AI safety and security

• The role of independent research in AI safety

• Legal and ethical considerations in AI development

• ---

  Timeline

• 00:13 — Intro: Stella Biderman and EleutherAI, a real non-profit in AI

• 02:05 — News of the week: Kimi K3, and OpenAI's model autonomously hacking Hugging Face

• 05:49 — "Frontier labs can't be trusted": repeated containment failures, air-gapped networks and SCIFs vs. sandboxes

• 22:45 — Can governments ban open or Chinese models? Import restrictions and the six-month open/closed gap

• 27:05 — Why Stella is still pro-open-source: unchecked corporate power as the real danger

• 31:11 — The opioid epidemic analogy: avoiding both regulatory failure and overcorrection

• 34:57 — Offense vs. defense: why open access to AI has empirically favored defenders

• 37:28 — Chinese labs, the CCP, and why safety and fine-tuning are low-prestige work in China

• 42:19 — Sovereign AI: does every country need its own foundation model?

• 49:29 — Sampling, harnesses, and why ChatGPT was really a UX breakthrough

• 54:09 — AI solves the Jacobian conjecture: domain data beats raw intelligence

• 58:02 — Safety is contextual, not a model property — and what HAL 9000 got right

• 1:01:42 — Is Stella optimistic about the future?

• 1:02:50 — Deep Ignorance, the science of AI training dynamics, and how to get involved with EleutherAI

• ---

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Since coding agents became much better at the end of last year, there has been a huge debate about what comes next. If LLMs can transform coding so quickly, why not biology, drug discovery, or healthcare?

Anthropic is now putting a lot of attention into these areas. Dario Amodei has gone much further, arguing that powerful AI could help cure most diseases within five to ten years.

I posted skeptically about this on social media, and I don’t remember ever receiving so many supportive DMs. Many came from people working in biology, running labs, or developing drugs.

I should say clearly that I’m not an expert in drug development. I have a bachelor’s degree in computational biology and a PhD in computational neuroscience, but I don’t currently work directly in biology. For this post, I spoke with several people who run biological labs or work at the intersection of AI and biology.

Of course, nobody really knows what AI systems will be capable of five years from now. But based on what we have today, I think there is a basic problem with the argument that progress in coding will transfer to biology.

The verifier problem

One useful way to think about AI tasks is to ask how easily an answer can be verified.

The fields where scaling and reinforcement learning have produced the most dramatic progress usually have cheap, fast feedback. Code has compilers and unit tests. Formal mathematics has proof checkers. Games have scores and clearly defined rules.

An AI can generate an answer, test it, learn from the result, and repeat this loop millions of times. This is extremely important. It allows the system to search for answers without a human having to check every attempt.

This is why I don’t find the argument “AI can solve hard mathematical problems, so drugs are next” very convincing. The mathematical result may be impressive, but it also shows what happens when you can generate a candidate and check it almost for free.

What happens when checking an answer takes three months? What if it costs hundreds of thousands of dollars? What if the experiment is noisy, and even after it ends you still don’t know whether the result will generalize to humans?

That is much closer to biology.

A drug has to survive several different problems.

A simplified drug-development process has at least four parts: mechanism, molecule, development, and market. A program must succeed at all four. Failure in any one of them can kill the entire project.

Mechanism is the question of which biological process to intervene in.

We are worse at this than people think. High HDL cholesterol is associated with lower cardiovascular risk, yet several drugs that successfully raised HDL failed to improve clinical outcomes. The correlation was real. The intervention was wrong.

Vioxx worked as an anti-inflammatory, but it also increased cardiovascular risk. A drug can do exactly what it was designed to do and still cause serious harm somewhere else in the body. You cannot necessarily discover that by reading more papers or reasoning more carefully.

Molecule is the question of what physical object can implement the intervention.

Liraglutide and semaglutide target the same receptor and are structurally similar, yet one has a half-life of roughly 13 hours while the other lasts about a week. The difference is not simply whether they bind to the receptor. Small molecular modifications change how they bind to albumin, resist degradation, and survive in circulation.

This is why success in binder design should not be confused with success in drug design. Binding matters, but a real drug must also reach the correct tissue, remain stable, avoid unwanted interactions, be manufacturable, and behave safely inside a complete organism.

Development is where most of the time disappears.

Animal experiments take biological time. Some disease models require weeks or months before there is anything to measure. Then the experiment often has to be repeated to make sure the result was not noise.

Human trials are even harder. People must be recruited. Safety must be monitored. Some diseases progress slowly, so researchers need to wait long enough to see whether the treatment changes an actual outcome. According to the FDA, Phase II trials can take several months to two years, while Phase III trials often take one to four years.

You cannot solve all of this by buying more GPUs.

The market also matters, even if people prefer not to discuss it. A known mechanism may be less risky but lead to a crowded market. A novel mechanism may have much greater value, but it also carries a much higher risk of failure. Some diseases have too few patients or too little purchasing power to attract enough investment.

There is no version of this process where everything becomes easy at the same time.

Biology has verifiers, but they are bad ones.

It would be too strong to say that biology has no verifiers. It has experiments.

The problem is that the important experiments are often slow, expensive, noisy, and incomplete.

Near the beginning of the pipeline, researchers can test whether a molecule binds to a target. But after that, the questions become much more difficult. Does it survive in the body? Does it reach the right tissue? Is it toxic? Does it work in an animal? Does the animal model actually predict what will happen in a person?

Eventually, you cannot test whether a compound is safe in humans without giving it to humans. You cannot identify rare side effects without studying enough people for long enough. You cannot determine whether a treatment delays a slowly progressing disease by running a faster simulation.

A clinical trial is not simply a very slow unit test. It is a noisy experiment conducted on a heterogeneous population, subject to ethical and operational constraints. In many cases, you get only a few serious attempts before the money or patent window runs out.

What Anthropic actually showed

None of this means that Anthropic’s recent result is unimportant.

Anthropic reported that Claude managed de novo protein-binder design campaigns and produced successful binders for 14 of 15 evaluated targets. External laboratories then produced and tested the designs. This is a genuinely impressive result. Anthropic also released the prompts, designs, and experimental data.

The company’s research post is actually careful. It says that minibinders are not therapeutics and that a high-affinity binder is only the initial step toward a drug-like molecule.

But this caveat is not a small detail. It is the main point.

Binder design is one of the biological problems most compatible with current AI. You can generate many candidates computationally, rank them, and then test them with a relatively standardized experiment. It looks more like the problems where AI has already succeeded.

That is probably why we are seeing progress there first.

The mistake is assuming that because AI compressed this step, it will compress every later step at the same rate.

The scientific result is real. The questionable part is how the result changes as it moves upward: from designing protein binders to accelerating drug discovery to compressing decades of biological progress to curing most diseases within ten years.

At every step, a caveat disappears.

The economics are different too.

There is also a mismatch between the economics of frontier AI and those of drug development.

Software can be deployed globally, measured immediately, and updated continuously. Drug development requires long capital cycles and often results in many binary failures. A molecule can take years and cost hundreds of millions of dollars before a clinical trial shows it does not work.

This does not mean AI companies should avoid biology. But once a company begins to own and develop drugs, the economics start to look much more like pharma than software.

The promise to cure disease also does more than describe a research program. It provides a moral justification for enormous investments in compute, energy, and AI infrastructure. It makes the concentration of money and power around a few AI companies sound not only profitable yet necessary for humanity.

That doesn’t invalidate the scientific work. It does mean we should demand stronger evidence before accepting the larger story.

What will happen?

I use AI every day, and I am thrilled about projects that use it to search molecular space, design proteins, interpret experiments, and generate hypotheses.

AI will improve biological research. It will likely yield more drug candidates, and some may be much better than those we can design today.

But producing a prospective candidate starts the process. It does not finish it.

My view is that AI will substantially accelerate some parts of drug discovery without proportionally shortening the entire path from an idea to a safe, effective, widely available treatment.

The longest parts of that process are not simply periods when scientists sit around thinking too slowly. Researchers have to manufacture molecules, perturb living systems, recruit patients, observe outcomes, and wait for biology to reveal what happens.

This may change. Better simulations, robotic labs, stronger models, and more predictive animal or cellular systems could shorten some of these stages. AI may eventually become so accurate that it can select the rare molecule that satisfies efficacy, safety, delivery, stability, and manufacturing requirements on the first attempt.

But we have not seen evidence for that yet. And even doing it once would be very different from curing most diseases.

Anthropic’s binder result supports a narrower conclusion. It is less dramatic than curing most diseases, but still important:

AI is becoming extremely good at accelerating the parts of biology that resemble computation. Unfortunately, much of medicine is slow precisely because the decisive parts do not move quickly.

In this episode, Frank Hutter joins us to talk about TabPFN and why tabular data is suddenly the hottest problem in deep learning. Frank is a professor at the University of Freiburg and spent 15 years building the AutoML field before founding Prior Labs, which SAP just acquired for over a billion dollars.

We get into why deep learning failed on tables for a decade and what in-context learning changed, how TabPFN is trained entirely on synthetic data, and why a model that never saw a real time series ended up beating specialized forecasting models. Frank also explains the architecture tricks behind scaling from 10,000 to a million rows, where LLMs fit into data science (and where they embarrassingly don't), and what happens to XGBoost from here.

Beyond the research, Frank talks about the jump from professor to co-CEO, why he refused to merge his 45-person team into SAP's 110,000 employees, the open-weights licensing debate, and the case for building a frontier lab in Freiburg rather than San Francisco.

key topics

• The role of foundation models in tabular data

• Impact of SAP acquisition on Pro Labs

• The evolution of AutoML and hyperparameter optimization

• Challenges and solutions for large context in models

• Open source models and licensing strategies

• The importance of independence for startup agility

• Future directions in AI for science and medicine

• ---

  00:00 Intro
  00:34 The SAP acquisition and staying independent
  07:39 Why tabular data is the next big thing in deep learning
  14:19 What makes tabular data hard
  19:14 AutoML, AutoGluon, and fifteen years of hyperparameter tuning
  28:27 Scaling TabPFN: context limits and architectures
  34:35 Agentic data science and LLMs
  39:30 Online learning, time series, and Bayesian inference in a forward pass
  47:05 Open weights and the license debate
  54:51 Will LLMs and tabular models merge?
  1:00:01 From academia to startup
  1:09:42 Why build in Europe
  1:12:53 Audience questions and hiring

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks.

We talk about why deep learning theory is finally catching up to practice,  including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics.

The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire.

We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm.

key topics

• Connections between physics, neuroscience, and AI

• Emergent properties in complex systems

• Scaling laws in language models

• Data efficiency and pruning in AI

• Neuroscience insights into consciousness and self

• The future of AI and brain modeling

• ---

  Chapters

  00:00 Introduction to Surya Ganguli

  00:57 Surya's Background: From String Theory to Neuroscience

  02:22 Emergent Properties in Physics, Neuroscience, and AI

  03:16 Energy Landscapes and Loss Landscapes in High Dimensions

  04:07 Why Local Minima Don't Exist in High-Dimensional AI

  05:22 Gradient-Based vs. Gradient-Free Learning Methods

  08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges

  13:48 Scaling Laws and Data Efficiency in Language Models

  18:10 Properties of Data that Affect Scaling Laws

  22:04 Constructing Non-Redundant Data Sets for Better Learning

  24:32 Theory vs. Empirical Results in AI Research

  32:19 Fundamental Components of Deep Learning: Are They Changing?

  34:31 Future Paradigms in AI Beyond Current Models

  37:22 Teaching AI and Humans: Paradigm Shifts in Learning

  41:37 Consciousness, Self, and the Brain: Surya's Perspectives

  49:49 Neuroscience and AI: Understanding the Brain and Consciousness

  01:02:03 Understanding the Brain: Challenges and Opportunities

  01:09:21 Brain-Computer Interfaces and AI in Neuroscience

• ---

  Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.