Andrew Dai spent over a decade at Google Brain and DeepMind, where he co-wrote the 2015 paper that introduced language model pre-training followed by fine-tuning, and later co-led pre-training data for Gemini. He's now co-founder and CEO of Elorian AI, which is building models for visual reasoning.

We talk about how his pre-training result started as a bug, why next-token prediction scales better than other objectives, and what went wrong for Google in the early LLM race. The second half is about vision: why today's frontier models still can't count objects in a photo, why he thinks reasoning is fundamentally visual, and how his view of world models differs from JEPA.

Chapters

00:00 Intro
00:48 Andrew's background and the accidental discovery of pre-training
07:10 Why next-token prediction scales
10:09 How Google fell behind and the early days of Gemini
18:28 What makes training data good
28:16 Where visual understanding breaks down
46:01 World models, JEPA and robotics
55:27 Generation vs understanding, and what's next for Elorian

Topics

• Pre-training and fine-tuning

• Scaling and next-token prediction

• Gemini and Google's AI history

• Data quality and synthetic data

• Visual reasoning and counting

• World models and JEPA

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Zachary Lipton is an associate professor at Carnegie Mellon University and a co-founder of Abridge, a healthcare AI company (and a jazz saxophonist!). His research spans machine learning, healthcare, and the broader impact of AI.

In this episode, we discuss what AI can (and cannot yet) do in healthcare. We talk about why medicine is harder to automate than coding, AI scribes and clinical decision support, drug discovery, whether AI could eventually replace doctors, and the role of open models and intelligent routing.

In the second half, we turn to the future of AI research: whether academia can still compete, what AI PhD students should work on, and how Zach thinks about automation and the future of research.

Timeline

00:00 — Introduction
00:28 — Why healthcare is hard for AI
05:00 — Zach’s path into healthcare AI
22:13 — Where AI can have the biggest impact
32:01 — Can AI replace doctors?
41:41 — Open models and AI infrastructure
56:20 — The future of AI research
1:03:03 — What should AI PhD students work on?
1:27:00 — Advice for researchers and founders

Topics

AI in healthcare • AI doctors • drug discovery • clinical decision support • open-source AI • AI research • academia vs. industry • future of work

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

We talk with Sara Hooker, co-founder and CEO of Adaptation Lab, about why the next generation of AI may look very different from today's static models. Sara argues that models should continuously adapt to new tasks, data, users, and environments—and that doing this efficiently will require rethinking much more than fine-tuning.

We discuss continual learning, AutoScientist and automated research, why non-verifiable tasks may become the next major bottleneck, and why interfaces could be as important as the models themselves. We also get into open vs. closed models, distillation and Chinese AI labs, AI regulation and safety, cybersecurity and biorisk, AI companionship, and what may eventually come after Transformers and tokenization.

Topics

• Continuous learning and adaptive AI

• Fine-tuning, memory, and AutoScientist

• AI agents and automated research

• Non-verifiable tasks and human feedback

• Adaptive interfaces

• Open vs. closed models and distillation

• AI safety, regulation, cyber risk, and biorisk

• AI companionship and persuasion

• The limits of Transformers

• Multilingual models and tokenization

Chapters

00:00 — Introduction
02:15 — Why start another AI lab? The return of research
05:46 — What continuous learning actually means
12:04 — Should every company have its own adapting model?
13:59 — Fine-tuning and platforms like Tinker
18:04 — AutoScientist and automated optimization
22:52 — Can AI really improve its own research?
28:38 — The problem of non-verifiable tasks
31:30 — Human feedback and the limits of exponential progress
34:43 — Why the AI interface matters
40:36 — Distillation, China, and open models
49:05 — Open-model licensing
52:19 — Will open models catch closed models?
58:43 — AI regulation and compute thresholds
1:03:07 — AI safety and agent failures
1:10:19 — Biorisk vs. cybersecurity
1:14:03 — Persuasion, AI companionship, and overlooked risks
1:20:41 — Where will AI have the biggest real-world impact?
1:25:41 — What is missing from current AI architectures?
1:29:03 — Neurosymbolic AI
1:31:30 — Multilingual models and tokenization
1:34:02 — Byte-level models and alternatives to tokenization
1:35:03 — Closing

Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Alexia Jolicoeur-Martineau is a Principal Researcher at Microsoft and the author of "Less is More: Recursive Reasoning with Tiny Networks," the paper behind the Tiny Recursive Model that hit about 45% on ARC-AGI-1 with a fraction of the parameters of frontier systems. It won the 2025 ARC Prize paper award.

She read the hierarchical reasoning paper, thought the potential was real and the explanation was not, and rebuilt it without the mouse brains: a small network that carries a hidden state and a current answer, thinks for a few steps, updates, and repeats, with the gradient truncated at each loop. We get into why puzzles suit this and autoregression doesn't, why she thinks LLMs are bad at molecules and more data won't fix it, and what she'd do with a trillion dollars.

Timeline

• 00:01 Intro

• 01:06 Leaving biostatistics, and why the field stagnated

• 06:47 GANs, diffusion, and research on four GPUs

• 12:58 What was wrong with the hierarchical reasoning paper

• 16:31 Tiny recursive models explained without the biology

• 22:35 Why puzzles favor recursion over left to right generation

• 24:15 Is the bitter lesson really bitter?

• 27:28 With infinite compute, would you still want small models?

• 32:00 Self improvement, memory, and a trillion dollars

• 37:01 Test time compute beyond chain of thought

• 40:41 Why chain of thought fails on molecules

• 45:17 Is there a universal representation?

• 48:06 What people are already building with TRM

• 55:22 Fixed point models and DEQ

• ---

  Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Topics

• Tiny Recursive Models and the ARC-AGI results

• What the hierarchical reasoning model was really doing

• Deep supervision and truncated backprop

• Looping transformers and parameter efficiency

• Why puzzles favor whole-context iteration over left to right generation

• Test time compute beyond chain of thought

• Latent reasoning and the Coconut line of work

• Why LLMs fail on chemistry and physics

• Representation learning and whether a universal representation exists

Rohan Anil spent eleven and a half years at Google, where he went from writing memory allocators to large-scale linear solvers, then optimization at Google Brain, where he co-developed distributed Shampoo and led optimization for PaLM and Gemini pre-training, including the work that produced Gemini Flash. He then joined Anthropic's pre-training team, and left before the IPO to co-found Core Automation with Jerry Tworek (ex-VP of Research at OpenAI). We talk with him about how Brain worked at its peak, why he left two of the world's best labs, and what he thinks is missing from today's models.

Rohan's view is that pre-training and RL were split by organizational convenience rather than by science. Pre-training builds a prior, and RL sharpens it to the tasks we care about, and neither gives a model a way to absorb new data or learn from its own experience once it is deployed. Post-training more every day plateaus, on-policy distillation plateaus, and in-context learning only goes as far as the context does. He argues the next architecture needs better ways to fold in new knowledge at inference time, and that this is a fundamental optimization question rather than a harness-engineering one.

We also get into why coding agents still fail on low-level systems work, his take on Muon, why second-order methods matter once you leave the noise-dominated regime, and why nobody can yet use a few million GPUs for a single training run.

Timeline

00:00 Intro
01:09 From computer vision to Google systems engineering
02:37 Large-scale linear solvers and sparse features
06:01 Getting into optimization: SDCA and Yonghui Wu's team
07:29 Joining the Shampoo crew
09:35 The Google Brain ethos, and why 2017 to 2019 was special
14:23 Is open research going to keep winning?
15:53 Frontier models are only as good as the prior you give them
17:30 Missing the language model wave, then Common Crawl and online distillation
18:31 Paternity leave, DALL-E Mini, and the 14 days that became two years
20:32 PaLM, Gemini pre-training, and Gemini Flash
23:59 The Shampoo origin story: Tomer Koren's two-week proof
26:54 Why leave Google for Anthropic
30:20 Why leave Anthropic for a startup
31:33 Meeting Jerry Tworek at Dolores Park
34:00 What Core Automation is building
36:17 Continual learning and the pre-training vs RL split
39:11 Why coding agents fail at kernels and low-level pipelines
42:00 The QR factorization kernel competition and reward hacking
44:57 Numerics, verification, and hardware that keeps changing
46:07 Are LLMs creative, or just good at search?
49:50 Getting models to extrapolate instead of interpolate
52:03 Why did we ever call it pre-training?
55:02 What RL is really learning
56:27 Competing with the big labs with fewer people
58:31 Will kernel generation keep old GPUs alive? Amdahl's law
1:01:58 Open source plans
1:02:55 Audience question: agentic optimizers
1:04:53 Audience question: Muon, Shampoo, and the future of second-order methods
1:08:58 Hiring at Core Automation

key topics

• Journey from Google Brain to startup

• Evolution of AI research and optimization

• Pre-training and reinforcement learning

• Kernel optimization and system efficiency

• Open source AI and collaborative research

• Challenges in AI creativity and exploration

• Future directions in continual learning and model scaling

  ---

• Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.

We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.

Topics:

• Implicit vs. explicit world models, and the case against JEPA-style objectives

• Compact belief states: why compression beats a growing KV cache

• Does research still matter in the age of scale? The Kimi K3 argument

• Agent-driven research: 2,000 pre-training experiments in 90 days

• The invention of CAPTCHA

• Optimizers from SGD and Vowpal Wabbit to Muon

• ---

Chapters

• 00:00 Why world models: the sample-complexity gap

• 09:48 The case against JEPA; a transformer-style implicit world model

• 15:52 Compact belief states: epicycles vs. heliocentrism

• 23:41 Does research still matter? The Kimi K3 argument

• 27:35 Open vs. closed models

• 35:57 Recursive self-improvement and agent-driven research

• 42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility

• 54:54 The invention of CAPTCHA

• 1:00:53 Optimizers: from Vowpal Wabbit to Muon

• ---

  Music

• "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Tabular data is still where most of machine learning actually happens in industry, and the field has changed a lot in the last few years. In this episode we talk with David Holzmüller, a researcher at INRIA and one of the people behind TabArena, TabICL and RealMLP, about what the state of the art looks like right now and how to pick a model for your own data.

We cover the shift to TabPFN-style foundation models that learn to learn from whole tables, why TabArena was built and what earlier benchmarks got wrong, what Google's new TabFM means for the leaderboard, and when gradient boosted trees are still the right tool. David explains why LLMs struggle with tables, shares an early result comparing Claude Opus against TabICL on tiny datasets, and walks through how to embed text columns for tabular models. We also get into time series vs tabular data, the open research problems he thinks matter most, and why classical ML libraries are so bad out of the box.

Links:
TabArena: https://tabarena.ai

Topics

• Tabular foundation models and in-context learning on tables

• TabArena and Beyond Arena: building a benchmark that stays honest

• TabFM, TabPFN, TabICL and the tradeoffs between them

• When boosted trees and MLPs still win (large data, CPU, fast inference)

• Why LLMs are inefficient on tabular data and where they might help

• Embedding text columns with language models

• Explainability, calibration and class imbalance

• Time series vs tabular data

• Open problems: invariances, synthetic data, uncertainty, scaling down

• Where the field is heading in the next five years

  ---

Chapters

0:00 Intro
0:31 What changed in tabular ML: TabPFN-style foundation models
2:22 Which model to try first? TabArena and how it was built
5:14 What older benchmarks got wrong, and Beyond Arena
8:45 GPU AutoML vs foundation models
10:40 Reading the leaderboard: TabFM, TabPFN, TabICL and the tradeoffs
12:47 Calibration, class imbalance and small vs large data
19:45 Explainability for black-box tabular models
21:34 Why LLMs are bad at tabular data
25:39 Claude Opus 4.6 vs TabICL on tiny datasets
27:51 New classifiers, five-year outlook, real vs synthetic pretraining
33:13 Embedding text columns for tabular foundation models
36:13 Time series vs tabular data
39:59 When gradient boosted trees still win, and feature engineering
45:31 Open research problems and where the field is heading
52:54 Better MLPs and why classical defaults are bad out of the box

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Charles Frye (Modal, ex-Weights & Biases, Berkeley PhD) joins Ravid and Allen to explain why modern AI research is bottlenecked by compute, and why simply buying more GPUs doesn't solve it. We cover the three problems every lab hits (underutilization, saturation, resource sharing), when companies should actually train their own models, why inference is a "bad algorithm" for today's hardware, NVIDIA's monopoly, the OpenAI/Hugging Face hack and what it says about open models, and whether we're in a compute bubble.

Key topics

• AI infrastructure challenges and when to train your own models

• GPU resource management and virtualization

• Inference optimization and speculative decoding

• The economics and future of AI hardware

• Agents, sandboxing, and open-model security

Chapters
00:00 Intro
01:03 Why AI needs special-purpose compute
03:22 Buying vs renting GPUs: the three problems
07:15 Modal's approach, and doing more with less compute
09:46 Do we actually need to spend more? The conflict-of-interest question
13:08 Should companies train their own models?
14:47 Efficient fine-tuning and prompts as fast weights
17:37 Are we in a compute bubble?
20:21 Why inference will dominate compute (the SQLite analogy)
22:42 Speculative decoding
26:44 Why scaling inference is hard, and neuromorphic hardware
28:36 Why NVIDIA's monopoly persists
33:09 Inference chip startups and the hardware lottery
35:24 How Modal stays hardware-agnostic (GPU snapshot restore)
38:45 Will agentic coding erode CUDA's moat?
41:18 Running one agent vs thousands: sandboxing at scale
46:27 The OpenAI/Hugging Face hack and open models as defenders
52:28 Rogue AI, self-replication, and fast takeoff
56:09 What's next: evals, embodiment, edge inference
1:00:27 Modal is hiring (modal.jobs)

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0

Stella Biderman, Executive Director of EleutherAI, joins us the week an OpenAI model autonomously broke out of its sandbox and hacked Hugging Face. Stella calls it what she thinks it is, an offensive cyber operation, and argues it’s part of a pattern: this is not the first containment failure at a frontier lab, and sandboxes have failed basically every time they’ve been tested for real.

So we spend a good chunk of the episode on what actual containment would look like. Stella’s argument is that the tools already exist, the labs just don’t use them: run dangerous capability evals on air-gapped networks with no route to the public internet, put the most sensitive testing in SCIF-style secure facilities, and treat model evaluation the way the security world treats classified systems rather than the way startups treat staging environments.

And yet Stella remains one of the world’s most prominent open-source advocates. From her perspective, the biggest risk isn’t the technology; it’s unchecked corporate power, and the only durable check on it is an independent scientific research establishment that doesn’t depend on the AI industry for its funding or its facts.

From there the conversation spans the geopolitics of Chinese open models and whether governments can restrict them, sovereign AI and what it would actually take for other countries to train their own models, why harnesses and UX drive more of AI’s perceived progress than raw intelligence, the AI-found counterexample to the Jacobian conjecture, and EleutherAI’s “Deep Ignorance” approach to making open-weight models safe by filtering hazardous knowledge out of pretraining.

key topics

• AI governance and regulation

• Cybersecurity incidents involving AI models

• Open source AI safety and security

• The role of independent research in AI safety

• Legal and ethical considerations in AI development

• ---

  Timeline

• 00:13 — Intro: Stella Biderman and EleutherAI, a real non-profit in AI

• 02:05 — News of the week: Kimi K3, and OpenAI's model autonomously hacking Hugging Face

• 05:49 — "Frontier labs can't be trusted": repeated containment failures, air-gapped networks and SCIFs vs. sandboxes

• 22:45 — Can governments ban open or Chinese models? Import restrictions and the six-month open/closed gap

• 27:05 — Why Stella is still pro-open-source: unchecked corporate power as the real danger

• 31:11 — The opioid epidemic analogy: avoiding both regulatory failure and overcorrection

• 34:57 — Offense vs. defense: why open access to AI has empirically favored defenders

• 37:28 — Chinese labs, the CCP, and why safety and fine-tuning are low-prestige work in China

• 42:19 — Sovereign AI: does every country need its own foundation model?

• 49:29 — Sampling, harnesses, and why ChatGPT was really a UX breakthrough

• 54:09 — AI solves the Jacobian conjecture: domain data beats raw intelligence

• 58:02 — Safety is contextual, not a model property — and what HAL 9000 got right

• 1:01:42 — Is Stella optimistic about the future?

• 1:02:50 — Deep Ignorance, the science of AI training dynamics, and how to get involved with EleutherAI

• ---

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Since coding agents became much better at the end of last year, there has been a huge debate about what comes next. If LLMs can transform coding so quickly, why not biology, drug discovery, or healthcare?

Anthropic is now putting a lot of attention into these areas. Dario Amodei has gone much further, arguing that powerful AI could help cure most diseases within five to ten years.

I posted skeptically about this on social media, and I don’t remember ever receiving so many supportive DMs. Many came from people working in biology, running labs, or developing drugs.

I should say clearly that I’m not an expert in drug development. I have a bachelor’s degree in computational biology and a PhD in computational neuroscience, but I don’t currently work directly in biology. For this post, I spoke with several people who run biological labs or work at the intersection of AI and biology.

Of course, nobody really knows what AI systems will be capable of five years from now. But based on what we have today, I think there is a basic problem with the argument that progress in coding will transfer to biology.

The verifier problem

One useful way to think about AI tasks is to ask how easily an answer can be verified.

The fields where scaling and reinforcement learning have produced the most dramatic progress usually have cheap, fast feedback. Code has compilers and unit tests. Formal mathematics has proof checkers. Games have scores and clearly defined rules.

An AI can generate an answer, test it, learn from the result, and repeat this loop millions of times. This is extremely important. It allows the system to search for answers without a human having to check every attempt.

This is why I don’t find the argument “AI can solve hard mathematical problems, so drugs are next” very convincing. The mathematical result may be impressive, but it also shows what happens when you can generate a candidate and check it almost for free.

What happens when checking an answer takes three months? What if it costs hundreds of thousands of dollars? What if the experiment is noisy, and even after it ends you still don’t know whether the result will generalize to humans?

That is much closer to biology.

A drug has to survive several different problems.

A simplified drug-development process has at least four parts: mechanism, molecule, development, and market. A program must succeed at all four. Failure in any one of them can kill the entire project.

Mechanism is the question of which biological process to intervene in.

We are worse at this than people think. High HDL cholesterol is associated with lower cardiovascular risk, yet several drugs that successfully raised HDL failed to improve clinical outcomes. The correlation was real. The intervention was wrong.

Vioxx worked as an anti-inflammatory, but it also increased cardiovascular risk. A drug can do exactly what it was designed to do and still cause serious harm somewhere else in the body. You cannot necessarily discover that by reading more papers or reasoning more carefully.

Molecule is the question of what physical object can implement the intervention.

Liraglutide and semaglutide target the same receptor and are structurally similar, yet one has a half-life of roughly 13 hours while the other lasts about a week. The difference is not simply whether they bind to the receptor. Small molecular modifications change how they bind to albumin, resist degradation, and survive in circulation.

This is why success in binder design should not be confused with success in drug design. Binding matters, but a real drug must also reach the correct tissue, remain stable, avoid unwanted interactions, be manufacturable, and behave safely inside a complete organism.

Development is where most of the time disappears.

Animal experiments take biological time. Some disease models require weeks or months before there is anything to measure. Then the experiment often has to be repeated to make sure the result was not noise.

Human trials are even harder. People must be recruited. Safety must be monitored. Some diseases progress slowly, so researchers need to wait long enough to see whether the treatment changes an actual outcome. According to the FDA, Phase II trials can take several months to two years, while Phase III trials often take one to four years.

You cannot solve all of this by buying more GPUs.

The market also matters, even if people prefer not to discuss it. A known mechanism may be less risky but lead to a crowded market. A novel mechanism may have much greater value, but it also carries a much higher risk of failure. Some diseases have too few patients or too little purchasing power to attract enough investment.

There is no version of this process where everything becomes easy at the same time.

Biology has verifiers, but they are bad ones.

It would be too strong to say that biology has no verifiers. It has experiments.

The problem is that the important experiments are often slow, expensive, noisy, and incomplete.

Near the beginning of the pipeline, researchers can test whether a molecule binds to a target. But after that, the questions become much more difficult. Does it survive in the body? Does it reach the right tissue? Is it toxic? Does it work in an animal? Does the animal model actually predict what will happen in a person?

Eventually, you cannot test whether a compound is safe in humans without giving it to humans. You cannot identify rare side effects without studying enough people for long enough. You cannot determine whether a treatment delays a slowly progressing disease by running a faster simulation.

A clinical trial is not simply a very slow unit test. It is a noisy experiment conducted on a heterogeneous population, subject to ethical and operational constraints. In many cases, you get only a few serious attempts before the money or patent window runs out.

What Anthropic actually showed

None of this means that Anthropic’s recent result is unimportant.

Anthropic reported that Claude managed de novo protein-binder design campaigns and produced successful binders for 14 of 15 evaluated targets. External laboratories then produced and tested the designs. This is a genuinely impressive result. Anthropic also released the prompts, designs, and experimental data.

The company’s research post is actually careful. It says that minibinders are not therapeutics and that a high-affinity binder is only the initial step toward a drug-like molecule.

But this caveat is not a small detail. It is the main point.

Binder design is one of the biological problems most compatible with current AI. You can generate many candidates computationally, rank them, and then test them with a relatively standardized experiment. It looks more like the problems where AI has already succeeded.

That is probably why we are seeing progress there first.

The mistake is assuming that because AI compressed this step, it will compress every later step at the same rate.

The scientific result is real. The questionable part is how the result changes as it moves upward: from designing protein binders to accelerating drug discovery to compressing decades of biological progress to curing most diseases within ten years.

At every step, a caveat disappears.

The economics are different too.

There is also a mismatch between the economics of frontier AI and those of drug development.

Software can be deployed globally, measured immediately, and updated continuously. Drug development requires long capital cycles and often results in many binary failures. A molecule can take years and cost hundreds of millions of dollars before a clinical trial shows it does not work.

This does not mean AI companies should avoid biology. But once a company begins to own and develop drugs, the economics start to look much more like pharma than software.

The promise to cure disease also does more than describe a research program. It provides a moral justification for enormous investments in compute, energy, and AI infrastructure. It makes the concentration of money and power around a few AI companies sound not only profitable yet necessary for humanity.

That doesn’t invalidate the scientific work. It does mean we should demand stronger evidence before accepting the larger story.

What will happen?

I use AI every day, and I am thrilled about projects that use it to search molecular space, design proteins, interpret experiments, and generate hypotheses.

AI will improve biological research. It will likely yield more drug candidates, and some may be much better than those we can design today.

But producing a prospective candidate starts the process. It does not finish it.

My view is that AI will substantially accelerate some parts of drug discovery without proportionally shortening the entire path from an idea to a safe, effective, widely available treatment.

The longest parts of that process are not simply periods when scientists sit around thinking too slowly. Researchers have to manufacture molecules, perturb living systems, recruit patients, observe outcomes, and wait for biology to reveal what happens.

This may change. Better simulations, robotic labs, stronger models, and more predictive animal or cellular systems could shorten some of these stages. AI may eventually become so accurate that it can select the rare molecule that satisfies efficacy, safety, delivery, stability, and manufacturing requirements on the first attempt.

But we have not seen evidence for that yet. And even doing it once would be very different from curing most diseases.

Anthropic’s binder result supports a narrower conclusion. It is less dramatic than curing most diseases, but still important:

AI is becoming extremely good at accelerating the parts of biology that resemble computation. Unfortunately, much of medicine is slow precisely because the decisive parts do not move quickly.

In this episode, Frank Hutter joins us to talk about TabPFN and why tabular data is suddenly the hottest problem in deep learning. Frank is a professor at the University of Freiburg and spent 15 years building the AutoML field before founding Prior Labs, which SAP just acquired for over a billion dollars.

We get into why deep learning failed on tables for a decade and what in-context learning changed, how TabPFN is trained entirely on synthetic data, and why a model that never saw a real time series ended up beating specialized forecasting models. Frank also explains the architecture tricks behind scaling from 10,000 to a million rows, where LLMs fit into data science (and where they embarrassingly don't), and what happens to XGBoost from here.

Beyond the research, Frank talks about the jump from professor to co-CEO, why he refused to merge his 45-person team into SAP's 110,000 employees, the open-weights licensing debate, and the case for building a frontier lab in Freiburg rather than San Francisco.

key topics

• The role of foundation models in tabular data

• Impact of SAP acquisition on Pro Labs

• The evolution of AutoML and hyperparameter optimization

• Challenges and solutions for large context in models

• Open source models and licensing strategies

• The importance of independence for startup agility

• Future directions in AI for science and medicine

• ---

  00:00 Intro
  00:34 The SAP acquisition and staying independent
  07:39 Why tabular data is the next big thing in deep learning
  14:19 What makes tabular data hard
  19:14 AutoML, AutoGluon, and fifteen years of hyperparameter tuning
  28:27 Scaling TabPFN: context limits and architectures
  34:35 Agentic data science and LLMs
  39:30 Online learning, time series, and Bayesian inference in a forward pass
  47:05 Open weights and the license debate
  54:51 Will LLMs and tabular models merge?
  1:00:01 From academia to startup
  1:09:42 Why build in Europe
  1:12:53 Audience questions and hiring

• Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Surya Ganguli is a professor at Stanford and VP at General Catalyst, working at the intersection of physics, neuroscience, and AI. He started in string theory, moved to theoretical neuroscience, and now uses tools from statistical physics to understand both brains and neural networks.

We talk about why deep learning theory is finally catching up to practice,  including his group's recent work explaining neural scaling laws, and why smarter data selection could beat them entirely. He also tells the origin story of diffusion models, which were invented in his lab as an attempt to violate the second law of thermodynamics.

The second half turns to the brain: what happens to a mouse's sense of self on ketamine, how stimulating a handful of neurons can induce hallucinations, and a method his lab developed to get a neuron deep in a monkey's brain to describe, in English, what makes it fire.

We close on where he thinks AI is going wrong: models train on ten trillion tokens while humans hear a hundred million words, because we don't teach children with gradients; we tell them the algorithm.

key topics

• Connections between physics, neuroscience, and AI

• Emergent properties in complex systems

• Scaling laws in language models

• Data efficiency and pruning in AI

• Neuroscience insights into consciousness and self

• The future of AI and brain modeling

• ---

  Chapters

  00:00 Introduction to Surya Ganguli

  00:57 Surya's Background: From String Theory to Neuroscience

  02:22 Emergent Properties in Physics, Neuroscience, and AI

  03:16 Energy Landscapes and Loss Landscapes in High Dimensions

  04:07 Why Local Minima Don't Exist in High-Dimensional AI

  05:22 Gradient-Based vs. Gradient-Free Learning Methods

  08:21 AI in Mathematics and Drug Discovery: Opportunities and Challenges

  13:48 Scaling Laws and Data Efficiency in Language Models

  18:10 Properties of Data that Affect Scaling Laws

  22:04 Constructing Non-Redundant Data Sets for Better Learning

  24:32 Theory vs. Empirical Results in AI Research

  32:19 Fundamental Components of Deep Learning: Are They Changing?

  34:31 Future Paradigms in AI Beyond Current Models

  37:22 Teaching AI and Humans: Paradigm Shifts in Learning

  41:37 Consciousness, Self, and the Brain: Surya's Perspectives

  49:49 Neuroscience and AI: Understanding the Brain and Consciousness

  01:02:03 Understanding the Brain: Challenges and Opportunities

  01:09:21 Brain-Computer Interfaces and AI in Neuroscience

• ---

  Music

  • "Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Brendan O’Donoghue, research director at Google DeepMind, makes the case for text diffusion as a real alternative to autoregressive generation. He walks through how discrete diffusion works, why diffusion samples are far more diverse and what that unlocks for RL, where the Gemma diffusion model actually stands against frontier models, and why the whole training and serving stack being hyper-optimized for autoregression is the main thing holding the approach back. The conversation also covers hardware trends favoring flops over bandwidth, AGI timelines and real-world bottlenecks, and why he thinks RL is still underhyped.

Key topics
- Discrete diffusion for text vs autoregressive generation
- Why diffusion samples are more diverse, and what that unlocks for RL
- Where diffusion already wins: latency, on-device, robotics
- Why serving cost, not quality, is the real blocker
- RL as the most underhyped area in AI

Timeline
00:00 Introduction
00:50 What diffusion models are and how text diffusion works
04:40 Why Brendan bet on text diffusion in 2023
07:15 Diversity, creativity, and why it helps RL
11:00 The best diffusion LLM today and the gap to frontier models
14:25 Latency, serving cost, and why it needs more chips
17:14 Where diffusion already wins: on-device, robotics, battery
20:14 One model, two modes: diffusion for thinking, AR for answering
22:24 Samplers and the stuttering problem
26:27 Theory, BERT, and why now is a good time to work on this
31:48 Pipelines built for autoregression, and continuous diffusion
35:35 Hardware: flops vs bandwidth
39:49 AGI timelines and real-world bottlenecks
50:15 Is AI engineering or science?
54:14 Most overhyped and most underhyped ideas
58:35 RL on diffusion, value functions, and exploration
1:07:30 Go download the model and break it

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Nathan Lambert spent three years as post-training lead at Ai2, where he built the OLMo models, and he writes Interconnects, one of the most-read technical newsletters in AI. He left Ai2 in June and is now working on a new project. He’s also the author of the RLHF book. We talked a lot about open models, their capabilities, and why they are better than he expected. We get into what that means over the next two to five years, why he thinks recursive self-improvement is overblown, what the market for training environments actually looks like now, and why he expects Anthropic’s famously open internal culture to break after its IPO.

Key Topics

• Open vs closed models and who actually captures the value

• Anthropic and OpenAI as opposite cultures, and the talent concentration problem

• Boom vs bubble, and why token spend hasn’t produced 10x better products

• Continual learning, RSI skepticism, and what Nathan wants to work on next

• What the open ecosystem needs economically to survive

Timeline

00:00 Intro
00:27 Open vs closed models, and who actually captures the value
05:12 China, harnesses, and where the real training leverage sits
08:40 Sovereign compute and the national security case for building models
11:18 Uncensored open weights and the bioweapon question
14:29 Anthropic vs OpenAI, ideology and politics
19:35 The Mythos ban and the Fable 5 delays
24:30 The AGI narrative, the talent drain, and antitrust
28:12 Why researchers join Anthropic, and the open Slack culture
34:04 Nathan’s next 12 months: character training and big RL runs
37:55 Continual learning, RSI, and why Nathan is skeptical
43:19 Boom or bubble, tokens vs GPUs
45:12 Why all that token spend never produced 10x products
48:38 Job displacement and the small-business future
52:49 Robotics, world models, and why multimodal lags
57:44 What the open ecosystem should actually do
1:03:17 Why NVIDIA isn’t building a frontier model
1:07:34 The RLHF book, and whether RLHF still matters
1:11:06 GRPO vs PPO and on-policy distillation

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Daphne Koller wrote the book that many of us learned probabilistic graphical models from, founded Coursera, and now runs insitro, which is trying to make drug discovery a machine-learning problem.

We start with the bitter lesson. She agrees with most of it and then says where it stops working: biology doesn’t have enough data, structure is how people understand anything, and making a drug is a question about an intervention that hasn’t happened yet, not a pattern in data you already have.

Most of the episode is about why drug discovery is hard. Ninety percent of drugs that reach the clinic fail, and mostly not because the molecule was bad. The molecule usually does what it was designed to do. It just turns out the thing it was designed to do had nothing to do with the disease. Only 22% of diseases have any approved drug at all, and she calls that an upper bound on what we understand, not a lower bound.

She also gets into what agents are and aren’t good for in a wet lab, why cells don’t grow faster no matter how many GPUs you point at them, what it would take to have real foundation models for biology, and why almost all of biology is still out of distribution.

Plus GLP-1s and what human data keeps teaching us, whether AI can make the kind of leap that turned a bacterial immune system into CRISPR, and what she’d build if she were starting Coursera today.

Key Topics

• The impact of scaling and data in machine learning

• The importance of structure and causality in AI

• Challenges in drug discovery and biological understanding

• The role of foundation models in biology

• Ethical considerations in AI and biomedical research

Chapters

00:00 Introduction to Machine Learning and Drug Discovery

02:00 The Bitter Lesson and Its Implications

06:48 Challenges in Drug Design and Discovery

11:48 Ethical Considerations in Human Research

17:20 The Drug Discovery Pipeline Explained

29:30 Integrating AI in Experimental Design

35:38 The Role of Human Judgment in Drug Design

37:14 Future of Drug Design: Efficiency vs. Automation

39:37 Challenges in AI and Data Availability for Biology

41:08 Foundation Models: Potential and Limitations

43:39 Causality in Biological Data: Importance and Challenges

45:18 Creativity vs. Understanding in Drug Design

48:17 Balancing Investments in Data, Algorithms, and Experiments

50:07 The Value of Simulations in Drug Discovery

52:03 Mathematical Frameworks in Biology: Utility and Limitations

54:14 The Future of Drug Discovery: Optimism and Innovations

56:28 The Impact of Coursera on Education

01:00:33 The Role of Universities in Lifelong Learning

01:04:06 Connecting Dots: The Fun of Variety in Work

01:05:46 Optimism for the Future of Drug Discovery

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

In this episode, Joseph Suarez from PufferAI explains why he thinks RL never had an algorithm problem, but it had a code problem. Every part of the standard RL stack was running about a thousand times slower than it should have been, and once that got fixed, problems that used to take months started getting solved in seconds on one GPU. We talk about what makes a simulator good for RL, why most of their sims run on CPU, what he wants to do with scientific simulation, and why he open sources all of it instead of writing papers.

Key topics

• Types of RL and their applications

• Challenges in scaling reinforcement learning

• The role of simulators and hardware in RL

• RL in gaming: from chess to complex games like NetHack and RuneScape

• Future directions: scientific simulation and biological modeling

Chapters

00:00 - Introduction to RL and Puff AI

01:50 - Different settings for RL: Games, Robots, Finance

04:10 - RL in LM and other domains

07:00 - Challenges and solutions in RL scaling

09:55 - Building fast, efficient simulators

15:10 - RL for scientific research and simulation

19:57 - RL in complex games: NetHack, RuneScape, Dwarf Fortress

29:55 - Future of RL: Scientific discovery and beyond

Resources

Puff AI - Official Site - https://puffer.ai

NetHack - https://www.nethack.org/

RuneScape - https://www.runescape.com/

Dwarf Fortress - http://www.bay12games.com/dwarves/

OpenAI Gym - https://github.com/openai/gym

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

Florian Brand builds evals at Prime Intellect. The premise of the conversation is that writing a benchmark is the easy part now. Keeping the model from cheating it is the job, and it takes longer than the benchmark itself.

We get into why he thinks you can’t evaluate a model apart from the CLI it runs in, what happens to statistics when a single run costs five figures, and whether the feeling that a model just works can ever become a number.

He also has a few stories about agents finding their way around the scoring that are worth hearing cold.

Timeline

• 00:13 Intro

• 01:00 What evals are for

• 04:05 Agentic benchmarks

• 07:10 Kimi K2 and model diversity

• 08:23 Long-horizon coding tasks

• 10:29 Building a benchmark

• 12:15 MirrorCode

• 14:27 Rubrics and LLM judges

• 16:30 The cost of expert labelers

• 17:49 Long runs and variance

• 19:44 Evaluating the harness

• 24:29 Chinese labs building CLIs

• 30:00 More reward hacking

• 37:45 Tau-bench and economic tasks

• 39:43 Benchmaxxing and GLM 5.2

• 45:15 Statistics and cost

• 47:56 Frontier convergence

• 52:04 Misuse in open and closed models

• 55:35 Self-improvement

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About

The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

Most labs build language models by scraping the web and filtering afterward. Pierre-Carl Langlais runs it the other way around. At Pleias, the French-German lab he co-founded, the models are built from data he can actually account for, which in practice means open and public-domain sources plus a lot of synthetic data the lab generates itself. It sounds like a self-imposed handicap. It mostly isn’t. One of their models is a 600 million parameter system that runs live inside the Paris subway’s monitoring pipeline.

We cover the SYNTH pretraining dataset and why he thinks “ethical data” has to mean more than copyright-free. He explains why barely 2% of their Common Corpus appears in typical web crawls, and why that gap is really a preservation problem. From there, he gets blunt about benchmark maxing and whether GLM really earns its Opus-class reputation. He also argues that the quiet move by closed labs to hide reasoning traces is mostly about claiming ownership of model outputs. He’s skeptical of sovereign AI, and not shy about how Mistral drifted from frontier research toward French corporate consulting. We finish on NVIDIA’s persona datasets and the odd idea of training on the conditions that produced a text rather than the text itself.

Timeline

• (00:02) Welcome and introductions

• (00:49) Why synthetic data matters, and the SYNTH set

• (04:15) Three reasons to control your training data

• (07:18) What “ethical data” actually means

• (11:08) How Common Corpus got built, from Wikipedia to PDFs

• (16:35) Agentic harnesses and synthetic data

• (20:03) Evaluating data when you train on reasoning traces

• (25:27) General versus specialized pretraining

• (27:08) Benchmark maxing and the GLM question

• (31:51) Getting diversity in, and the NVIDIA personas

• (35:02) Hidden reasoning traces and the fight over model IP

• (38:17) Mid-training and the “It’s All Training” thesis

• (41:47) Can small models actually compete

• (45:01) Cybersecurity and Europe’s strategic gap

• (47:08) Do you need a big model to orchestrate the small ones

• (52:08) Sovereign AI and the limits of national champions

• (56:42) Scaling laws when you control the data

• (01:00:41) The NVIDIA persona datasets

• (01:04:52) What you actually do with synthetic personas

• (01:08:22) Closing thoughts

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About

The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

Dhruv Batra spent years leading Embodied AI at Meta, training virtual robots to navigate photorealistic 3D scans of real buildings with pure reinforcement learning. Then he left to co-found Yutori and build agents for a very different environment: the web browser.

In this episode, Dhruv explains why he sees these as the same problem. Web agents, in his framing, are robots that act in a browser (pixels in, actions out), and the web turns out to be just as messy an environment as the physical world.

Along the way, we cover his definition of intelligence as “navigation in idea space,” why robotics is lagging LLMs, the sim-to-real gap and why you can’t fake friction coefficients, the teleoperation counterexample to the “it’s a sensor problem” argument, and his provocative claim that under the current paradigm, we solved machine learning and didn’t even realize it. He also makes the case for why the scaling hypothesis isn’t falsifiable, why JEPA-style arguments deserve to be grappled with, how Yutori trains its Navigator models with RL on live websites, and what happens to the ad-supported web when agents, not eyeballs, do the browsing.

Timeline

00:01 — Intro
00:54 — What embodied AI actually means
06:47 — Intelligence as navigation in idea space
13:26 — Habitat: training robots with pure RL, no maps
20:04 — Why robotics is behind LLMs
28:24 — Sim-to-real: what you can and can’t fake
33:34 — “We solved ML and nobody noticed”
37:12 — Leaving Meta, founding Yutori
43:21 — Web agents: screenshots in, actions out
48:15 — Why the web won’t rebuild itself for agents
53:32 — Training Navigator: RL on live websites
1:01:04 — Who pays for the web when agents browse?
1:09:17 — What Yutori means, closing thoughts

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About

The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.

Ion Stoica has done what almost no academic ever does — repeatedly turned university research into billion-dollar companies. He co-founded Databricks (now valued at over $100 billion), Anyscale, Arena AI and Conviva, while his Berkeley lab produced the open source projects the entire AI industry runs on: Ray, vLLM, and SGLang.

In this episode, we ask him how it’s actually done. His answer is surprisingly unromantic: solve a problem people already care about, build an artifact good enough that they adopt it, and pay attention to the moment users start asking “who maintains this after the students graduate?” - that’s when a project becomes a company. He’s also insistent that the credit belongs to his students.

From there, the conversation goes deep into what he’s watching now: why the AI stack has become an order of magnitude more complex than the Hadoop/Spark era, why maximizing GPU utilization is “the name of the game” for any enterprise, and why coding agents will struggle with distributed systems long after they’ve mastered web apps. He shares a memorable reward-hacking story — a load balancer that maximized throughput by dropping requests — explains why the gap between open and closed models sits at about six months, and closes with his case for regulating AI by outcomes, not capabilities.

Timeline

• 00:00 — Introduction: welcoming Ion Stoica

• 01:21 — The playbook: how research projects become companies

• 05:22 — Will vLLM and SGLang stay open source?

• 07:47 — The real bottleneck in the AI stack: complexity, not just hardware

• 14:31 — Should algorithms follow infrastructure, or the other way around?

• 16:13 — Can AI coding tools write distributed systems and GPU kernels?

• 21:09 — Verifiers, harnesses, and the limits of outsourcing understanding

• 25:41 — Reward hacking: the load balancer that dropped requests

• 25:58 — How should enterprises consume GPUs? Utilization as the name of the game

• 30:23 — GPU scarcity: will the compute crunch ever end?

• 35:27 — Hyper-optimization and the risk of locking in today’s architectures

• 37:17 — Open vs. closed models: why every company wants to own the stack

• 40:35 — The six-month gap, and the rising cost of training frontier models

• 43:58 — Kimi, Qwen, and who’s incentivized to keep open models alive

• 45:39 — Regulation: outcomes, not capabilities

• 47:41 — Self-regulation, concentration of power, and auditing open models

• 48:32 — Wrap-up

Music

• “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.

About

The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.