Jean-Francois Puget is a Director and Distinguished Engineer at NVIDIA, where he leads the Kaggle Grandmasters team, and he’s ranked third on Kaggle’s all-time list. We caught him on the day NVIDIA announced Nemotron Ultra and its new agent skills repo. We talk about what skills actually are, why they beat MCP tools on context cost, and how NVIDIA built an evaluation pipeline to separate skills that help from skills that don’t.
From there we talk about the thing JFP cares about most: evaluation. He explains why most LLM benchmarks reward overfitting, how his team discovered O3 could pick the right files to fix SWE-bench issues without reading them, and why the only benchmarks he trusts are the ones where you commit before you see the score, which is exactly how Kaggle works. He predicts a “bloodbath” for the wave of competitors letting coding agents chase leaderboard scores with no notion of validation.
We also get into what coding agents are actually good for (”a mix of a genius and a dumb person”), the multi-agent system at NVIDIA that built a working PyTorch clone that runs 10x slower than the real thing, his unfiltered take on frontier lab PR and the Mythos release, whether AI is a bubble, and the story of how his team won ARC-AGI with a 4-billion-parameter model at 20 cents a task, including jumping from third to first in the final hours of a seven-month competition.
Timeline
00:00 — Intro
01:05 — NVIDIA’s announcements: Nemotron Ultra and the agent skills repo
07:21 — Skills vs MCP tools, and progressive disclosure
10:24 — Agents that write their own skills: a new form of learning
13:33 — When overfitting is fine (and when it isn’t)
15:47 — Why most LLM benchmarks reward overfitting
17:06 — The SWE-bench contamination story: O3 picks files without reading them
19:45 — How LLMs changed Kaggle, and the coming “bloodbath”
25:40 — What makes a good data scientist: evaluation and one-bit experiments
28:56 — Running Codex at scale: the top token consumers at NVIDIA
29:37 — Did coding agents kill AutoML?
30:16 — Genius and dumb at once: the limits of coding agents
35:21 — Humans in the loop, sandboxing, and the teenage hacker who never wrote code
37:42 — Mythos, frontier lab PR, and open source
40:08 — Why NVIDIA builds open models, and where it’s already frontier
43:48 — World models, robots, and the coffee test
49:20 — Why agents still can’t play Dota
50:24 — Is AI a bubble?
53:14 — Winning ARC-AGI with a 4B model at 20 cents a task
57:39 — Kaggle is a legal drugMusic:
Music: “Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
About: The Information Bottleneck is hosted by Ravid Shwartz-Ziv and Allen Roush, featuring in-depth conversations with leading AI researchers about the ideas shaping the future of machine learning.
