In this episode, Joseph Suarez from PufferAI explains why he thinks RL never had an algorithm problem, but it had a code problem. Every part of the standard RL stack was running about a thousand times slower than it should have been, and once that got fixed, problems that used to take months started getting solved in seconds on one GPU. We talk about what makes a simulator good for RL, why most of their sims run on CPU, what he wants to do with scientific simulation, and why he open sources all of it instead of writing papers.
Key topics
Types of RL and their applications
Challenges in scaling reinforcement learning
The role of simulators and hardware in RL
RL in gaming: from chess to complex games like NetHack and RuneScape
Future directions: scientific simulation and biological modeling
Chapters
00:00 - Introduction to RL and Puff AI
01:50 - Different settings for RL: Games, Robots, Finance
04:10 - RL in LM and other domains
07:00 - Challenges and solutions in RL scaling
09:55 - Building fast, efficient simulators
15:10 - RL for scientific research and simulation
19:57 - RL in complex games: NetHack, RuneScape, Dwarf Fortress
29:55 - Future of RL: Scientific discovery and beyond
Resources
Puff AI - Official Site - https://puffer.ai
NetHack - https://www.nethack.org/
RuneScape - https://www.runescape.com/
Dwarf Fortress - http://www.bay12games.com/dwarves/
OpenAI Gym - https://github.com/openai/gym
Music
“Kid Kodi” - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0.
