Adam Jelley and Eloi Alonso - Diffusion for World Modeling: Visual Details Matter in Atari (DIAMOND)

Описание: UoE RL Reading Group | 14 November 2024

Speaker: Adam Jelley and Eloi Alonso (University of Edinburgh, University of Geneva)

Title: Diffusion for World Modeling: Visual Details Matter in Atari (DIAMOND)

Abstract: World models constitute a promising approach for training reinforcement learning agents in a safe and sample-efficient manner. Recent world models predominantly operate on sequences of discrete latent variables to model environment dynamics. However, this compression into a compact discrete representation may ignore visual details that are important for reinforcement learning. Concurrently, diffusion models have become a dominant approach for image generation, challenging well-established methods modeling discrete latents. Motivated by this paradigm shift, we introduce DIAMOND (DIffusion As a Model Of eNvironment Dreams), a reinforcement learning agent trained in a diffusion world model. We analyze the key design choices that are required to make diffusion suitable for world modeling, and demonstrate how improved visual details can lead to improved agent performance. DIAMOND achieves a mean human normalized score of 1.46 on the competitive Atari 100k benchmark; a new best for agents trained entirely within a world model. We further demonstrate that DIAMOND's diffusion world model can stand alone as an interactive neural game engine by training on static Counter-Strike: Global Offensive gameplay.

Link(s): https://arxiv.org/abs/2405.12399

Bio: Adam is a PhD student at the University of Edinburgh, supervised by Professor Amos Storkey in the School of Informatics and Sam Devlin at Microsoft Research Cambridge. His research is focused on developing efficient reinforcement learning approaches, via the use of world models, offline data and human feedback. Eloi is a PhD student at the University of Geneva, supervised by Professor François Fleuret. His research is focused on world modeling via generative models, and reinforcement learning.