Riccardo Zamboni - Pure Exploration in POMDP: limits and possible solutions

Описание: UoE RL Reading Group | 22 August 2024

Speaker: Riccardo Zamboni (Politecnico di Milano)

Title: Pure Exploration in POMDP: limits and possible solutions

Abstract: The problem of pure exploration in MDPs has been cast as maximizing the entropy over the state distribution induced by the agent’s policy, an objective that has been extensively studied. However, little attention has been dedicated to state entropy maximization under partial observability, despite the latter being ubiquitous in applications, e.g., finance and robotics, in which the agent only receives noisy observations of the true state governing the system’s dynamics. How can we address state entropy maximization in those domains? In this talk, we first provide lower and upper bounds to the approximation of the true state entropy that only depend on some properties of the observation function. Then, we study the simple approach of maximizing the entropy over observations in place of true latent states and we show how knowledge of the latter can be exploited to compute a principled regularization of the observation entropy to improve performance. Finally, we briefly provide some insights on possible ways to pass over this approach and take into account beliefs over the latent states.

Link: https://arxiv.org/pdf/2406.12795

Bio: Riccardo is a PhD Student under the supervision of M. Restelli at Politecnico di Milano. His research focuses on developing principled algorithms to pass over current limitations in Multi-Agent RL.