Jadwal Sholat

Memuat jadwal sholatโ€ฆ

Pendidikan editorial

Open AccessOA2023

Nonstationary Reinforcement Learning: The Blessing of (More) Optimism

This paper develops algorithms for undiscounted reinforcement learning in nonstationary Markov decision processes with model uncertainty, introducing a confidence-widening technique to incorporate additional optimism and achieve low dynamic regret bounds.
Wang Chi Cheung; D. Simchi-Levi; Rui-Hao Zhuยท Management Sciencesยท 2023ยท DOI 10.1287/mnsc.2023.4704

The core problem

The paper addresses undiscounted reinforcement learning (RL) in Markov decision processes (MDPs) where both reward and state transition distributions evolve over time, subject to variation budgets. Motivated by operations research applications such as inventory control and real-time bidding, the authors consider model uncertainty and temporal drifts. The key challenge is that under nonstationarity, historical data may falsely indicate that certain state transitions rarely occur, undermining the conventional optimism in the face of uncertainty principle. The goal is to design algorithms that achieve low dynamic regret despite these drifts.

Innovation

The authors propose two algorithms: Sliding Window Upper Confidence Bound for Reinforcement Learning with Confidence-Widening (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BO-RL). SWUCRL2-CW uses a sliding window to handle nonstationarity and introduces a confidence-widening technique that adds extra optimism to counteract misleading historical data. This technique ensures that the algorithm remains optimistic about rarely visited state-action pairs, even if recent data suggest low transition probabilities. BO-RL adaptively tunes the window size and confidence-widening parameter without knowledge of the variation budgets, achieving the same dynamic regret bound in a parameter-free manner. The dynamic regret is measured against a sequence of benchmark policies that can adapt to the changing environment. The variation budgets are quantified using suitable metrics on the total variations of reward and transition distributions.
Introduction
The paper addresses undiscounted reinforcement learning (RL) in Markov decision processes (MDPs) where both reward and state transition distributions evolve over time, subject to variation budgets. Motivated by operations research applications such as inventory control and real-time bidding, the authors consider model uncertainty and temporal drifts. The key challenge is that under nonstationarity, historical data may falsely indicate that certain state transitions rarely occur, undermining the conventional optimism in the face of uncertainty principle. The goal is to design algorithms that achieve low dynamic regret despite these drifts.

Why it matters

The key insight is that under nonstationarity, historical data can be misleading, causing conventional optimism to fail. The confidence-widening technique addresses this by incorporating additional optimism, which is shown to be beneficial. The parameter-free BO-RL algorithm makes the approach practical by removing the need to know variation budgets. The application to inventory control demonstrates the versatility of the framework. The paper was accepted by J. George Shanthikumar for the Data Science department, and funded by MIT Data Science Laboratory, MITโ€“IBM partnership in AI, and Singapore MOE Tier 2 Grant MOE-T2EP20121-0012. The supplemental material is available at the DOI provided.

Who should read this

Pembaca Research Digest dan praktisi rumpun ini.

Opening member contentโ€ฆ