Online Policy Optimization for Robust MDP
Abstract
Reinforcement learning (RL) has exceeded human performance in many synthetic settings such as video games and Go.
However, real-world deployment of endto-end RL models is less common, as RL models can be very sensitive to slight perturbation of the environment.
The robust Markov decision process (MDP) framework—in which the transition probabilities belong to an uncertainty set around a nominal model—provides one way to develop robust models.
While previous analysis shows RL algorithms are effective assuming access to a generative model, it remains unclear whether RL can be efﬁcient under a more realistic online setting, which requires a careful balance between exploration and exploitation.
In this work, we consider online robust MDP by interacting with an unknown nominal system.
We propose a robust optimistic policy optimization algorithm that is provably efﬁcient.
To address the additional uncertainty caused by an adversarial environment, our model features a new optimistic update rule derived via Fenchel conjugates.
Our analysis establishes the ﬁrst regret bound for online robust MDPs.
Introduction
The rapid progress of reinforcement learning (RL) algorithms enables trained agents to navigate around complicated environments and solve complex tasks.
The standard reinforcement learning methods, however, may fail catastrophically in another environment, even if the two environments only differ slightly in dynamics [11, 22, 7, 31, 25].
In practical applications, such mismatch of environment dynamics are common and can be caused by a number of reasons, e g., model deviation due to incomplete data, unexpected perturbation and possible adversarial attacks.
To model the potential mismatch between system dynamics, the framework of robust MDP is introduced to account for the uncertainty of the parameters of the MDP [27, 35, 21, 12].
Under this framework, the dynamic of an MDP is no longer ﬁxed but can come from some uncertainty set, such as the rectangular uncertainty set, centered around a nominal transition kernel.
The agent sequentially interacts with the nominal transition kernel to learn a policy, which is then evaluated on the worst possible transition from the uncertainty set.
Therefore, the objective is to ﬁnd the worst-case best-performing policy.
If a generative model (also known as a simulator) of the environment or a suitable ofﬂine dataset is available, one could obtain a ✏ -optimal robust policy with ˜ O
( ✏  2 ) samples under a rectangular uncertainty set [24, 23, 34, 18].
Yet the presence of a generative model is stringent to fulﬁll for real applications.
In a more practical online setting, the agent sequentially interacts with the environment and tackles the exploration-exploitation challenge as it balances between exploring the state space and exploiting the high-reward actions.
in the online setting, which is captured by the regret, is more challenging to achieve than algorithm convergence.
In the robust MDP setting, previous sample complexity results cannot directly imply a sublinear regret in general Dann et al.
[8] and so far no asymptotic result is available.
A more detailed review of the related works are deferred to the
Appendix.
In this paper, we propose the ﬁrst policy optimization algorithm for robust MDP under a rectangular uncertainty set.
One of the challenges for deriving a regret guarantee for robust MDP stems from its adversarial nature.
As the transition dynamic can be picked adversarially from a predeﬁned set, the optimal policy is in general randomized [36].
This is in contrast with conventional MDPs, where there always exists a deterministic optimal policy, which can be found with value-based methods and a greedy policy (e.g. UCB-VI algorithms).
Bearing this observation, we resort to policy optimization (PO)-based methods, which directly optimize a stochastic policy in an incremental way.
With a stochastic policy, our algorithm explores robust MDPs in an optimistic manner.
To achieve this robustly, we propose a carefully designed bonus function via the dual conjugate of the robust bellman equation.
This quantiﬁes both the uncertainty stemming from the limited historical data and the uncertainty of the MDP dynamic.
In the episodic setting of robust MDPs, we show that our algorithm attains sublinear regret O ( p K ) for both ( s, a ) and s -rectangular uncertainty set, where K is the number of episodes.
In the case where the uncertainty set contains only the nominal transition model, our results recover the previous regret upper bound of non-robust policy optimization [30].
Our result achieves the ﬁrst provably efﬁcient regret bound in the online robust MDP problem.
We further validated our algorithm with experiments.
Problem formulation
In this section, we describe the formal setup of robust MDP.
We start with deﬁning some notations.
Robust Markov decision process We consider an episodic ﬁnite horizon robust MDP, which can denoted by a tuple M = h S , A , H, {P} Hh =1 , { r } Hh =1 i .
Here S is the state space, A is the action space, { r } Hh =1 is the time-dependent reward function, and H is the length of each episode.
Instead of a ﬁxed step of time-dependent uncertainty kernels, the transitions of the robust MDP is governed by kernels that are within a time-dependent uncertainty set {P} Hh =1 , i .
e . , time-dependent transition P h 2 P h ✓  S at time h .
We consider the case where the rewards are stochastic.
This is, on state-action ( s, a ) at time h , the immediate reward is R h ( s, a ) 2 [0 , 1] , which is drawn i.i.d from a distribution with expectation r h ( s, a ) .
With the described setup of robust MDPs, we now deﬁne the policy and its associated value.
Policy and robust value function A time-dependent policy ⇡ is deﬁned as ⇡ = { ⇡ h } Hh =1 , where each ⇡ h is a function from S to the probability simplex over actions,  ( A ) .
If the transition kernel is ﬁxed to be P , the performance of a policy ⇡ starting from state s at time h can be measured by its value function, which is deﬁned as V ⇡ ,P h (
[Equation]
i .
In robust
MDP, the robust value function instead measures the performance of ⇡ under the worst possible choice of transition P within the uncertainty set.
Speciﬁcally, the value and the Q-value function of a policy given the state action pair ( s, a ) at step h are deﬁned as
[Equation]
The optimal value function is deﬁned to be the best possible value attained by a policy
V
⇤ h
( s ) =
max [Equation]
( s ) .
The optimal policy is then deﬁned to be the policy that attains the optimal value.
Robust Bellman equation Similar to non-robust MDP, robust MDP has the following robust bellman equation, which characterizes a relation to the robust value function.
Q ⇡ h ( s, a ) =
r ( s, a )+  P h ( V ⇡ h +1 )
( s, a ) ,
V ⇡ h ( s ) =
h Q ⇡ h ( s, · ) , ⇡ h ( · , s ) i , where  P h
( V ⇡ h +1 )
( s, a ) =
min P h 2 P h
P h ( · | s, a ) V ⇡ h +1 ,
P h ( · | s, a ) V = P s 0 2 S P h ( s 0 | s, a ) V ( s 0 ) .
Without additional assumptions on the uncertainty set, the optimal policy and value of the robust
MDP are in general NP-hard to solve [36].
Thus, to limit the level of perturbations, we assume that the transition kernels is close to the nominal transition measured via ` 1 distance.
We consider two cases.
Deﬁnition 2.1 ( ( s, a ) -rectangular uncertainty set Iyengar [12], Wiesemann et al.
For all time step h and with a given state-action pair ( s, a ) , the ( s, a ) -rectangular uncertainty set P h ( s, a ) is deﬁned as P h ( s, a ) =  k [Equation]
[Equation] s, a ) k [Equation]  ⇢ , P
[Equation] s, a ) 2  ( S ) , where P oh is the nominal transition kernel at h , P oh ( · | s, a ) > 0 , 8 ( s, a ) 2 S ⇥ A , ⇢ is the level of uncertainty.
One way to relax the ( s, a ) -rectangular assumption is to instead let the uncertain transition kernels within the set take value independent for each s only.
This characterization is then more general and its solution gives a stronger robustness guarantee.
Deﬁnition 2.2 ( s -rectangular uncertainty set Wiesemann et al.
For all time step h and with a given state s , the s -rectangular uncertainty set P h ( s ) is deﬁned as P h ( s )
=
P
[Equation]
, P h ( · | s, · ) 2  ( S ) A , where P oh is the nominal tran- sition kernel at h , P oh ( · | s, a ) > 0 , 8 ( s, a ) 2 S ⇥ A , ⇢ is the level of uncertainty.
Different from the ( s, a ) -rectangular assumption, which guarantees the existence of a deterministic optimal policy, the optimal policy under s -rectangular set may need to be randomized [36].
We also remark that the requirement of P oh ( · | s, a ) > 0 is mostly for technical convenience.
Equipped with the characterization of the uncertainty set, we now describe the deﬁnition of regret under the robust MDP.
Learning protocols and regret We consider a learning agent repeatedly interacts with the environment deﬁned by the nominal transition model in an episodic manner, over K episodes.
We remark that if the agent is asked to interact with a potentially adversarially chosen transition, the learning problem is NP-hard [10].
We assume the agents always start from a ﬁxed initial state s .
The performance of the learning agent is measured by the cumulative regret incurred over the K episodes, which is deﬁned to be the cumulative difference between the robust value of ⇡ k and the robust value of the optimal policy.
That is,
[Equation]
initial [Equation]
Algorithm
Our algorithm performs policy optimization with empirical estimates and encourages exploration by adding a bonus to less explored states.
However, we need to propose a new efﬁciently computable bonus that is robust to adversarial transitions.
We achieve this via solving a sub-optimization problem derived from Fenchel conjugate.
We present Robust Optimistic Policy Optimization (ROPO) and elaborate on its design components.
To start, as our algorithm has no access to the actual reward and transition function, we use the following empirical estimator of the transition and reward: [Equation]
where N kh ( s, a ) = max nP k
[Equation]
Robust Policy Evaluation step In each episode, the algorithm estimates Q -values with an optimistic variant of the bellman equation.
Speciﬁcally, to encourage exploration in the robust MDP, we add a bonus term b kh ( s, a ) , which compensates for the lack of knowledge of the actual reward and transition model as well as the uncertainly set, with order b kh
[Equation]
s, [Equation] = O
[Equation]
Intuitively, the bonus term b kh desires to characterize the optimism required for efﬁcient exploration for both the estimation errors of P and the robustness of P .
It is hard to control the two quantities in their primal form because of the coupling between them.
We propose the following procedure to address the problem.
Note that the key difference between our algorithm and standard policy optimization is that  ˆ P h
( ˆ V ⇡ h +1 )
( s ) requires solving an inner minimization.
Through relaxing the constraints with Lagrangian multiplier and Fenchel conjugates, under ( s, a ) -rectangular set, the inner minimization problem can be reduced to a one-dimensional unconstrained convex optimization problem on R (Lemma 4).
[Equation]
The optimum of Equation (2) is then computed efﬁciently with bisection or sub-gradient methods.
Similarly, in the case of s -rectangular set, the inner minimization problem is equivalent to a Adimensional convex optimization problem, which can be computed efﬁciently in ˜ O
( A ) iterations by methods like gradient descent.
In addition to reducing computational complexity, the dual form decouples the uncertainty in estimation error and in robustness, as ⇢ and ˆ P
oh are not in different terms.
The exact form of b kh is presented in the Equation (4) and (5).
In the case of s -rectangular set, the inner minimization problem is similarly equivalent to the following A -dimensional convex optimization problem.
[Equation]
[Equation] Policy Improvement Step Using the optimistic Q -value obtained from policy evaluation, the algorithm improves the policy with a KL regularized online mirror descent step,
[Equation]
where  is the learning rate.
In the non-robust case, this improvement step is also shown to be theoretically efﬁcient [30, 37].
Many empirically successful policy optimization algorithms, such as
PPO [29] and TRPO [28], also take a similar approach to KL regularization for non-robust policy improvement.
Main results
We are now ready to analyze the theoretical results of our algorithm under the uncertainly set.
Theorem 1 (Regret under ( s, a ) -rectangular uncertainty set) .
With learning rate  = q 2log A H 2 K and bonus term b kh as (4), with probability at least 1   , the regret incurred by Algorithm 1 over K episodes is bounded by O
[Equation]
Remark 4.1.
When ⇢ = 0 , the problem reduces to non-robust reinforcement learning.
In such case our regret upper bound is ˜ O
[Equation]
⌘ , which is in the same order of policy optimization algorithms for the non-robust case Shani et al.
Beyond the ( s, a ) -rectangular uncertainty set, we also extends to s -rectangular uncertainty set (Deﬁnition 2.2).
Input: learning rate  , bonus function b kh .
for k = 1 , . . .
K do Collect a trajectory of samples by executing ⇡ k . # Robust Policy Evaluation for h = H, . . . , 1 do for 8 ( s, a ) 2 S ⇥
A do
Solve  ˆ P h
( ˆ V ⇡ h +1 )
( s, a ) according to Equation (2) for ( s, a ) -rectangular set or Equation (3) for s -rectangular set.
ˆ
[Equation]
min n
[Equation]
o . end for for 8 s 2 S do ˆ V kh ( s ) =
D
[Equation]
Q kh ( s, · ) , ⇡ kh ( · | s )
E .
end for end for # Policy Improvement for 8 h, s, a 2 [ H ] ⇥ S ⇥ A do ⇡
k +1 h ( a | s ) =
⇡ kh exp(
[Equation]
Q
[Equation]
,a ))
P
[Equation]
end for
Update empirical estimate ˆ r , ˆ P with Equation (1).
end for
Theorem 2 (Regret under s -rectangular uncertainty set) .
With learning rate  = q 2log A H 2 K and bonus term b kh as (5), with probability at least 1   , the regret of Algorithm 1 is bounded by
O ⇣
SA 2 Hp
K log( SA 2 H 2 K 3 / 2 (1 + ⇢ ) /  ) ⌘ .
Remark 4.2.
When ⇢ = 0 , the problem reduces to non-robust reinforcement learning.
In such case our regret upper bound is ˜
O ⇣
SA
[Equation]
H [Equation] p K ⌘ .
Our result is the ﬁrst theoretical result for learning a robust policy under s -rectangular uncertainty set, as previous results only learn the robust value function [38].
We defer the proof of these theorems, along with the experiments results of the proposed algorithm to the Appendix.
Conclusion
In this paper, we studied the problem of regret minimization in robust MDP with a rectangular uncertainty set.
We proposed a robust variant of optimistic policy optimization, which achieves sublinear regret in all uncertainty sets considered.
Our algorithm delicately balances the exploration-exploitation trade-off through a carefully designed bonus term, which quantiﬁes not only the uncertainty due to the limited observations but also the uncertainty of robust MDPs.
Our results are the ﬁrst regret upper bounds in robust MDPs as well as the ﬁrst non-asymptotic results in robust MDPs without access to a generative model. Acknowledgement
Jing Dong and Baoxiang Wang are partially supported by National Natural Science Foundation of China (62106213, 72150002) and
Shenzhen Science and Technology Program (RCBS20210609104356063, JCYJ20210324120011032).
Jingzhao Zhang is supported by Tsinghua
University Initiative Scientiﬁc Research Program.