Challenging Common Assumptions in Convex Reinforcement Learning
Abstract
The classic Reinforcement Learning (RL) formulation concerns the maximization of a scalar reward function.
More recently, convex RL has been introduced to extend the RL formulation to all the objectives that are convex functions of the state distribution induced by a policy.
Notably, convex RL covers several relevant applications that do not fall into the scalar formulation, including imitation learning, risk-averse RL, and pure exploration.
In classic RL, it is common to optimize an inﬁnite trials objective, which accounts for the state distribution instead of the empirical state visitation frequencies, even though the actual number of trajectories is always ﬁnite in practice.
This is theoretically sound since the inﬁnite trials and ﬁnite trials objectives are equivalent and thus lead to the same optimal policy.
In this paper, we show that this hidden assumption does not hold in convex RL.
In particular, we prove that erroneously optimizing the inﬁnite trials objective in place of the actual ﬁnite trials one, as it is usually done, can lead to a signiﬁcant approximation error.
Since the ﬁnite trials setting is the default in both simulated and real-world RL, we believe shedding light on this issue will lead to better approaches and methodologies for convex RL, impacting relevant research areas such as imitation learning, risk-averse RL, and pure exploration among others.
Introduction
Standard Reinforcement Learning (RL) [51] is concerned with sequential decision-making problems in which the utility can be expressed through a linear combination of scalar reward terms.
The coefﬁcients of this linear combination are given by the state visitation distribution induced by the agent’s policy.
Thus, the objective function can be equivalently written as the inner product between the mentioned state distribution and a reward vector.
However, not all the relevant objectives can be encoded through this linear representation [2].
Several works have thus extended the standard RL formulation to address non-linear objectives of practical interest.
These include imitation learning [31, 43], or the problem of ﬁnding a policy that minimizes the distance between the induced state distribution and the state distribution provided by experts’ interactions [1, 30, 33, 34, 23, 17], risk-averse RL [21], in which the objective is sensitive to the tail behavior of the agent’s policy [53, 44, 52, 16, 15, 8, 62], pure exploration [28], where the goal is to ﬁnd a policy that maximizes the entropy of the induced state distribution [54, 34, 42, 41, 60, 25, 37, 48, 56, 40, 39], diverse skills discovery [24, 20, 26, 49, 11, 36, 29, 59], constrained RL [4, 3, 9, 38, 46, 57, 6], and others.
All this large body of work has been recently uniﬁed into a unique framework, called convex RL [61, 58, 22], which admits as an objective any convex function of the state distribution induced by the agent’s policy.
The convex RL problem has been showed to be largely tractable either computationally, as it admits a dual formulation akin to standard RL [45], or statistically, as principled algorithms achieving sub-linear regret rates that are slightly worse than standard RL have been developed [61, 58].
Finite Trials
[Equation]
assumption which is rarely met in practice.
Indeed, the objective is written as a function of the state distribution, which is an expectation over the empirical state distributions that are actually obtained by running the policy in a given episode.
In practice, we always run our policy for a ﬁnite number of episodes (or trials ), which in general prevents the empirical state distribution from converging to its expectation.
This has never been a problem in standard RL: due to the scalar objective, optimizing the policy over inﬁnite trials or ﬁnite trials is equivalent, as it leads to the same optimal policy.
Crucially, in this paper, we show that this property does not hold for the convex RL formulation: a policy optimized over inﬁnite trials can be signiﬁcantly sub-optimal when deployed over ﬁnite trials (Figure 1).
In light of this observation, we reformulate the convex RL problem from a ﬁnite trials perspective, developing insights that can be used to partially rethink the way convex objectives have been previously addressed in RL, with potential ripple effects to research areas of signiﬁcant interest, such as imitation learning, risk-averse RL, pure exploration, and others.
In this paper, we formalize the notion of a ﬁnite trials RL problem, in which the objective is a function of the empirical state distribution induced by the agent’s policy over n trials rather than its expectation over inﬁnite trials.
As an illustrative example, consider a ﬁnancial application, in which we aim to optimize a trading strategy.
In the real world, we can only deploy the strategy over a single trial.
Thus, we are only interested in the performance of the strategy in the real-world realization, rather than the performance of the strategy when averaging different realizations.
Similar considerations apply to other relevant real-world applications, such as autonomous driving or treatment optimization in a healthcare domain.
Following this intuition, we ﬁrst deﬁne the (linear) ﬁnite trial RL formulation (Section 3), for which it is trivial to prove the equivalence with standard RL.
In Section 4, we provide the ﬁnite trial convex RL formulation, for which we prove an upper bound on the approximation error made by optimizing the inﬁnite trials as a proxy of the ﬁnite trials objective.
In light of this ﬁnding, we challenge the hidden assumption that (1) convex RL can be equivalently addressed with an inﬁnite trials formulation , even if the setting is ﬁnite-trial.
We corroborate this result with an additional numerical analysis showing that the approximation bound is non-vacuous for relevant applications (Section 6).
Finally, in Section 5 we include an in-depth analysis of the single trial convex RL, which suggests that other common assumptions in the convex RL literature, i.e., that (2) the problem is always computationally tractable and that (3) stationary policies are in general sufﬁcient , should be reconsidered as well.
The proofs of the reported results can be found in the Appendix.
Preliminaries
In this section, we report the notation and background useful to understand the paper.
We denote with [ H ] a set of integers { 1 , . . .
, H } , and with a lower case letter a a scalar or a vector, according to the context.
For two vectors a, b ∈ R d , we denote with a · b = (cid:80) di =
a i b i the inner product between a, b .
2.1 Probabilities and Percentiles
Let X denote a measurable space, we will denote with ∆( X ) the probability simplex over X , and with p ∈ ∆( X ) a probability measure over X .
For two probability measures p, q over X , we deﬁne their (cid:96) p -distance as (cid:107) p [Equation]
p := (cid:0)
[Equation]
x
[Equation] (cid:12)(cid:12) p [Equation] ) (cid:12)(cid:12) p (cid:1) 1 / p , and their Kullback-Leibler (KL) divergence as KL ( p || q ) :=
(cid:80) x ∈X p ( x ) log (cid:0) p ( x ) /q
( x ) (cid:1) .
Let X be a random variable distributed according to p , having a cumulative density function F X ( x ) =
Pr ( X ≤ x ) .
We denote with E [ X ] its expected value, and its α -percentile is denoted as VaR α ( X )
= inf (cid:8) x | F X ( x ) ≥ α (cid:9) =
F − 1 X ( α ) , where α ∈ (0 , 1) is a conﬁdence level, and VaR α stands for Value at Risk (VaR) at level α .
We denote the expected value of X within its α -percentile as CVaR α ( X ) =
E (cid:2)
X | X ≤ VaR α ( X ) (cid:3) , where
CVaR α stands for Conditional Value at Risk (CVaR) at level α .
2.2 Markov Decision Processes
A tabular Markov Decision Process (MDP) [45] is deﬁned as M :=
( S , A , P, H, µ, r ) , where
S is a state space of size S , A is an action space of size A , P is a Markovian transition model
P : S ×A → ∆( S ) , such that P ( s (cid:48) | s, a ) denotes the conditional probability of the next state s (cid:48) given the current state s and action a , H is the episode horizon, µ is an initial state distribution µ : ∆(
S ) , and r is a scalar reward function r : S → R , such that r ( s ) is the reward collected in the state s .
In the typical interaction episode, an agent ﬁrst observes the initial state s 0 ∼ µ of the MDP.
Then, the agent select an action a 0 , so that the MDP transitions to the next state s 1 ∼ P ( ·| s 0 , a 0 ) , and the agent collects the reward r ( s 1 ) .
Having observed s 1 , the agent then selects an action a 1 triggering a subsequent transition to s 2 ∼ P ( ·| s 1 , a 1 ) .
This process carries on repeatedly until the episode ends.
A policy π deﬁnes the behavior of an agent interacting with an MDP, i.e., the strategy for which an action is selected at any step of the episode.
It consists of a sequence of decision rules ( π t ) ∞ t =
that maps the current trajectory 2 h t
= ( s i , a i ) t − 1 i =
∈ H t with a distribution over actions π t : H t → ∆( A ) , where
H t denotes the set of trajectories of length t .
A non-stationary policy is a sequence of decision rules π t : S → ∆( A ) .
A stationary (Markovian) policy is a time-consistent decision rule π : S → ∆( A ) , such that π ( a | s ) denotes the conditional probability of taking action a in state s .
A trajectory h , obtained from an interaction episode, induces an empirical distribution d over the states of the MDP M ,
such that d ( s ) = 1 | h | (cid:80) s t ∈
h 1 ( s t = s ) .
We denote with p π the probability of drawing d by following the policy π .
For n ∈ N , we denote with d n the empirical distribution d n ( s ) =
1 n (cid:80) ni
=1 d i ( s ) , and with p πn the probability of drawing d n by following the policy π for n episodes.
Finally, we call the expectation d π = E d ∼ p
π
[ d ] the state distribution induced by π .
Reinforcement Learning in Finite Trials
In the standard RL formulation [51], an agent aims to learn an optimal policy by interacting with an unknown MDP M .
An optimal policy is a decision strategy that maximizes the expected sum of rewards collected during an episode.
Especially, we can represent the value of a policy π through the value function V πh ( s ) :
= E π (cid:2) (cid:80)
Hh (cid:48)
=
h r ( s h (cid:48) ) (cid:12)(cid:12) s h
= s (cid:3) .
The value function allows us to write the RL objective as max π ∈ Π E s 1 ∼ µ [ V π 1 ( s 1 )] , where Π is the set of all the stationary policies.
Equivalently, we can rewrite the RL objective into its dual formulation [45], i.e.,
[Equation]
where we denote with r ∈ R S a reward vector, and with d π the state distribution induced by π .
We call the problem (1) the inﬁnite trials RL formulation.
Indeed, the objective J ∞ ( π ) considers the sum of the rewards collected during an episode, i.e., r · d π , that we can achieve on the average of an inﬁnite number of episodes drawn with π .
This is due to the state distribution d π being an expectation of empirical distributions d π = E
d ∼ p π
[ d ] .
However, in practice, we can never draw inﬁnitely many episodes following a policy π .
Instead, we draw a small batch of episodes d n ∼ p πn .
Thus, we can instead conceive a ﬁnite trials RL formulation that is closer to what is optimized in practice.
[Table]
One could then wonder whether optimizing the ﬁnite trials objective (2) leads to results that differ from the inﬁnite trials one (1).
To this point [Table] it is straightforward to see that the two objective functions are actually equivalent
[Equation]
since r is a constant vector and the expectation is a linear operator.
It follows that the inﬁnite trials and the ﬁnite trials RL formulations admit the same optimal policies.
Hence, one can enjoy both the computational tractability of the inﬁnite trials formulation and, at the same time, optimize the objective function that is employed in practice.
In the next section, we will show that a similar result does not hold true for the convex RL formulation.
Convex Reinforcement Learning in Finite Trials
Even though the RL formulation covers a wide range of sequential decision-making problems, several relevant applications cannot be expressed by means of the inner product between a linear reward vector r and a state distribution d π [2, 50].
These include imitation learning, pure exploration, constrained problems, and risk-sensitive objectives, among others.
Recently, a convex RL formulation [61, 58, 22] has been proposed to unify these applications in a unique general framework, which is
[Equation]
where F : ∆( S ) → R is a function 3 of the state distribution d π .
In Table 1, we recap some of the most relevant problems that fall under the convex RL formulation, along with their speciﬁc F function.
As it happens for linear RL, in any practical simulated or real-world scenario, we can only draw ﬁnite number of episodes with a policy π .
From these episodes, we obtain an empirical distribution d n ∼ p πn rather than the actual state distribution d π , where n is the number of episodes.
This can cause a mismatch from the objective that is typically considered in convex RL [61, 58], and what can be optimized in practice.
To overcome this mismatch, we deﬁne a ﬁnite trials formulation of the convex RL objective, as we did in the previous section for the linear RL formulation.
[Equation]
However, in the following we will consider functions F that are either convex, concave or even non-convex
In general, problem (3) takes the form of a max problem for concave F or a min problem for convex F .
Comparing objectives (3) and (4), one can notice that both of them include an expectation over the episodes, being d π = E d ∼ p π [ d ] .
Especially, we can write
[Equation]
through the Jensen’s inequality.
As a consequence, optimizing the inﬁnite trials objective ζ ∞ ( π ) does not necessarily lead to an optimal behavior for the ﬁnite trials objective ζ n ( π ) .
From a mathematical perspective, this is due to the fact that the empirical distributions d n induced by the policy π are averaged by the expectation d π before computing the F function into objective (3), thus losing a measure of the performance F for each batch of episodes, which we instead keep in the objective (4).
4.1 Approximating the Finite Trials Objective with Inﬁnite Trials
Despite the evident mismatch between the ﬁnite trials and the inﬁnite trials formulation of the convex
RL problem, most existing works consider (3) as the standard objective, even if only a ﬁnite number of episodes can be drawn in practice.
Thus, it is worth investigating how much we can lose by approximating a ﬁnite trials objective with an inﬁnite trials one.
First, we report a useful assumption on the structure of the function F .
Assumption 4.1 (Lipschitz) .
A function F : X → R is Lipschitz-continuous if it holds for some constant L (cid:12)(cid:12)
[Equation]
Then, we provide an upper bound on the approximation error in the following theorem.
Theorem 4.1 (Approximation Error) .
Let n ∈ N be a number of trials, let δ ∈ (0 , 1] be a conﬁdence level, let π † ∈ arg max π ∈ Π ζ n ( π ) and π (cid:63) ∈ arg max π ∈ Π ζ ∞ ( π ) .
Then, it holds with probability at least
[Equation]
The previous result establishes an approximation error rate err
=
O ( LH (cid:112)
S/n ) that is polynomial in the number of episodes n .
Unsurprisingly, the guarantees over the approximation error scales with
O (1 / √ n ) , as one can expect the empirical distribution d n to concentrate around its expected value for large n [55].
This implies that approximating the ﬁnite trials objective ζ n ( π ) with the inﬁnite trials ζ ∞ ( π ) can be particularly harmful in those settings in which n is necessarily small.
As an example, consider training a robot through a simulator and deploying the obtained policy in the real world, where the performance measures are often based on a single episode ( n = 1 ).
The performance that we experience from the deployment can be much lower than the expected ζ ∞ ( π ) , which might result in undesirable or unsafe behaviors.
However, Theorem 4.1 only reports an instance-agnostic upper bound, and it does not necessarily imply that there would be a signiﬁcant approximation error in a speciﬁc instance, i.e., a speciﬁc pairing of an MDP M and a function F .
Nevertheless, in this paper we argue that the upper bound of the approximation error is not vacuous in several relevant applications, and we provide an illustrative numerical corroboration of this claim in Section 6.
Challenged assumption 1.
The convex RL problem can be equivalently addressed with an inﬁnite trials formulation.
Finally, in Figure 2 we report a visual representation 4 of the approximation error deﬁned in Theorem 4.1.
Notice that the ﬁnite trials objective ζ n converges uniformly to the inﬁnite trials objective ζ ∞ as a trivial consequence of Theorem 4.1.
This is particularly interesting as it results in π † converging to π (cid:63) in the limit of large n as shown Figure 2.
In-Depth Analysis of Single Trial Convex Reinforcement Learning Setting
Having established a signiﬁcant mismatch between the inﬁnite trials convex RL setting that is usually considered in previous works, i.e., ζ ∞ ( π ) , and the ﬁnite trials formulation that is actually targeted in [Figure] (cid:13)(cid:13) d n − d π (cid:63) (cid:13)(cid:13) 1 , in
(cid:12)(cid:12)
[Equation] − [Equation] ) (cid:12)(cid:12)
(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12) .
practice, i.e., ζ n ( π ) , it is now worth taking a closer look at the ﬁnite trials optimization problem (4).
Indeed, to avoid the approximation error that can occur by optimizing (4) through the inﬁnite trials formulation (Theorem 4.1), one could instead directly address the optimization of (4).
Especially, how does the ﬁnite trials convex RL problem compare to its inﬁnite trials formulation and the linear RL problem?
What kind of policies do we need to optimize the ﬁnite trials objective?
Is the underlying learning process statistically harder than inﬁnite trials convex RL?
In this section, we investigate the answers to these relevant questions.
To this purpose, we will focus on a single trial setting, i.e., ζ n ( π ) with n = 1 , which allows for a clearer analysis, while analogous considerations should extend to a general number of trials n > 1 .
Single Trial Convex RL
[Equation]
[Equation]
Taking inspiration from [58], we can cast the problem (5) deﬁned over an MDP M into a convex MDP
CM :=
( S , A , P, H, µ, F ) , where S , A , P, H, µ are deﬁned as in a standard MDP (see Section 2), and
F : ∆( S ) → R is a convex function that deﬁnes the objective ζ 1 ( π ) .
Is solving a convex MDP
CM signiﬁcantly harder than solving an MDP M ?
5.1 Extended MDP Formulation of the Single Trial Convex RL Setting
We can show that any ﬁnite-horizon convex MDP CM can be actually translated into an equivalent
MDP M (cid:96) = ( S (cid:96) , A (cid:96) , P (cid:96) , µ (cid:96) , r (cid:96) ) , which we call an extended MDP.
The main idea is to temporally- extend
CM so that each state contains the information of the full trajectory leading to it, so that the convex objective can be cast into a linear reward.
To do this, we deﬁne the extended state space S (cid:96) to be the set of all the possible histories up to length H , so that s (cid:96) ∈ S (cid:96) now represents a history.
Then, we can keep
[Equation]
equivalent to A , P, µ of the original CM , where for the extended transition model P (cid:96)
[Equation] s [Equation] | s (cid:96) , a ) we solely consider the last state in the history s (cid:96) to deﬁne the conditional probability to the next history s (cid:48) (cid:96) .
Finally, we just need to deﬁne a scalar reward function r (cid:96) : S (cid:96) → R such that r (cid:96) ( s (cid:96) )
=
F ( d s (cid:96) ) for all the histories s (cid:96) of length H and r (cid:96) ( s (cid:96) )
=
otherwise, where we denoted with d s (cid:96) the empirical state distribution induced by [Equation]
Notably, the problem of ﬁnding an optimal policy for the extended MDP M (cid:96) , i.e., π ∗ ∈ arg max π ∈ Π r (cid:96) · d π , is equivalent to solve the problem (5). Indeed, we have
[Equation] Whereas M (cid:96) can be solved with classical MDP methods [45], the size of the policy π : S (cid:96) → ∆( A (cid:96) ) to be learned does scale with the size of
M (cid:96) , which grows exponentially in the episode horizon as we have |S (cid:96) | > S H .
Thus, the extended MDP formulation and the resulting insight cast some doubts on the notion that the convex RL is not signiﬁcantly harder than standard RL [61, 58].
Challenged assumption 2.
Convex RL is only slightly harder than the standard RL formulation.
Partially Observable MDP Formulation of the Single Trial Convex RL Setting
Instead of temporally extending the convex MDP CM as in the previous section, which causes the policy space to grow exponentially with the episode horizon H , we can alternatively formulate
CM as a Partially Observable MDP (POMDP) [5, 32] PM = ( S (cid:96) , A (cid:96) , P (cid:96) , µ (cid:96) , r (cid:96) , Ω , O ) , in which Ω denotes an observation space, and O : S (cid:96) → ∆(Ω) is an observation function.
The process to build PM is rather similar to the one we employed for the extended MDP M (cid:96) , and the components
S (cid:96) , A (cid:96) , P (cid:96) , µ (cid:96) , r (cid:96) remain indeed unchanged.
However, in a POMDP the agent does not directly access a state s (cid:96) ∈ S (cid:96) , but just a partial observation o ∈ Ω that is given by the observation function O .
Here O ( s (cid:96) )
= o is a deterministic function such that the given observation o is the last state in the history
s (cid:96) . Since the agent only observes o , a stationary policy can be deﬁned as a function π : Ω → ∆( A ) , for which the size depends on the number of states S of the convex MDP CM , being Ω = S .
However, it is well known [32] that history-dependent policies should be considered for the problem of optimizing a POMDP.
This is in sharp contrast with the current convex MDP literature, which only considers stationary policies due to the inﬁnite trials formulation [61].
Challenged assumption 3.
The set of stationary randomized policies is sufﬁcient for convex RL.
5.3 Online Learning in Single Trial Convex RL
Let us assume to have access to a planning oracle that returns an optimal policy π ∗ for a given CM , so that we can sidestep the concerns on the computational feasibility reported in previous sections.
It is worth investigating the complexity of learning π ∗ from online interactions with an unknown CM .
A typical measure of this complexity is the online learning regret R ( N ) , which is deﬁned as
[Equation]
where N is the number of learning episodes, V ∗ =
V π ∗ 1 ( s 1 ) is the value of the optimal policy,
V ( t ) =
V π t is the value of the policy π t deployed at the episode t .
We now aim to assess whether there exists a principled algorithm that can achieve a sub-linear regret R ( N ) in the worst case.
To this purpose, we can cast our learning problem in the Once-Per-Episode (OPE) RL formulation [12].
To translate our ζ 1 ( π ) =
E d ∼ p
π
[ F ( d )] into the OPE framework [12], we have to encode F into a linear representation, which we can get through
[Equation]
in which w is a d w -dimensional vector of linear parameters and B F ,d w is a multivariate Bernstein polynomial [18, Ch. 9] of degree d w .
Then, with some other minor changes that are detailed in the Appendix, we can invoke the analysis of OPE-UCBVI in [12] to provide an upper bound to the regret R ( N ) in our setting.
Before stating the regret result, we recall that F is L F -Lipschitz (Assumption 4.1) and we report an additional smoothness assumption on the value function.
Assumption 5.1.
Let Lip (∆( S ))
the space of the L F -Lipschitz functions on ∆( S ) , and let F (cid:48) , F (cid:48)(cid:48) ∈ Lip (∆(
S )) .
The value function V : Lip (∆( S ))
→ R is L V
-Lipschitz continuous, i.e.,
[Equation]
Theorem 5.1 (Regret) .
For any δ ∈ (0 , 1] , and unknown convex MDP CM , the regret of the
OPE-UCBVI algorithm is upper bounded as
[Equation]
[Table]
with probability 1 − δ , where R δ ( N ) is sub-linear in N .
Compared to the analysis of [12], our regret bound admits an additional approximation term that scales linearly with N .
This is the cost we have for approximating the unknown F with a linear model.
Even if we can overcome the linearity in N with a sufﬁciently large representation d w , this would negatively impact the pure learning term, which scales with O ( d 3 w √ N ) [12, Thr. 3.2].
Notably, for the online learning setting with known function F , d w = 1 , the approximation term vanishes, leading to a cumulatively sub-linear regret O ( √ N ) .
This observation motivates the development of F -speciﬁc algorithms to address the online learning problem in the single trial convex RL setting. [Figure]
Illustrative Examples and Experimental Approximation Error
In this section, we evaluate the performance achieved by a policy π † ∈ arg max π ∈ Π ζ n ( π ) , obtained by maximizing the ﬁnite trials objective (4), against a policy π (cid:63) ∈ arg max π ∈ Π ζ ∞ ( π ) , obtained by maximizing the inﬁnite trials objective (3), over a ﬁnite amount of trajectories . The latter inﬁnite trials π ∗ can be obtained by solving a dual optimization on the convex MDP (see Sec. 6.2 in [42]), [Equation]
subject to [Equation] To get the ﬁnite trials π † , we ﬁrst recover the extended MDP as explained in Section 5.1, and then we apply standard dynamic programming [7].
In the experiments, we show that optimizing the inﬁnite trials objective can lead to sub-optimal policies across a wide range of applications.
In particular, we cover examples from imitation learning, risk-averse RL, and pure exploration.
We carefully selected MDPs that are as simple as possible in order to stress the generality of our results.
For the sake of clarity, we restrict the discussion to the single trial setting ( n = 1 ).
Pure Exploration We consider the maximum state entropy exploration objective as stated in [28], i.e., F ( d π )
= H (cid:0)
[Equation]
=
[Equation]
(cid:2) log d π ( s ) (cid:3) , and the MDP in Figure 3a.
In this example, the agent aims to maximize the state entropy over ﬁnite trajectories of H steps.
Notice that this happens when a policy induces an empirical state distribution that is close to uniform.
In Figure 4a, we compare the average entropy induced by the optimal ﬁnite trials policy π † and the optimal inﬁnite trials policy π (cid:63) .
An agent following the policy π † always achieves a uniform empirical state distribution leading to the maximum entropy.
Moreover, π † is a non-Markovian deterministic policy.
In contrast, the policy π ∗ is randomized in all the three states.
As a result, this policy induces sub-optimal empirical state distributions with strictly positive probability, as shown in Figure 4d.
Risk-Averse RL
For the risk-averse example, we study the Conditional Value-at-Risk (CVaR)
F ( d π ) =
CVaR α ( λ · d π ) , where d π is an admissible state distribution and r is a reward vector [47].
Consider an instance of the MDP in Figure 3b in which the agent aims to maximize the CVaR over ﬁnite trajectories of H steps.
First, notice that a ﬁnancial semantics can be attributed to the given
MDP: an agent, starting in state 2 , can decide whether to invest in risky assets, e.g., crypto-currencies, or in safe ones, e.g., treasury bills.
Because of the stochastic transitions, a policy would need to be reactive to the realizations of the transition model in order to maximize the single trial objective (5).
This kind of behavior is achieved by an optimal ﬁnite trials policy π † .
As a matter of fact, π † is a non-Markovian policy, which can take decisions as a function of the history, and thus takes into account the realizations of the stochastic transition model.
On the other hand, an optimal inﬁnite trials policy π ∗ is a Markovian policy, and it cannot take into account the current history.
As a result, the policy π ∗ induces sub-optimal trajectories with strictly positive probability (see Figure 4e).
In
Figure 4b, we compare the average CVaR on a single trajectory induced by the optimal ﬁnite trials policy π † and the optimal inﬁnite trials policy π (cid:63) .
Overall, π † performs signiﬁcantly better than π (cid:63) .
Imitation Learning We consider the distribution matching formulation of the imitation learning problem [33], i.e., F ( d π ) =
KL ( d π || d E ) , and the MDP in Figure 3c.
The agent aims to learn a policy π inducing an empirical state distribution d π close to the empirical state distribution d E demonstrated by an expert.
In Figure 4c, we compare the average KL -divergence induced by the optimal ﬁnite trials policy π † and the optimal inﬁnite trials policy π (cid:63) .
An agent following the policy π † induces a state visitation distribution that perfectly matches the expert.
In contrast, an agent following π ∗ induces sub-optimal trajectories with strictly positive probability, as shown in Figure 4f. [Figure]
Related Work
To the best of our knowledge, [61] were the ﬁrst to introduce the convex RL problem, under the name of RL with general utilities, with the goal of unifying a set of practically relevant non-scalar objectives for RL under a unique general framework.
Especially, they investigated a hidden convexity of the convex RL objective 5 that allows for statistically efﬁcient policy optimization in the inﬁnite trials setting.
More recently, [58, 22] have considered reinterpretations of the inﬁnite trials convex RL formulation from game-theoretic perspectives.
The former [58] notes that the convex RL problem can be seen as a min-max game between the policy player and a cost player.
The latter [22] shows that the convex RL problem is a subclass of mean-ﬁeld games.
Another relevant branch of literature is the one investigating the expressivity of scalar (Markovian) rewards [2, 50].
Especially, [2] shows that not all the notions of task, such as inducing a set of admissible policies, a (partial) policy ordering, a trajectory ordering, can be naturally encoded with a scalar reward function.
Whereas the convex RL formulation extends the expressivity of scalar
RL w.r.t.
all these three notions of task, it is still not sufﬁcient to cover any instance.
Convex RL is powerful in terms of the policy ordering it can induce, but it is inherently limited on the trajectory ordering as it only accounts for the inﬁnite trials state distribution.
Instead, the ﬁnite trials convex RL setting that we presented in this paper is naturally expressive in terms of trajectory orderings, at the expense of a diminished expressivity on the policy orderings w.r.t.
inﬁnite trials convex RL.
Previous works concerning RL in the presence of trajectory feedback are also related to this work.
Most of this literature assumes an underlying scalar reward model [e.g., 19] which only delays the feedback at the end of the episode.
One notable exception is the once-per-episode formulation in [12], which we have already commented on in Section 5.
Finally, the work in [13, 14] considers inﬁnite-horizon MDPs with vectorial rewards as a mean to encode convex objectives in RL with a multi-objective ﬂavor.
They show that stationary policies are in general sub-optimal for the introduced online learning setting, where non-stationarity becomes essential.
In this setting, they provide principled procedures to learn an optimal policy with sub-linear regret.
Their work essentially complement our analysis in the inﬁnite-horizon problem formulation, where the difference between ﬁnite trials and inﬁnite trials fades away.
Conclusion and Future Directions
While in classic RL the optimization of an inﬁnite trials objective leads to the optimal policy for the ﬁnite trials counterpart, we have shown that true convex RL does not have this property.
First, we have formalized the concept of ﬁnite trials convex RL, which captures a problem that until now has been cast into an unfounded optimization problem.
Then, we have given an upper bound on the approximation error obtained by erroneously optimizing the inﬁnite trials objective, as it is currently done in practice.
Finally, we have presented intuitive, yet general, experimental examples to show that the approximation error can be signiﬁcant in relevant applications.
Since the ﬁnite trials setting is the standard in both simulated and real-world RL, we believe that shedding light on the above mentioned performance gap will lead to better approaches for convex RL and related areas.
Future work could target approximate solutions to the ﬁnite trials objective rather than the inﬁnite trials one, which can cause sub-optimality even when solved exactly.
Methods in POMDPs [27, 35] or optimistic planning algorithms [10] could provide useful inspiration.
Acknowledgements
Riccardo De Santi and Piersilvio De Bartolomeis thank professor Niao He for offering graduate students at ETH the opportunity to work in exciting research areas within the “Foundations of Reinforcement Learning” course. Further, we thank Ali Batuhan Yardim for his generous feedback on an early version of this work.
Theorem 4.1 (Approximation Error) . Let n ∈ N be a number of trials, let δ ∈ (0 , 1] be a conﬁdence level, let π † ∈ arg max π ∈ Π ζ n ( π ) and π (cid:63) ∈ arg max π ∈ Π ζ ∞ ( π ) . Then, it holds with probability at least
[Equation] Proof. Let us ﬁrst upper bound the approximation error as err := [Equation]
[Equation]
where (6) follows from the deﬁnition of π † , π (cid:63) , we derive (7) by plugging the deﬁnitions of ζ n , ζ ∞ in (6), then we obtain (8) from | E [ X ] | ≤ E [ | X | ] , we apply the Lipschitz assumption on F to write (9) from (8), we maximize over the policies to write (10), and we ﬁnally obtain (11) through a maximization over the episode’s step by noting that d n = 1 H (cid:80)
[Equation]
∈ [Equation] [Equation] d
[Equation]
t ∈ [ H ] d πt .
Then, we seek to bound with high probability
[Equation]
where (cid:15) > 0 is a positive constant
[Equation]
and we applied a union bound to get (13) from (12).
From concentration inequalities for empirical distributions (see Theorem [Equation] 1 in [55] and Lemma 16 in [19]) we have [Equation]
By setting δ (cid:48) = δ/ 2 H in (15) [Equation]
and then plugging (15) in (14), and again (14) in (11), we have that with probability at least 1 − δ
[Equation]
Proposition 1 (Finite Trials vs Inﬁnite Trials) .
We provide here some results on the objectives discussed in Table 1.
(i) Let F ( d )
=
r · d then min π
[Equation] , ∀ n ∈ N (ii) Let F ( d ) =
r · d s.t. λ · d ≤ c then min π ∈ Π ζ ∞ ( π ) =
{+min π ∈ Π ζ n ( π ) , ∀ n ∈ N (iii) Let F ( d ) = (cid:107) d − d E (cid:107) 
+}
then min π ∈ Π ζ ∞ ( π ) < min π ∈ Π ζ n ( π ) , ∀ n ∈ N
[Equation]
Proof.
We report below the corresponding derivations.
[Equation]
Theorem
[Equation]
1 (Regret) .
For any δ ∈ (0 , 1] , and unknown convex MDP CM , the regret of the
OPE-UCBVI algorithm is upper bounded as
[Equation]
with probability 1 − δ , where R δ ( N ) is sub-linear in N .
Proof.
In this proof we use the notion d τ to refer to the state visitation frequency induced by a trajectory τ .
We start by splitting the regret measure into two components, an approximation term and an actual online learning regret term,
[Equation]
[Equation]
where N is the total number of episodes played during the learning phase and [Equation] where π ∗ and (cid:101) π ∗ are the optimal policies w.r.t. F and B F ,d w respectively.
Recall that B F ,d w is the multivariate Bernstein polynomial of ﬁxed degree d w for each of the S components, which we can write as follows [Equation] where d τ,j stands for the j -th element of d τ ∈ ∆( S ) . We can now upper bound the ﬁrst term of (16) as follows
N [Equation] In the third step we have exploited the L V -Lipschitz-continuous assumption on V (Assumption 5.1). Notice that in equation (18) we can take the supremum over d τ , since the L V -Lipschitz-continuity holds uniformly over ∆( S ) .
In the last step, we use the following upper bound on the Bernstein approximation error
[Equation]
We now show that the second term of equation (16) can be bounded according to [12, Theorem 3.2]. In particular, they show that, if the learning problem respects a speciﬁc set of assumptions, there exists an algorithm that achieves sub-linear regret in the number of episodes N . We now report such assumptions and show that the learning problem under consideration respects them.
Assumption A.1 (Logistic reward model) .
Given any trajectory τ ∈ Γ , the rewards are said to be drawn from a logistic model, if the law y ( τ ) is
[Equation]
where for any z ∈ R µ ( z ) = 1 1+ exp ( − z ) is the logistic function. We shall refer to w ∗ as the reward parameters .
Notice that by distributing the product, we can rewrite the Bernstein polynomial as follows
[Equation]
Hence, we can reduce our setting to this by setting w ∗(cid:62) φ ( τ ) :
= B F ,d w ( d τ ) .
Since we have [Equation]
[Equation] by using the logistic reward model speciﬁed above, the ordering of the policy w.r.t.
the optimization of B F ,d w is preserved. Hence, the optimal policy (cid:101) π ∗ can be found by using a logistic reward model.
We now report the second assumption. Assumption A.2 (Bounded features and parameters) . We assume that • (cid:107) w ∗ (cid:107) 2 ≤ B for some B > 0 , • ∀ τ ∈ Γ (cid:107) φ ( τ ) (cid:107) 2 ≤ 1 .
Observe that (cid:107) w ∗ (cid:107) 2 :=
(cid:113)(cid:80) j ∈ [ S ] ,k j ∈ [ d w ] | w ∗ ,j,k j | 2 where w j,k j = F [Equation]
Sj [Equation] ≤ k for some k > 0 , since all terms are bounded.
Therefore there exists a value B > 0 s.t. (cid:107) w ∗ (cid:107) 2 ≤ B .
As for the second point in A.2, we have [Equation]
Notice that, in principle, this term can be greater than 1 , but we can apply a normalization procedure to make it arbitrarily small without changing the evaluation of the polynomial.
This can be achieved by dividing it for a ﬁnite value ν by which we can multiply the reward parameters, i.e., the left term in the polynomial in (20).
By doing so, the second assumption in A.2 is satisﬁed.
Hence, we can bound the second term in (16) with
[Equation]
where B > 0 and κ :=
max τ
[Equation]
sup
[Equation]