Challenging Common Assumptions in Convex Reinforcement Learning
Abstract
The classic Reinforcement Learning (RL) formulation concerns the maximization of a scalar reward function.
More recently, convex RL has been introduced to extend the RL formulation to all the objectives that are convex functions of the state distribution induced by a policy.
Notably, convex RL covers several relevant applications that do not fall into the scalar formulation, including imitation learning, risk-averse RL, and pure exploration.
In classic RL, it is common to optimize an inﬁnite trials objective, which accounts for the state distribution instead of the empirical state visitation frequencies, even though the actual number of trajectories is always ﬁnite in practice.
This is theoretically sound since the inﬁnite trials and ﬁnite trials objectives are equivalent and thus lead to the same optimal policy.
In this paper, we show that this hidden assumption does not hold in convex RL.
In particular, we prove that erroneously optimizing the inﬁnite trials objective in place of the actual ﬁnite trials one, as it is usually done, can lead to a signiﬁcant approximation error.
Since the ﬁnite trials setting is the default in both simulated and real-world RL, we believe shedding light on this issue will lead to better approaches and methodologies for convex RL, impacting relevant research areas such as imitation learning, risk-averse RL, and pure exploration among others.
Introduction
Standard Reinforcement Learning (RL) [50] is concerned with sequential decision-making problems in which the utility can be expressed through a linear combination of scalar reward terms.
The coefﬁcients of this linear combination are given by the state visitation distribution induced by the agent’s policy.
Thus, the objective function can be equivalently written as the inner product between the mentioned state distribution and a reward vector.
However, not all the relevant objectives can be encoded through this linear representation [2].
Several works have thus extended the standard
RL formulation to address non-linear objectives of practical interest.
These include imitation learning [30, 42], or the problem of ﬁnding a policy that minimizes the distance between the induced state distribution and the state distribution provided by experts’ interactions [1, 29, 32, 33, 22, 17], risk-averse RL [20], in which the objective is sensitive to the tail behavior of the agent’s policy [52, 43, 51, 16, 15, 8, 61], pure exploration [27], where the goal is to ﬁnd a policy that maximizes the entropy of the induced state distribution [53, 33, 41, 40, 59, 24, 36, 47, 55, 39, 38], diverse skills discovery [23, 19, 25, 48, 11, 35, 28, 58], constrained RL [4, 3, 9, 37, 45, 56, 6], and others.
All this large body of work has been recently uniﬁed into a unique framework, called convex RL [60, 57, 21], which admits as an objective any convex function of the state distribution induced by the agent’s policy.
The convex RL problem has been showed to be largely tractable either computationally, as it admits a dual formulation akin to standard RL [44], or statistically, as principled algorithms achieving sub-linear regret rates that are slightly worse than standard RL have been developed [60, 57].
Finite Trials
[Equation]
assumption which is rarely met in practice.
Indeed, the objective is written as a function of the state distribution, which is an expectation over the empirical state distributions that are actually obtained by running the policy in a given episode.
In practice, we always run our policy for a ﬁnite number of episodes (or trials ), which in general prevents the empirical state distribution from converging to its expectation.
This has never been a problem in standard RL: due to the scalar objective, optimizing the policy over inﬁnite trials or ﬁnite trials is equivalent, as it leads to the same optimal policy.
Crucially, in this paper, we show that this property does not hold for the convex RL formulation: a policy optimized over inﬁnite trials can be signiﬁcantly sub-optimal when deployed over ﬁnite trials (Figure 1).
In light of this observation, we reformulate the convex RL problem from a ﬁnite trials perspective, developing insights that can be used to partially rethink the way convex objectives have been previously addressed in RL, with potential ripple effects to research areas of signiﬁcant interest, such as imitation learning, risk-averse RL, pure exploration, and others.
In this paper, we formalize the notion of a ﬁnite trials RL problem, in which the objective is a function of the empirical state distribution induced by the agent’s policy over n trials rather than its expectation over inﬁnite trials.
As an illustrative example, consider a ﬁnancial application, in which we aim to optimize a trading strategy.
In the real world, we can only deploy the strategy over a single trial.
Thus, we are only interested in the performance of the strategy in the real-world realization, rather than the performance of the strategy when averaging different realizations.
Similar considerations apply to other relevant real-world applications, such as autonomous driving or treatment optimization in a healthcare domain.
Following this intuition, we ﬁrst deﬁne the (linear) ﬁnite trial RL formulation (Section 3), for which it is trivial to prove the equivalence with standard RL.
In Section 4, we provide the ﬁnite trial convex RL formulation, for which we prove an upper bound on the approximation error made by optimizing the inﬁnite trials as a proxy of the ﬁnite trials objective.
In light of this ﬁnding, we challenge the hidden assumption that (1) convex RL can be equivalently addressed with an inﬁnite trials formulation , even if the setting is ﬁnite-trial.
We corroborate this result with an additional numerical analysis showing that the approximation bound is non-vacuous for relevant applications (Section 6).
Finally, in Section 5 we include an in-depth analysis of the single trial convex RL, which suggests that other common assumptions in the convex RL literature, i.e., that (2) the problem is always computationally tractable and that (3) stationary policies are in general sufﬁcient , should be reconsidered as well.
The proofs of the reported results can be found in the Appendix.
Preliminaries
In this section, we report the notation and background useful to understand the paper.
We denote with [ T ] a set of integers { 1 , . . .
, T } , and with a lower case letter a a scalar or a vector, according to the context.
For two vectors a, b ∈ R d , we denote with a · b = (cid:80) di =
a i b i the inner product between a, b .
2.1 Probabilities and Percentiles
Let X denote a measurable space, we will denote with ∆( X ) the probability simplex over X , and with p ∈ ∆( X ) a probability measure over X .
For two probability measures p, q over X , we deﬁne their (cid:96) p -distance as (cid:107) p
[Equation]
p := (cid:0)
[Equation]
x ∈X (cid:12)(cid:12) p
[Equation]
) (cid:12)(cid:12) p (cid:1) 1 / p , and their Kullback-Leibler (KL) divergence as KL( p || q ) :=
[Equation] ( x [Equation] log (cid:0) p ( x ) /q
( x ) (cid:1) .
Let X be a random variable distributed according to p , having a cumulative density function F X ( x ) =
Pr ( X ≤ x ) .
We denote with E [ X ] its expected value, and its α -percentile is denoted as VaR α ( X )
= inf (cid:8) x | F X ( x ) ≥ α (cid:9) =
F − 1 X ( α ) , where α ∈ (0 , 1) is a conﬁdence level, and VaR α stands for Value at Risk (VaR) at level α .
We denote the expected value of X within its α -percentile as CVaR α ( X ) =
E (cid:2)
X | X ≤ VaR α ( X ) (cid:3) , where
CVaR α stands for Conditional Value at Risk (CVaR) at level α .
2.2 Markov Decision Processes
A tabular Markov Decision Process (MDP) [44] is deﬁned as M :=
( S , A , P, T, µ, r ) , where S is a state space of size S ,
A is an action space of size A , P is a Markovian transition model
P : S ×A → ∆( S ) , such that P ( s (cid:48) | s, a ) denotes the conditional probability of the next state s (cid:48) given the current state s and action a , T is the episode horizon, µ is an initial state distribution µ : ∆(
S ) , and r is a scalar reward function r : S → R , such that r ( s ) is the reward collected in the state s .
In the typical interaction episode, an agent ﬁrst observes the initial state s 0 ∼ µ of the MDP.
Then, the agent select an action a 0 , so that the MDP transitions to the next state s 1 ∼ P ( ·| s 0 , a 0 ) , and the agent collects the reward r ( s 1 ) .
Having observed s 1 , the agent then selects an action a 1 triggering a subsequent transition to s 2 ∼ P ( ·| s 1 , a 1 ) .
This process carries on repeatedly until the episode ends.
A policy π deﬁnes the behavior of an agent interacting with an MDP, i.e., the strategy for which an action is selected at any step of the episode.
It consists of a sequence of decision rules ( π t ) ∞ t =
that maps the current trajectory 2 h t
= ( s i , a i ) t − 1 i =
∈ H t with a distribution over actions π t : H t → ∆( A ) , where
H t denotes the set of trajectories of length t .
A non-stationary policy is a sequence of decision rules π t : S → ∆( A ) .
A stationary (Markovian) policy is a time-consistent decision rule π : S → ∆( A ) , such that π ( a | s ) denotes the conditional probability of taking action a in state s .
A trajectory h , obtained from an interaction episode, induces an empirical distribution d over the states of the MDP M ,
such that d ( s ) = 1 | h | (cid:80) s t ∈
h 1 ( s t = s ) .
We denote with p π the probability of drawing d by following the policy π .
For n ∈ N , we denote with d n the empirical distribution d n ( s ) =
1 n (cid:80) ni
=1 d i ( s ) , and with p πn the probability of drawing d n by following the policy π for n episodes.
Finally, we call the expectation d π = E d ∼ p
π
[ d ] the state distribution induced by π .
Reinforcement Learning in Finite Trials
In the standard RL formulation [50], an agent aims to learn an optimal policy by interacting with an unknown MDP M .
An optimal policy is a decision strategy that maximizes the expected sum of rewards collected during an episode.
Especially, we can represent the value of a policy π through the value function V πt ( s ) :
= E π (cid:2) (cid:80)
Tt (cid:48)
=
t r ( s t (cid:48) ) (cid:12)(cid:12) s t
= s (cid:3) .
The value function allows us to write the RL objective as max π ∈ Π E s 1 ∼ µ [ V π 1 ( s 1 )] , where Π is the set of all the stationary policies.
Equivalently, we can rewrite the RL objective into its dual formulation [44], i.e.,
[Equation]
where we denote with r ∈ R S a reward vector, and with d π the state distribution induced by π .
We call the problem (1) the inﬁnite trials RL formulation.
Indeed, the objective J ∞ ( π ) considers the sum of the rewards collected during an episode, i.e., r · d π , that we can achieve on the average of an inﬁnite number of episodes drawn with π .
This is due to the state distribution d π being an expectation of empirical distributions
d π = E
d ∼ p π
[ d ] .
However, in practice, we can never draw inﬁnitely many episodes following a policy π .
Instead, we draw a small batch of episodes d n ∼ p πn .
Thus, we can instead conceive a ﬁnite trials RL formulation that is closer to what is optimized in practice.
[Table]
One could then wonder whether optimizing the ﬁnite trials objective (2) leads to results that differ from the inﬁnite trials one (1).
To this point
[Table]
it is straightforward to see that the two objective functions are actually equivalent
[Equation]
since r is a constant vector and the expectation is a linear operator.
It follows that the inﬁnite trials and the ﬁnite trials RL formulations admit the same optimal policies.
Hence, one can enjoy both the computational tractability of the inﬁnite trials formulation and, at the same time, optimize the objective function that is employed in practice.
In the next section, we will show that a similar result does not hold true for the convex RL formulation.
Convex Reinforcement Learning in Finite Trials
Even though the RL formulation covers a wide range of sequential decision-making problems, several relevant applications cannot be expressed by means of the inner product between a linear reward vector r and a state distribution d π [2, 49].
These include imitation learning, pure exploration, constrained problems, and risk-sensitive objectives, among others.
Recently, a convex RL formulation [60, 57, 21] has been proposed to unify these applications in a unique general framework, which is
[Equation]
where F : ∆( S ) → R is a function 3 of the state distribution d π .
In Table 1, we recap some of the most relevant problems that fall under the convex RL formulation, along with their speciﬁc F function.
As it happens for linear RL, in any practical simulated or real-world scenario, we can only draw ﬁnite number of episodes with a policy π .
From these episodes, we obtain an empirical distribution d n ∼ p πn rather than the actual state distribution d π , where n is the number of episodes.
This can cause a mismatch from the objective that is typically considered in convex RL [60, 57], and what can be optimized in practice.
To overcome this mismatch, we deﬁne a ﬁnite trials formulation of the convex RL objective, as we did in the previous section for the linear RL formulation.
[Equation]
However, in the following we will consider functions F that are either convex, concave or even non-convex
In general, problem (3) takes the form of a max problem for concave F or a min problem for convex F .
Comparing objectives (3) and (4), one can notice that both of them include an expectation over the episodes, being d π = E d ∼ p π [ d ] .
Especially, we can write
[Equation]
through the Jensen’s inequality.
As a consequence, optimizing the inﬁnite trials objective ζ ∞ ( π ) does not necessarily lead to an optimal behavior for the ﬁnite trials objective ζ n ( π ) .
From a mathematical perspective, this is due to the fact that the empirical distributions d n induced by the policy π are averaged by the expectation d π before computing the F function into objective (3), thus losing a measure of the performance F for each batch of episodes, which we instead keep in the objective (4).
4.1 Approximating the Finite Trials Objective with Inﬁnite Trials
Despite the evident mismatch between the ﬁnite trials and the inﬁnite trials formulation of the convex
RL problem, most existing works consider (3) as the standard objective, even if only a ﬁnite number of episodes can be drawn in practice.
Thus, it is worth investigating how much we can lose by approximating a ﬁnite trials objective with an inﬁnite trials one.
First, we report a useful assumption on the structure of the function F .
Assumption 4.1 (Lipschitz) .
A function F : X → R is Lipschitz-continuous if it holds for some constant L (cid:12)(cid:12)
[Equation]
Then, we provide an upper bound on the approximation error in the following theorem.
Theorem 4.1 (Approximation Error) .
Let n ∈ N be a number of trials, let δ ∈ (0 , 1] be a conﬁdence level, let π † ∈ arg max π ∈ Π ζ n ( π ) and π (cid:63) ∈ arg max π ∈ Π ζ ∞ ( π ) .
Then, it holds with probability at least
[Equation]
The previous result establishes an approximation error rate err
=
O ( LT (cid:112)
S/n ) that is polynomial in the number of episodes n .
Unsurprisingly, the guarantees over the approximation error scales with
O (1 / √ n ) , as one can expect the empirical distribution d n to concentrate around its expected value for large n [54].
This implies that approximating the ﬁnite trials objective ζ n ( π ) with the inﬁnite trials ζ ∞ ( π ) can be particularly harmful in those settings in which n is necessarily small.
As an example, consider training a robot through a simulator and deploying the obtained policy in the real world, where the performance measures are often based on a single episode ( n = 1 ).
The performance that we experience from the deployment can be much lower than the expected ζ ∞ ( π ) , which might result in undesirable or unsafe behaviors.
However, Theorem 4.1 only reports an instance-agnostic upper bound, and it does not necessarily imply that there would be a signiﬁcant approximation error in a speciﬁc instance, i.e., a speciﬁc pairing of an MDP M and a function F .
Nevertheless, in this paper we argue that the upper bound of the approximation error is not vacuous in several relevant applications, and we provide an illustrative numerical corroboration of this claim in Section 6.
Challenged assumption 1.
The convex RL problem can be equivalently addressed with an inﬁnite trials formulation.
Finally, in Figure 2 we report a visual representation 4 of the approximation error deﬁned in Theorem 4.1.
Notice that the ﬁnite trials objective ζ n converges uniformly to the inﬁnite trials objective ζ ∞ as a trivial consequence of Theorem 4.1.
This is particularly interesting as it results in π † converging to π (cid:63) in the limit of large n as shown Figure 2.
In-Depth Analysis of Single Trial Convex Reinforcement Learning Setting
Having established a signiﬁcant mismatch between the inﬁnite trials convex RL setting that is usually considered in previous works, i.e., ζ ∞ ( π ) , and the ﬁnite trials formulation that is actually targeted in
[Figure]
(cid:13)(cid:13)
d n − d π (cid:63) (cid:13)(cid:13) 1 , in
(cid:12)(cid:12)
[Equation]
−
[Equation]
) (cid:12)(cid:12) (cid:12)(cid:12)
[Equation]
(cid:12)(cid:12) .
practice, i.e., ζ n ( π ) , it is now worth taking a closer look at the ﬁnite trials optimization problem (4).
Indeed, to avoid the approximation error that can occur by optimizing (4) through the inﬁnite trials formulation (Theorem 4.1), one could instead directly address the optimization of (4).
Especially, how does the ﬁnite trials convex RL problem compare to its inﬁnite trials formulation and the linear RL problem?
What kind of policies do we need to optimize the ﬁnite trials objective?
Is the underlying learning process statistically harder than inﬁnite trials convex RL?
In this section, we investigate the answers to these relevant questions.
To this purpose, we will focus on a single trial setting, i.e., ζ n ( π ) with n = 1 , which allows for a clearer analysis, while analogous considerations should extend to a general number of trials n > 1 .
Single Trial Convex RL
[Equation]
[Equation]
Taking inspiration from [57], we can cast the problem (5) deﬁned over an MDP M into a convex MDP
CM :=
( S , A , P, T, µ, F ) , where S , A , P, T, µ are deﬁned as in a standard MDP (see Section 2), and
F : ∆( S ) → R is a convex function that deﬁnes the objective ζ 1 ( π ) .
Is solving a convex MDP
CM signiﬁcantly harder than solving an MDP M ?
5.1 Extended MDP Formulation of the Single Trial Convex RL Setting
We can show that any ﬁnite-horizon convex MDP CM can be actually translated into an equivalent
MDP M (cid:96) =
( S (cid:96) , A (cid:96) , P (cid:96) , µ (cid:96) , r (cid:96) ) , which we call an extended MDP.
The main idea is to temporally- extend
CM so that each state contains the information of the full trajectory leading to it, so that the convex objective can be cast into a linear reward.
To do this, we deﬁne the extended state space S (cid:96) to be the set of all the possible histories up to length T , so that s (cid:96) ∈ S (cid:96) now represents a history.
Then, we can keep
[Equation]
equivalent to A , P, µ of the original CM , where for the extended transition model P (cid:96)
[Equation]
s
[Equation]
| s (cid:96) , a ) we solely consider the last state in the history s (cid:96) to deﬁne the conditional probability to the next history s (cid:48) (cid:96) .
Finally, we just need to deﬁne a scalar reward function r (cid:96) : S (cid:96) → R such that r (cid:96) ( s (cid:96) )
=
F ( d s (cid:96) ) for all the histories s (cid:96) of length T and r (cid:96) ( s (cid:96) )
=
otherwise, where we denoted with d s (cid:96) the empirical state distribution induced by
[Equation]
Notably, the problem of ﬁnding an optimal policy for the extended MDP M (cid:96) , i.e., π ∗ ∈ arg max π ∈ Π r (cid:96) · d π , is equivalent to solve the problem (5).
Indeed, we have
[Equation]
Whereas M (cid:96) can be solved with classical MDP methods [44], the size of the policy π : S (cid:96) → ∆( A (cid:96) ) to be learned does scale with the size of
M (cid:96) , which grows exponentially in the episode horizon as we have |S (cid:96) | > S T .
Thus, the extended MDP formulation and the resulting insight cast some doubts on the notion that the convex RL is not signiﬁcantly harder than standard RL [60, 57].
Challenged assumption 2.
Convex RL is only slightly harder than the standard RL formulation.
Partially Observable MDP Formulation of the Single Trial Convex RL Setting
Instead of temporally extending the convex MDP CM as in the previous section, which causes the policy space to grow exponentially with the episode horizon T , we can alternatively formulate
CM as a Partially Observable MDP (POMDP) [5, 31] PM = ( S (cid:96) , A (cid:96) , P (cid:96) , µ (cid:96) , r (cid:96) , Ω , O ) , in which Ω denotes an observation space, and O : S (cid:96) → ∆(Ω) is an observation function.
The process to build PM is rather similar to the one we employed for the extended MDP M (cid:96) , and the components
S (cid:96) , A (cid:96) , P (cid:96) , µ (cid:96) , r (cid:96) remain indeed unchanged.
However, in a POMDP the agent does not directly access a state s (cid:96) ∈ S (cid:96) , but just a partial observation o ∈ Ω that is given by the observation function O .
Here O ( s (cid:96) )
= o is a deterministic function such that the given observation o is the last state in the history
s (cid:96) .
Since the agent only observes o , a stationary policy can be deﬁned as a function π : Ω → ∆( A ) , for which the size depends on the number of states S of the convex MDP CM , being Ω = S .
However, it is well known [31] that history-dependent policies should be considered for the problem of optimizing a POMDP.
This is in sharp contrast with the current convex MDP literature, which only considers stationary policies due to the inﬁnite trials formulation [60].
Challenged assumption 3.
The set of stationary randomized policies is sufﬁcient for convex RL.
5.3 Online Learning in Single Trial Convex RL
Let us assume to have access to a planning oracle that returns an optimal policy π ∗ for a given CM , so that we can sidestep the concerns on the computational feasibility reported in previous sections.
It is worth investigating the complexity of learning π ∗ from online interactions with an unknown CM .
A typical measure of this complexity is the online learning regret R ( N ) , which is deﬁned as
[Equation]
where N is the number of learning episodes, V ∗ =
V π ∗ 1 ( s 1 ) is the value of the optimal policy,
V ( t ) =
V π t is the value of the policy π t deployed at the episode t .
We now aim to assess whether there exists a principled algorithm that can achieve a sub-linear regret R ( N ) in the worst case.
To this purpose, we can cast our learning problem in the Once-Per-Episode (OPE) RL formulation [12].
In the latter setting, the agent interacts with the MDP for T steps, receiving a 0 / 1 feedback at the end of the episode, where the feedback is computed according to a logistic model that is function of the history.
To translate our objective ζ 1 ( π ) =
E d ∼ p
π
[ F ( d )] into the OPE framework [12], we have to encode F into a linear representation.
With the following assumption, we state the existence of such representation.
Assumption 5.1 (Linear Realizability) .
The function F is linearly-realizable if it holds
[Equation]
where w ∗ ∈ R d w is a vector of parameters such that (cid:107) w ∗ (cid:107) 2 ≤ B for some known B > 0 , and φ ( d )
= ( φ j ( d ))
d w j =1 is a known vector of basis functions such that (cid:107) φ ( d ) (cid:107) 2 ≤ 1 , ∀ d ∈ ∆( S ) .
With the Assumption 5.1 and other minor changes that are detailed in the Appendix, we can invoke the analysis of OPE-UCBVI in [12] to provide an upper bound to the regret R ( N ) in our setting.
Theorem 5.1 (Regret) .
For any conﬁdence δ ∈ (0 , 1] and unknown convex MDP CM , the regret of the OPE-UCBVI algorithm is upper bounded as
[Equation]
with probability 1 − δ .
The latter result states that the problem of learning an optimal policy in a unknown convex MDP is at least statistically efﬁcient assuming linear realizability and the access to a planning oracle.
Those are fairly strong assumptions, but principled approximate solvers may be designed to overcome the planning oracle assumption, whereas in several convex RL settings the function F is assumed to be known, and thus trivially realizable.
[Figure]
Numerical Validation
In this section, we evaluate the performance over the ﬁnite trials objective (4) achieved by a policy π † ∈ arg max π ∈ Π ζ n ( π ) maximizing the same ﬁnite trials objective (4) against a policy π (cid:63) ∈ arg max π ∈ Π ζ ∞ ( π ) maximizing the inﬁnite trials objective (3) instead.
The latter inﬁnite trials π ∗ can be obtained by solving a dual optimization on the convex MDP (see Sec.
6.2 in [41]),
[Figure]
[Equation]
subject to
[Equation]
To get the ﬁnite trials π † , we ﬁrst recover the extended MDP as explained in Section 5.1, and then we apply standard dynamic programming [7].
In the experiments, we show that optimizing the inﬁnite trials objective can lead to sub-optimal policies across a wide range of applications.
In particular, we cover examples from pure exploration, risk-averse RL, and imitation learning.
We carefully selected
MDPs that are as simple as possible in order to stress the generality of our results.
For the sake of clarity, we restrict the discussion to the single trial setting ( n = 1 ).
Pure Exploration For the pure exploration setting, we consider the state entropy objective [27], i.e., F ( d )
=
H ( d ) = − d · log d, and the convex MDP in Figure 3a.
In this example, the agent aims to maximize the state entropy over ﬁnite trajectories of T steps.
Notice that this happens when a policy induces an empirical state distribution that is close to uniform.
In Figure 4a, we compare the average entropy induced by the optimal ﬁnite trials policy π † and the optimal inﬁnite trials policy π (cid:63) .
An agent following the policy π † always achieves a uniform empirical state distribution leading to the maximum entropy.
Moreover, π † is a non-Markovian deterministic policy.
In contrast, the policy π ∗ is randomized in all the three states.
As a result, this policy induces sub-optimal empirical state distributions with strictly positive probability, as shown in Figure 4d.
Risk-Averse RL
For the risk-averse RL setting, we consider a Conditional Value-at-Risk (CVaR) objective [46] given by F ( d ) =
CVaR α [ r · d ] , where r ∈ [0 , 1] S is a reward vector, and the convex
MDP in Figure 3b, in which the agent aims to maximize the CVaR over a ﬁnite-length trajectory of T steps.
First, notice that a ﬁnancial semantics can be attributed to the given MDP.
An agent, starting in state 2 , can decide whether to invest in risky assets, e.g., crypto-currencies, or in safe ones, e.g., treasury bills.
Because of the stochastic transitions, a policy would need to be reactive to the realizations of the transition model in order to maximize the single trial objective (5).
This kind of behavior is achieved by an optimal ﬁnite trials policy π † .
Indeed, π † is a non-Markovian deterministic policy, which can take decisions as a function of history, and thus takes into account the current realization.
On the other hand, an optimal inﬁnite trials policy π ∗ is a Markovian policy, and it cannot take into account the current history.
As a result, the policy π ∗ induces sub-optimal trajectories with strictly positive probability (see Figure 4e).
Finally, in Figure 4b we compare the single trial performance induced by the optimal single trial policy π † and the optimal inﬁnite trials policy π (cid:63) .
Overall, π † performs signiﬁcantly better than π (cid:63) .
For the imitation learning setting, we consider the distribution matching objective [32], i.e., F ( d ) =
KL ( d || d E ) , and the convex MDP in Figure 3c.
The agent aims to learn a policy π inducing an empirical state distribution d close to the empirical state distribution d E demonstrated by an expert.
In Figure 4c, we compare single trial performance induced by the optimal single trial policy π † and the optimal inﬁnite trials policy π (cid:63) .
An agent following π † induces an empirical state distribution that perfectly matches the expert.
In contrast, an agent following π ∗ induces sub-optimal realizations with strictly positive probability (see Figure 4f).
[Figure]
Related Work
To the best of our knowledge, [27] were the ﬁrst to introduce the convex RL problem, as a generalization of the standard RL formulation to non-linear utilities, especially the entropy of the state distribution. They show that the convex RL objective, while being concave (convex) in the state distribution, can be non-concave (non-convex) in the policy parameters. Anyway, they provide a provably efﬁcient algorithm that overcomes the non-convexity through a Frank-Wolfe approach. [60] study the convex RL problem under the name of RL with general utilities.
Especially, they investigated a hidden convexity of the convex RL objective that allows for statistically efﬁcient policy optimization in the inﬁnite-trials setting.
Recently, the inﬁnite trials convex RL formulation has been reinterpreted from game-theoretic perspectives [57, 21].
The former [57] notes that the convex RL problem can be seen as a min-max game between the policy player and a cost player.
The latter [21] shows that the convex RL problem is a subclass of mean-ﬁeld games.
Another relevant branch of literature is the one investigating the expressivity of scalar (Markovian) rewards [2, 49].
Especially, [2] shows that not all the notions of task, such as inducing a set of admissible policies, a (partial) policy ordering, a trajectory ordering, can be naturally encoded with a scalar reward function.
Whereas the convex RL formulation extends the expressivity of scalar
RL w.r.t.
all these three notions of task, it is still not sufﬁcient to cover any instance.
Convex RL is powerful in terms of the policy ordering it can induce, but it is inherently limited on the trajectory ordering as it only accounts for the inﬁnite trials state distribution.
Instead, the ﬁnite trials convex RL setting that we presented in this paper is naturally expressive in terms of trajectory orderings, at the expense of a diminished expressivity on the policy orderings w.r.t.
inﬁnite trials convex RL.
Previous works concerning RL in the presence of trajectory feedback are also related to this work.
Most of this literature assumes an underlying scalar reward model [e.g., 18] which only delays the feedback at the end of the episode.
One notable exception is the once-per-episode formulation in [12], which we have already commented on in Section 5.
Finally, the work in [13, 14] considers inﬁnite-horizon MDPs with vectorial rewards as a mean to encode convex objectives in RL with a multi-objective ﬂavor.
They show that stationary policies are in general sub-optimal for the introduced online learning setting, where non-stationarity becomes essential.
In this setting, they provide principled procedures to learn an optimal policy with sub-linear regret.
Their work essentially complement our analysis in the inﬁnite-horizon problem formulation, where the difference between ﬁnite trials and inﬁnite trials fades away.
Conclusion and Future Directions
While in classic RL the optimization of an inﬁnite trials objective leads to the optimal policy for the ﬁnite trials counterpart, we have shown that true convex RL does not have this property.
First, we have formalized the concept of ﬁnite trials convex RL, which captures a problem that until now has been cast into an unfounded optimization problem.
Then, we have given an upper bound on the approximation error obtained by erroneously optimizing the inﬁnite trials objective, as it is currently done in practice.
Finally, we have presented intuitive, yet general, experimental examples to show that the approximation error can be signiﬁcant in relevant applications.
Since the ﬁnite trials setting is the standard in both simulated and real-world RL, we believe that shedding light on the above mentioned performance gap will lead to better approaches for convex RL and related areas.
Future work could target approximate solutions to the ﬁnite trials objective rather than the inﬁnite trials one, which can cause sub-optimality even when solved exactly.
Methods in POMDPs [26, 34] or optimistic planning algorithms [10] could provide useful inspiration.
Acknowledgements
Riccardo De Santi and Piersilvio De Bartolomeis thank professor Niao He for offering graduate students at ETH the opportunity to work in exciting research areas within the “Foundations of
Reinforcement Learning” course.
Further, we thank Ali Batuhan Yardim for his generous feedback on an early version of this work.
A Proofs
Theorem 4.1 (Approximation Error) .
Let n ∈ N be a number of trials, let δ ∈ (0 , 1] be a conﬁdence level, let π † ∈ arg max π ∈ Π ζ n ( π ) and π (cid:63) ∈ arg max π ∈ Π ζ ∞ ( π ) .
Then, it holds with probability at least
[Equation]
Proof.
Let us ﬁrst upper bound the approximation error as err :=
[Equation]
where (6) is obtained by adding ± ζ ∞ ( π † ) and then applying the triangle inequality, (7) follows by noting that ζ ∞ ( π (cid:63) ) ≥ ζ ∞ ( π † ) , we derive (8) by plugging the deﬁnitions of ζ n , ζ ∞ in (7), then we obtain (9) from | E [ X ] | ≤ E [ | X | ] , we apply the Lipschitz assumption on F to write (10) from (9), we maximize over the policies to write (11), and we ﬁnally obtain (12) through a maximization over the episode’s step by noting that d n = 1 T
[Equation]
and
[Equation]
[Equation]
] d πt .
Then, we seek to bound with a high probability
[Equation]
[Equation]
where (cid:15) > 0 is a positive constant, and we applied a union bound to get (14) from (13).
From concentration inequalities for empirical distributions (see Theorem 2.1 in [54] and Lemma 16 in [18]) we have
[Equation]
[Equation]
By setting δ (cid:48)
=
δ/ 2 T in (16), and then plugging (16) in (15), and again (15) in (12), we have that with probability at least 1 − δ
[Equation]
which concludes the proof
[Equation]
Proposition 1 (Finite Trials vs Inﬁnite Trials) .
We provide here some results on the objectives discussed in Table 1.
(i) Let F ( d )
=
r · d then min π ∈ Π ζ ∞ ( π ) =
min π ∈ Π ζ n ( π ) , ∀ n ∈ N (ii) Let F ( d )
=
r · d s.t. λ · d ≤ c then min π ∈ Π ζ ∞ ( π ) =
min π ∈ Π ζ n ( π ) , ∀ n ∈ N (iii) Let F ( d ) = (cid:107) d − d E (cid:107)
then min π [Equation]
[Equation]
(iv) Let F ( d ) = − d · log( d ) =
H ( d ) then min π
[Equation]
n ( π ) , ∀ n ∈ N (v) Let F ( d )
=
KL( d || d E ) then min π
[Equation]
Proof.
We report below the corresponding derivations.
(i) min
[Equation]
(ii) min π
[Equation]
(iii) [Equation]
π [Equation] (iv) min π
[Equation]
(v) min π ∈ Π
[Equation]
Theorem 5.1 (Regret) .
For any conﬁdence δ ∈ (0 , 1] and unknown convex MDP CM , the regret of the OPE-UCBVI algorithm is upper bounded as
[Equation]
with probability 1 − δ
[Equation]
Proof.
To prove the result, we show that the described online learning setting can be translated into the once-per-episode framework [12].
The main difference between the setting in [12] and ours is that they assume a binary feedback y ∈ { 0 , 1 } coming from a logistic model
[Equation]
instead of our richer F ( d ) .
To transform the latter in the binary reward y , we note that F ( d ) =
w (cid:62)∗ φ ( d ) through linear realizability (Assumption 5.1), then we ﬁlter F ( d ) through a logistic model to obtain y =
σ
[Equation]
, which is then used as feedback for OPE-UCBVI.
In this way, we can call
Theorem 3.2 of [12] to obtain the same regret rate up to a constant factor, which is caused by the different range of per-episode contributions in the regret.
For detailed derivations and the complete regret upper bound see [12].