Original: A R EDUCTION A PPROACH TO C ONSTRAINED R EINFORCEMENT L EARNING
Revised: A R EDUCTION A PPROACH TO C ONSTRAINED R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many applications of reinforcement learning (RL) optimize a long-term reward subject to risk, safety, budget, diversity or other constraints.
Revised: Many applications of reinforcement learning (RL) optimize a long-term reward subject to risk, safety, budget, diversity or other constraints.

Original: Though constrained
Revised: Though constrained

Original: RL problem has been studied to incorporate various constraints, existing methods  require randomization among inﬁnitely many  policies  to approach a feasible solution, which is impractical.
Revised: RL problem has been studied to incorporate various constraints, existing methods either tie to speciﬁc families of RL algorithms or require storing inﬁnitely many individual policies found by an RL oracle to approach a feasible solution.

Original: In this paper, we present a  reduction approachto ﬁnd sparse policies that randomize among a constant number of policies forthe constrained RL problem.
Revised: In this paper, we present a novel reduction approach for constrained RL problem that ensures convergence when using any off-the-shelf RL algorithm to construct an RL oracle yet requires storing at most constantly many policies.

Original: The key idea is to reduce the constrained RL problem to a distance minimization problem, and a novel variant of Frank-Wolfetype algorithm is proposed for this task.
Revised: The key idea is to reduce the constrained RL problem to a distance minimization problem, and a novel variant of Frank-Wolfe algorithm is proposed for this task.

Original: Throughout the learning process, our method maintains a sparse combination of individual policies, and we show that the number of policies stored is  worst-case optimal to work with any RL algorithm.
Revised: Throughout the learning process, our method maintains at most constantly many individual policies, where the constant is shown to be worst-case optimal to ensure convergence of any RL oracle.

Original: Using any off-the-shelf RL algorithm as an oracle , our method is shown to strictly reduce the approximation error between each call of the oracle, and improvement in convergence rate over previous results is presented.
Revised: Our method comes with rigorous convergence and complexity analysis, and does not introduce any extra hyper-parameter.

Original: Experiments on a grid-world navigation task demonstrate  our method stores less policies and outperforms previous methods.
Revised: Experiments on a grid-world navigation task demonstrate the efﬁciency of our method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Contemporary approaches in reinforcement learning (RL) largely focus on optimizing the behavior of an agent against a single reward function.
Revised: Contemporary approaches in reinforcement learning (RL) largely focus on optimizing the behavior of an agent against a single reward function.

Revised: RL algorithms like value function methods (Zou et al., 2019; Zheng et al., 2018) or policy optimization methods (Chen et al., 2019; Zhao et al., 2017) are widely used in real-world tasks.

Original: This can be sufﬁcient for simple tasks.
Revised: This can be sufﬁcient for simple tasks.

Original: However, for complicated applications, designing a reward function that implicitly deﬁnes the desired behavior can be challenging.
Revised: However, for complicated applications, designing a reward function that implicitly deﬁnes the desired behavior can be challenging.

Original: For instance, applications concerning risk (Geibel & Wysotzki, 2005; Chow &
Revised: For instance, applications concerning risk (Geibel & Wysotzki, 2005; Chow &

Original: Ghavamzadeh, 2014; Chow et al., 2017), safety (Chow et al., 2018) or budget (Boutilier & Lu, 2016;
Revised: Ghavamzadeh, 2014; Chow et al., 2017), safety (Chow et al., 2018) or budget (Boutilier & Lu, 2016;

Original: Xiao et al., 2019)have been studied previously. Such tasks are naturally modelled by augmenting the RL problem with  constraints.
Revised: Xiao et al., 2019) are naturally modelled by augmenting the RL problem with orthant constraints.

Original: Constrained Markov decision process (CMDP) is a well-established framework to model RL problems under orthant constraints, and Altman (1999) shows that ﬁnite CMDPs with known transition model can be solved by linear programming. To incorporate new classes of constraints, such as to constrain the diversity of the trajectories for better exploration, Miryooseﬁ et al. (2019) proposed to study the convex constrained reinforcement learning problem (C2RL).
Revised: Exploration suggestions, such as to visit all states as evenly as possible, can be modelled by using a vector to measure the behavior of the agent, and to ﬁnd a policy whose measurement vector lies in a convex set (Miryooseﬁ et al., 2019). To solve RL problem under constraints, existing methods either ensure convergence only on a speciﬁc family of RL algorithms, or treat the underlying RL algorithms as a black box oracle to ﬁnd individual policy, and look for mixed policy that randomizes among these individual policies.

Original: In C2RL, instead of a scalar reward, the agent receives a measurement vector at each step. The goal is to ﬁnd a policy whose expected long-term measurement vector lies in a given convex set. They reformulate the problem to a zero sum game, and solve it with online learning. However, the policy found by this online learning approach requires randomization among all policies found during the iterations.
Revised: Though the second group of methods has the advantage of working with arbitrary RL algorithms that best suit the underlying problem, existing methods have practically infeasible memory requirement.

Original: To get an (cid:15) -approximate solution, it requires randomizing among O (1 /(cid:15)2 ) individual policies, and an exact solution requires randomization among inﬁnitely many policies.
Revised: To get an (cid:15) -approximate solution, they require storing O (1 /(cid:15) ) individual policies, and an exact solution requires storing inﬁnitely many policies.

Original: This limits the usage of such game-theoretic approach, especially whendeep neural networks are used to represent the policies, as storing the parameters of all intermediate deep neural networks is resource-demanding for largescale usages.
Revised: This limits the prevalence of such methods, especially when the individual policy uses deep neural networks.

Original: In this paper, we propose a  reduction approach to C2RL.
Revised: In this paper, we propose a novel reduction approach for the general convex constrained RL (C2RL) problem.

Revised: Our approach has the advantage of the second group of methods, yet requires storing at most constantly many policies.

Original: For a vector-valued Markov Decision Process (MDP) and any given target convex set, our method ﬁnds a mixed policy whose measurement vector lies in the target convex set, using any off-the-shelf RL algorithm that optimizes a scalar reward as a RL oracle .
Revised: For a vector-valued Markov Decision Process (MDP) and any given target convex set, our method ﬁnds a mixed policy whose measurement vector lies in the target convex set, using any off-the-shelf RL algorithm that optimizes a scalar reward as a RL oracle .

Original: To do so, the C2RL problem is reduced to a distance minimization problem between a polytope and a convex set, and a novel variant of Frank-Wolfe type algorithm is proposed 
Revised: To do so, the C2RL problem is reduced to a distance minimization problem between a polytope and a convex set, and a novel variant of Frank-Wolfe type algorithm is proposed to solve this distance minimization problem.

Revised: To ﬁnd an (cid:15) -approximate solution in an m -dimensional vector-valued MDP,

Original: [Table]
Revised: [Table]

Revised: et al

Original: [Table]
Revised: [Table]

Original: (cid:55) (cid:55)
Revised: [Table]

Original: Le et al.
Revised: Le et al.

Original: O (1 /(cid:15) ) (cid:55) (cid:51) O (1 /(cid:15) ) Miryooseﬁ et al.
Revised: [Table]

Original: O (1 /(cid:15) 2 ) O (1 /(cid:15) 2 ) (cid:51) O (1 /(cid:15) 2 ) SparseC2RL (this paper)
Revised: [Table]

Original:  O (1 /(cid:15) )O (1 /(cid:15) ) (cid:51) ≤ m +to solve this distance minimization problem.
Revised: our method only stores at most m + 1 policies, which improves from inﬁnitely many O (1 /(cid:15) )

Original: To ﬁnd an (cid:15) -approximate solution, the number of RL oracle invocation reduces from O (1 /(cid:15) 2 ) (Miryooseﬁ et al., 2019) to O (1 /(cid:15) ) (our method). Beside the improved performance, for an m -dimensional vector-valued MDP, our method only stores at most m + 1 policies, which improves from the quadratic O (1 /(cid:15) 2 ) (Miryooseﬁ et al., 2019) to a constant.
Revised: (Le et al., 2019; Miryooseﬁ et al., 2019) to a constant.

Original: The constant is also shown to be worst-case optimal to ensure convergence of RL algorithms using deterministic policies.
Revised: We also show this m + 1 constant is worstcase optimal to ensure convergence of RL algorithms using deterministic policies.

Original: Since the measurement vector of many constraints are low-dimensional (such as risk, safety or budget), this sparse property makes our method appealing for large scale usages. A preliminary experimental comparison with approachability-based policy optimization (ApproPO) (Miryooseﬁ et al., 2019) and reward constrained policy optimization (RCPO)
Revised: Moreover, our method introduces no extra hyper-parameter, which is favorable for practical usage.

Original: (Tessler et al., 2018) demonstrates the performance of the proposed method and the sparsity of the policy found.A theoretical comparison between our method and various approaches is provided in Table 1.
Revised: A preliminary experimental comparison demonstrates the performance of the proposed method and the sparsity of the policy found.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: For high dimensional constrained RL, one line of approaches incorporates the constraint as a penalty signal into the reward function, and makes updates in a multiple time-scale scheme (Tessler et al., 2018; Chow & Ghavamzadeh, 2014).
Revised: For high dimensional constrained RL, one line of approaches incorporates the constraint as a penalty signal into the reward function, and makes updates in a multiple time-scale scheme (Tessler et al., 2018; Chow & Ghavamzadeh, 2014).

Original: When used with policy gradient or actor-critic algorithms (Sutton & Barto, 2018), this penalty signal guides the policy to converge to a constraint satisfying one (Paternain et al., 2019; Chow et al., 2017).
Revised: When used with policy gradient or actor-critic algorithms (Sutton & Barto, 2018), this penalty signal guides the policy to converge to a constraint satisfying one (Paternain et al., 2019; Chow et al., 2017).

Original: However, the convergence guarantee requires the RL algorithm can ﬁnd a single policy that satisﬁes the constraint, hence ruling out methods that search for deterministic policies, such as Deep Q-Networks (DQN) (Mnih et al., 2013), Deep Deterministic
Revised: However, the convergence guarantee requires the RL algorithm can ﬁnd a single policy that satisﬁes the constraint, hence ruling out methods that search for deterministic policies, such as Deep Q-Networks (DQN) (Mnih et al., 2013), Deep Deterministic

Original: Policy Gradient (DDPG) (Lillicrap et al., 2015) and their variants (Van Hasselt et al., 2015; Wang et al., 2016; Fujimoto et al., 2018; Barth-Maron et al., 2018).
Revised: Policy Gradient (DDPG) (Lillicrap et al., 2015) and their variants (Van Hasselt et al., 2015; Wang et al., 2016; Fujimoto et al., 2018; Barth-Maron et al., 2018).

Original: Another line of approaches uses a game-theoretic framework, and does not tie to speciﬁc families of RL algorithm.
Revised: Another line of approaches uses a game-theoretic framework, and does not tie to speciﬁc families of RL algorithm.

Original: The constrained problem is relaxed to a zero-sum game, whose equilibrium is solved by online learning (Agarwal et al., 2018).
Revised: The constrained problem is relaxed to a zero-sum game, whose equilibrium is solved by online learning (Agarwal et al., 2018).

Original: The game is played repeatedly, each time any
Revised: The game is played repeatedly, each time any

Original: RL algorithm can be used to ﬁnd a best response policy to play against a no-regret online learner.
Revised: RL algorithm can be used to ﬁnd a best response policy to play against a no-regret online learner.

Original: The mixed policy that uniformly distributed among all played policies can be shown to converge to an optimal policy of the constrained problem (Freund & Schapire, 1999; Abernethy et al., 2011).
Revised: The mixed policy that uniformly distributed among all played policies can be shown to converge to an optimal policy of the constrained problem (Freund & Schapire, 1999; Abernethy et al., 2011).

Original: Taking this approach, Le et al.
Revised: Taking this approach, Le et al.

Original: (2019) uses Lagrangian relaxation to solve the orthant constraint case, and Miryooseﬁ et al.
Revised: (2019) uses Lagrangian relaxation to solve the orthant constraint case, and Miryooseﬁ et al.

Original: (2019) uses conic duality to solve the convex constraint case.
Revised: (2019) uses conic duality to solve the convex constraint case.

Original: However, since the convergence is established by the no-regret property, the policy found by these methods requires randomization among policies found during the learning process, which limits their prevalence.
Revised: However, since the convergence is established by the no-regret property, the policy found by these methods requires randomization among policies found during the learning process, which limits their prevalence.

Revised: Different from the game-theoretic approaches, we reduce the C2RL to a distance minimization problem and propose a novel variant of Frank-Wolfe (FW) algorithm to solve it.

Revised: Our result builds on recent ﬁnding that the standard FW algorithm emerges as computing the equilibrium of a special convex-convave zero sum game (Abernethy & Wang, 2017).

Revised: This connects our approach with previous approaches from game-theoretic framework (Agarwal et al., 2018; Le et al., 2019; Miryooseﬁ et al., 2019).

Original: P RELIMINARIES
Revised: The main advantage of our reduction approach is that the convergence of FW algorithm does not rely on the no-regret property of an online learner. Hence there is no need to introduce extra hyper-parameters, such as learning rate of the online learner, and intuitively, we can eliminate unnecessary policies to achieve better sparsity. To do so, we extend Wolfe’s method for minimum norm point problem (Wolfe, 1976) to solve our distance minimization problem. Throughout the learning process, we maintain an active policy set, and constantly eliminate policies whose measurement vector are afﬁnely dependent of others.

Revised: Unlike norm function in Wolfe’s method, our objective function is not strongly convex.

Revised: Hence we cannot achieve the linear convergence of Wolfe’s method as shown in Lacoste-Julien & Jaggi (2015).

Revised: Instead, we analyze the complexity of our method based on techniques from Chakrabarty et al.

Revised: A theoretical comparison between our method and various approaches in constrained RL is provided in Table 1.

Original: A vector-valued Markov decision process can be identiﬁed by a tuple {S , A , β, P, c } , where S is a set of states, A is the set of actions and β is the initial state distribution.
Revised: A vector-valued Markov decision process can be identiﬁed by a tuple {S , A , β, P, c } , where S is a set of states, A is the set of actions and β is the initial state distribution.

Original: At the start of each episode, an initial state s 0 is drawn following the distribution β .
Revised: At the start of each episode, an initial state s 0 is drawn following the distribution β .

Original: Then, at each step t = 0 , 1 , . . .
Revised: Then, at each step t = 0 , 1 , . . .

Original: , the agent observes a state s t ∈ S and makes a decision to take an action a t .
Revised: , the agent observes a state s t ∈ S and makes a decision to take an action a t .

Original: After a t is chosen, at the next observation the state evolves to state s t +1 ∈ S with probability P ( s t +1 | s t , a t ) .
Revised: After a t is chosen, at the next observation the state evolves to state s t +1 ∈ S with probability P ( s t +1 | s t , a t ) .

Original: However, instead of a scalar reward, in our setting, the agent receives a m -dimensional vector c t ∈ R m that may implicitly contain measurements of reward, risk or violation of other constraints.
Revised: However, instead of a scalar reward, in our setting, the agent receives an m -dimensional vector c t ∈ R m that may implicitly contain measurements of reward, risk or violation of other constraints.

Original: The episode ends after a certain number of steps, called the horizon, or when a terminate state is reached.
Revised: The episode ends after a certain number of steps, called the horizon, or when a terminate state is reached.

Original: Actions are typically selected according to a policy π , where π ( s ) is a distribution over actions for any s ∈ S .
Revised: Actions are typically selected according to a policy π , where π ( s ) is a distribution over actions for any s ∈ S .

Original: Policies that take a single action for any state are deterministic policies , and can be identiﬁed by the mapping π : S (cid:55)→ A .
Revised: Policies that take a single action for any state are deterministic policies , and can be identiﬁed by the mapping π : S (cid:55)→ A .

Original: The set of all deterministic policies is denoted by Π .
Revised: The set of all deterministic policies is denoted by Π .

Original: For a discount factor γ ∈ [0 , 1) , the discounted long-term measurement vector of a policy π ∈ Π is deﬁned as
Revised: For a discount factor γ ∈ [0 , 1) , the discounted long-term measurement vector of a policy π ∈ Π is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over trajectories generated by the described random process.
Revised: where the expectation is over trajectories generated by the described random process.

Original: Unlike unconstrained setting, for a constrained RL problem, it is possible that all feasible policies are non-deterministic (see Appendix D for an example).
Revised: Unlike unconstrained setting, for a constrained RL problem, it is possible that all feasible policies are non-deterministic (see Appendix D for an example).

Original: This limits the usage of RL algorithms that search for deterministic policies in the setting of constrained RL problem.
Revised: This limits the usage of RL algorithms that search for deterministic policies in the setting of constrained RL problem.

Original: One workaround is to use mixed policies.
Revised: One workaround is to use mixed policies.

Original: For a set of policies U , a mixed policy is a distribution over U , and the set of all mixed policies over U is denoted by ∆( U ) .
Revised: For a set of policies U , a mixed policy is a distribution over U , and the set of all mixed policies over U is denoted by ∆( U ) .

Original: To execute a mixed policy µ ∈ ∆( U ) , we ﬁrst select a policy π ∈ U according to π ∼ µ ( π ) , and then execute π for the entire episode.
Revised: To execute a mixed policy µ ∈ ∆( U ) , we ﬁrst select a policy π ∈ U according to π ∼ µ ( π ) , and then execute π for the entire episode.

Original: Altman (1999) shows that any c ( · ) achievable can be achieved by some mixed deterministic policies µ ∈ ∆(Π) .
Revised: Altman (1999) shows that any c ( · ) achievable can be achieved by some mixed deterministic policies µ ∈ ∆(Π) .

Original: Therefore, though an off-shelves RL algorithm may not converge to any constraint-satisfying policy, it can be used as a subroutine to ﬁnd individual policies (possibly deterministic), and a randomization among these policies can converge to a feasible policy.
Revised: Therefore, though an off-shelves RL algorithm may not converge to any constraint-satisfying policy, it can be used as a subroutine to ﬁnd individual policies (possibly deterministic), and a randomization among these policies can converge to a feasible policy.

Original: The discounted long-term measurement vector of a mixed policy µ ∈ ∆(Π) is deﬁned similarly
Revised: The discounted long-term measurement vector of a mixed policy µ ∈ ∆(Π) is deﬁned similarly

Original: [Equation]
Revised: [Equation]

Original: For a mixed policy µ ∈ ∆( U ) , its active set is deﬁned to be the set of policies with non-zero weights
Revised: For a mixed policy µ ∈ ∆( U ) , its active set is deﬁned to be the set of policies with non-zero weights

Original: A := { π ∈ U| µ ( π ) > 0 } .
Revised: A := { π ∈ U| µ ( π ) > 0 } .

Original: The memory requirement of storing µ , is then proportion to the size of its active set.
Revised: The memory requirement of storing µ , is then proportional to the size of its active set.

Original: Since a mixed policy can be interpreted as a convex combination of policies in its active set, in the following, the term sparsity of a mixed policy refers to the sparsity of this combination.
Revised: Since a mixed policy can be interpreted as a convex combination of policies in its active set, in the following, the term sparsity of a mixed policy refers to the sparsity of this combination.

Original: Our learning problem, the convex constrained reinforcement learning (C2RL), is to ﬁnd a policy whose expected long-term measurement vector lies in a given convex set; i.e., for a given convex target set C ⊂ R m , our target is to ﬁnd µ ∗ such thatc ( µ ∗ )
Revised: Our learning problem, the convex constrained reinforcement learning (C2RL), is to ﬁnd a policy whose expected long-term measurement vector lies in a given convex set; i.e., for a given convex target set C ⊂ R m , our target is to ﬁnd [Equation] that

Original: ∈ Ω (C2RL) .
Revised: [Equation]

Original:  Any policy µ ∗ that satisﬁes c ( µ ∗ ) ∈ Ω is called a feasible policy , and a C2RL problem is feasible if there exists some feasible policies.
Revised: [Equation] Any policy µ ∗ that satisﬁes c ( µ ∗ ) ∈ Ω is called a feasible policy , and a C2RL problem is feasible if there exists some feasible policies.

Original: In the following, we assume the C2RL problem is feasible.
Revised: In the following, we assume the C2RL problem is feasible.

Original:  A PPROACH , A LGORITHM AND A NALYSIS
Revised:  A PPROACH , A LGORITHM AND A NALYSIS

Original: We now show how the C2RL (3) can be reduced to a distance minimization problem (7) between a polytope and a convex set.
Revised: We now show how the C2RL (3) can be reduced to a distance minimization problem (7) between a polytope and a convex set.

Original: A novel variant of Frank-Wolfe-type algorithm is then proposed to solve the distance minimization problem, followed by theoretic analysis about convergence and sparsity of the proposed method.
Revised: A novel variant of Frank-Wolfe-type algorithm is then proposed to solve the distance minimization problem, followed by theoretic analysis about convergence and sparsity of the proposed method.

Original: 4. R EDUCE C2RL TO A D ISTANCE M INIMIZATION P ROBLEM
Revised: 4. R EDUCE C2RL TO A D ISTANCE M INIMIZATION P ROBLEM

Original: Let || · || denotes the Euclidean norm.
Revised: Let ||·|| denote the Euclidean norm.

Original: For a convex set Ω ∈ R m , let
Revised: For a convex set Ω ∈ R m , let

Original: Proj Ω ( x ) ∈ arg min y ∈ Ω
Revised: Proj Ω ( x ) ∈ arg min y ∈ Ω

Original: ||
Revised: ||

Original: x − y
Revised: x − y

Original: || be the projection operator, and dist be the squared Euclidean distance function: f Ω ( x ) : = 12 || x − Proj Ω ( x ) || 2 .
Revised: || be the projection operator, and dist 2 ( x , Ω) : = 1 2 || x − Proj Ω ( x ) || 2 be half of the squared Euclidean distance function.

Original: For the problem 
Revised: Then we consider the problem to ﬁnd a policy whose measurement vector is closest to the target convex set,

Original: [Equation]
Revised: [Equation]

Original: (4) a policy µ ∗ ∈ ∆(Π) is deﬁned to be an optimal solution if it minimizes (4).
Revised: A policy µ ∗ ∈ ∆(Π) is deﬁned to be an optimal solution if it minimizes (4).

Original: Otherwise, the approximation error of µ ∈ ∆(Π) is deﬁned as
Revised: Otherwise, the approximation error of µ ∈ ∆(Π) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where µ ∗ is an optimal solution, and a policy is deﬁned to be an (cid:15) -approximate solution if its approximation error is no larger than (cid:15) .
Revised: where µ ∗ is an optimal solution, and a policy is deﬁned to be an (cid:15) -approximate solution if its approximation error is no larger than (cid:15) .

Original: When C2RL (3) is feasible, the equivalence of being optimal to (4) and being feasible to C2RL can be easily established.
Revised: When C2RL (3) is feasible, the equivalence of being optimal to (4) and being feasible to C2RL can be easily established.

Original: Since a feasible policy of C2RL problem lies inside Ω , it minimizes the non-negative dist  function, and hence is optimal to (4).
Revised: Since a feasible policy of C2RL problem lies inside Ω , it minimizes the non-negative dist 2 function, and hence is optimal to (4).

Original: Vice versa, any optimal solution to (4) lies inside Ω and is a feasible solution to C2RL.
Revised: Vice versa, any optimal solution to (4) lies inside Ω and is a feasible solution to C2RL.

Original: From a geometric perspective, let c (Π) :
Revised: From a geometric perspective, let c (Π) :

Original: = { c ( π ) | π ∈ Π } be the set of all values achievable by deterministic policies.
Revised: = { c ( π ) | π ∈ Π } be the set of all values achievable by deterministic policies.

Original: If the MDP has ﬁnite states and actions (though may be extremely large), then Π is ﬁnite as well, and hence c (Π) contains ﬁnitely many points in R m .
Revised: If the MDP has ﬁnite states and actions (though may be extremely large), then Π is ﬁnite as well, and hence c (Π) contains ﬁnitely many points in R m .

Original: Then the set of values achievable by mixed deterministic policiesis
Revised: Then the set of values achievable by mixed deterministic policies

Original: [Equation]
Revised: [Equation]

Original: is the convex hull of c (Π) ; i.e., c (∆(Π)) is a m -dimension polytope whose vertices are c (Π) .
Revised: is the convex hull of c (Π) ; i.e., c (∆(Π)) is a m -dimension polytope whose vertices are c (Π) .

Original: Therefore ﬁnding a policy whose value is closest to the target convex set (4) is equivalent to minimizing the distance between the polytope c (∆(Π)) and the convex set Ω
Revised: Therefore ﬁnding a policy whose value is closest to the target convex set (4) is equivalent to ﬁnd a point in the polytope c (∆(Π)) that is closest to the convex set Ω

Original: [Equation]
Revised: [Equation]

Original: minimization
Revised: To solve this constrained optimization problem, it might be tempting to consider projection methods.

Original: [Equation]
Revised: However, constructing a projection operator for c (∆(Π)) is non-trivial. For any given measurement vector, it is obscure how to modify a general RL algorithm to update the parameters such that the discounted expected measurement vector is closest to the given value. Therefore, projection-free methods are preferable for this task.

Original: Such constrained optimization problem over a polytope can be solved by the Frank-Wolfe algorithm (FW).
Revised: Frank-Wolfe (FW) algorithm does not require any projection operation, instead it uses a linear minimizer oracle. Intuitively, ﬁnding a linear minimizer is similar to the reward maximization process of what a general RL algorithm does. In section 4.3, we formalize this idea. We show that after simple modiﬁcations, any RL algorithm that maximizes a scalar reward can be used to construct such a linear minimizer oracle.

Original: Before getting into details of our method, we discussthe standard FW and variants of FW-type algorithms over polytope.
Revised: Before getting into details of the construction process, we discuss FW-type algorithms over polytope and its applications in the distance minimization problem (7).

Original: 4.  F RANK -W OLFE - TYPE A LGORITHMSOVER P OLYTOPE
Revised: 4. D ISTANCE M INIMIZATION BY F RANK -W OLFE - TYPE A LGORITHMS

Original: The Frank-Wolfe algorithm (FW) is a ﬁrst-order method to minimize a convex function f : P (cid:55)→ R over a compact and convex set P , with only access to a linear minimizer oracle .
Revised: The Frank-Wolfe algorithm (FW) is a ﬁrst-order method to minimize a convex function f : P (cid:55)→ R over a compact and convex set P , with only access to a linear minimizer oracle .

Original: When the feasible set is a polytope P := conv ( { s 1 , s 2 , . .
Revised: When the feasible set is a polytope P := conv ( { s 1 , s 2 , . .

Original: , s n } )
Revised: , s n } )

Original: ⊂ R m deﬁned as the convex hull of ﬁnitely many points, FW-type algorithms are discussed by (Lacoste-Julien & Jaggi, 2015) to optimize min x∈X f ( x ) using
Revised: ⊂ R m deﬁned as the convex hull of ﬁnitely many points, FW-type algorithms are discussed by Lacoste-Julien & Jaggi (2015) to optimize min x

Original: [Equation]
Revised: [Equation]

Original: The standard FW (Algorithm 2 in Appendix A.1) consists of making repeated calls to the linear minimizer oracle (line 2), followed by a convex averaging step of the current iterate and the oracle’s output (line 3). Though the standard FW can be applied to solve the distance minimization problem (7), an (cid:15) -approximate solution found by standard FW requires maintaining of an active set of policies of size O
Revised: f (

Original: (1 /(cid:15) ) , which is memory demanding. Variants of FW-type algorithms has been studied. In particular, when the objective function is the Euclidean distance to a ﬁxed point, Wolfe’s method for minimum norm point has been proposed (Wolfe, 1976).
Revised: [Equation]

Original: In Wolfe’s method (Algorithm 3 in Appendix A.2), the loop in FW is called a major cycle , and the convex averaging step (line 3) is replaced by a weight optimization process which is called a minor cycle .
Revised: ) using

Original: Wolfe’s method maintains an active set S , and the current point can represented by a sparse combination of points in the active set. The minor cycles maintain S to be an afﬁnely independent set such that the afﬁne minimizer is inside S t , which Wolfe calls corrals . Recall an afﬁne minimizer is deﬁned as arg min s ∈ aff ( S ) || s || 2 , where aff ( S )
Revised: Oracle

Original: [Equation]
Revised: [Equation]

Original: ∈S α T z x , (cid:80) z ∈S α z = 1 } is the afﬁne hull formed by S . Since the active set is afﬁnely independent, the number of active atoms is at most m + 1 at any time.
Revised: The standard FW (Algorithm 2 in Appendix A.1) consists of making repeated calls to the linear minimizer oracle to ﬁnd an improving point s , followed by a convex averaging step of the current iterate x t − 1 and the oracle’s output s .

Original: Wolfe’s method is shown to strictly decrease the approximation error between two major cycles. In the reminder of this section, we propose a novel variant of FW type algorithm to solve the distance minimization problem (7) between a polytope and a convex set.
Revised: If we have already constructed a RL oracle ( λ ) that outputs a policy π ∈ arg min π ∈ Π λ T c ( π ) together with its measurement vector c ( π ) , then the distance minimizing problem (7) can be solved with standard FW by using [Equation]

Original: At the core of our approach, we propose a novel variant of Frank-Wolfe algorithm, to minimize the distance between the polytope of all values achievable by mixed deterministic policy c (∆(Π)) and the target convex set Ω . The proposed method can be seen as an extension of Wolfe’s method, whose
Revised: Input.

Original: Initialize
Revised: Initialize.

Revised: Random policy π , value x = c ( π ) , active sets S p :=

Revised: [ π ] , S c :=

Revised: [ x ] and weight λ = [1] .

Original: Output
Revised: Output.

Revised: Mixed policy µ and its value c ( µ ) s.t. c ( µ ) minimizes the distance to the target set Ω .

Original: 1: while true do //
Revised: 1: while true do //

Original: Major cycle 2: ω ← Proj Ω ( x ) 3: π, c ( π ) ← RL Oracle ( x − ω ) //
Revised: Major cycle 2: ω ← Proj Ω ( x ) 3: π, c ( π ) ← RL Oracle ( x − ω ) //

Original: Potential improving point 4: if ( x − ω ) T ( x − c ( π )) ≤ (cid:15) then break 5: if S c ∪ { c ( π ) } is afﬁnely independent then S c ← S c ∪ { c ( π ) } , S p ← S p ∪ { π } 6: while true do // Minor cycle 7: y , α ← AffineMinimizer ( S c , ω ) //
Revised: Potential improving point 4: if ( x − ω ) T ( x − c ( π )) ≤ (cid:15) then break 5: if S c ∪ { c ( π ) } is afﬁnely independent then S c ← S c ∪ { c ( π ) } , S p ← S p ∪ { π } 6: while true do // Minor cycle 7: y , α ← AffineMinimizer ( S c , ω ) //

Original: y = arg min s ∈ aff ( S c )
Revised: y = arg min s ∈ aff ( S c )

Original: || s − ω
Revised: || s − ω

Original: ||8: if α s > 0 for all s then break // y ∈ conv ( S c ) 9: //
Revised: ||8: if α s > 0 for all s then break // y ∈ conv ( S c ) 9: //

Original: If y / ∈ conv ( S c ) , then update y to the intersection of conv ( S c ) and segment joining x and y .
Revised: If y / ∈ conv ( S c ) , then update y to the intersection of conv ( S c ) and segment joining x and y .

Original: Then remove points in S c unnecessary for describing y .
Revised: Then remove points in S c unnecessary for describing y .

Original: 10: θ ← min i : α
Revised: 10: θ ← min i : α

Original: i ≤λ
Revised: i ≤λ

Original: i
Revised: i

Original: λ
Revised: λ

Original: i − α
Revised: i − α

Original: i // Recall λ satisﬁes x = (cid:80) s ∈S c λ s s 11:
Revised: i // Recall λ satisﬁes x = (cid:80) s ∈S c λ s s 11:

Original: y ← θ y + (1 − θ )
Revised: y ← θ y + (1 − θ )

Original: x , λ i = θα i + (1 − θ )
Revised: x , λ i = θα i + (1 − θ )

Original: λ i 12:
Revised: λ i 12:

Original: S c ← { c ( π i ) | c ( π i ) ∈
Revised: S c ← { c ( π i ) | c ( π i ) ∈

Original: S c
Revised: S c

Original: and λ i > 0 } , S p ← { π i | π i ∈ S p and λ i > 0 } 13: end while 14:
Revised: and λ i > 0 } , S p ← { π i | π i ∈ S p and λ i > 0 } 13: end while 14:

Original: Update µ ← (cid:80) π ∈S p λ π π , x ← y , λ ← α .
Revised: Update µ ← (cid:80) π ∈S p λ π π , x ← y , λ ← α .

Original: 15: end while 16: return µ , c ( µ ) ← xobjective is to minimize the distance to a ﬁxed point over a polytope.
Revised: 15: end while 16: return µ , c ( µ ) ← x to ﬁnd an improving policy and its measurement vector. For η t := 2 t +2 , the convex averaging steps [Equation]

Original: Using any RL algorithm as an oracle, our method minimizes the distance to a target convex set over a polytope, and the  found point is maintained as a sparse combination of vertices of  the polytope.
Revised: then maintain the mixed policy, and the corresponding measurement vector, respectively. However, after T rounds of iteration, the µ t found has an active set containing up to T individual polices, and is not sparse enough. If neural networks are used to parameterize the policy, that requires storing T copies of parameters for the individual network, which is unaffordable for largescale usage. To ﬁnd even more sparse policies, we turn to variants of FW-type algorithms.

Original: We shall deﬁne some basic notions.
Revised: In particular, Wolfe’s method for minimum norm point in a polytope (Wolfe, 1976; De Loera et al., 2018). In Wolfe’s method (Algorithm 3 in Appendix A.2), the loop in FW is called a major cycle , and the convex averaging step is replaced by a weight optimization process, called minor cycle . Wolfe’s method maintains an active set S , and the current point can be represented by a sparse combination of points in the active set. The minor cycles maintain S to be an afﬁnely independent set such that the afﬁne minimizer is inside S t , which Wolfe calls corrals .

Original: For a set of points in S ⊂ R m , and x ∈ R m , we deﬁne afﬁne minimizer with respect to x as arg min s ∈ aff ( S ) 
Revised: Recall an afﬁne minimizer is deﬁned as arg min s ∈ aff ( S ) || s || 2 , where aff ( S ) : = { y | y =

Revised: (cid:80) z ∈S α T

Revised: z x , (cid:80)

Revised: z

Revised: ∈S α z = 1 } is the afﬁne hull formed by S .

Revised: Since the active set is afﬁnely independent, the number of active atoms is at most m + 1 at any time.

Revised: Wolfe’s method is shown to strictly decrease the approximation error between two major cycles.

Revised: 4. O UR M AIN A LGORITHM

Original: ||
Revised: The main obstacle to apply Wolfe’s method to our distance minimization problem (7) is that the objective function in Wolfe’s method is the norm function. However, in our problem, the objective function is the distance function to a convex set. Unlike the norm function, the distance function to a convex set is not strongly convex and afﬁne minimizer is ill-deﬁned with respect to a convex set. To tackle these problems, we modify the Wolfe’s method.

Revised: At the core of our new variant of FW algorithm, we add a projection step to Wolfe’s method.

Revised: Projection Step In each major cycle, we minimize the distance to a projected point ω :=

Original: s − x || 2 .
Revised: Proj Ω ( x ) .

Revised: Intuitively, since the distance to the convex set is upper bounded by the distance to this projected point ω , if the distance to ω converges, so does the distance to the target convex set.

Revised: Formally, for a set of points S ⊂ R m , and a point x ∈ R m , we extend the deﬁnition of an afﬁne minimizer to deﬁne afﬁne minimizer with respect to x as arg min s ∈ aff ( S ) || s − x || 2 .

Original: For x being the afﬁne minimizer of S with respect to ω , the  afﬁne minimizer property gives 
Revised: For x being the afﬁne minimizer of S with respect to ω , the extended afﬁne minimizer property gives Given ω, [Equation] v ∈ aff

Revised: [Equation]

Revised: S ) , ( v − x

Original: [Equation]
Revised: [Equation]

Revised: T

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: ω

Revised: [Equation]

Original: {+= 
 (Extended afﬁne minimizer property) [Equation]+} Similar to Wolfe’s method, our C2RL method (Algorithm 1) contains an outer loop (called major cycle) to ﬁnd improving policies and their measurement vectors, and an inner loop (called minor cycle) to maintain the afﬁnely independent property of the active set S c .
Revised: {+= 
 (Extended afﬁne minimizer property) [Equation]+} Similar to Wolfe’s method, our C2RL method (Algo. 1) contains an outer loop (called major cycle) to ﬁnd improving policies and their measurement vectors, and an inner loop (called minor cycle) to maintain the afﬁnely independent property of the active set S c .

Original: At the start of each major cycle step, the S c is an afﬁnely independent set.
Revised: At the start of each major cycle step, the S c is an afﬁnely independent set.

Original: Then, the RL oracle (deﬁned in (10)) ﬁnds a potential improving policy π ∈ U , and its long-term measurement vector c ( π ) .
Revised: Then, the RL oracle (deﬁned in (15)) ﬁnds a potential improving policy π ∈ U , and its long-term measurement vector c ( π ) .

Original: If the c ( π ) does not get strictly closer to the ω :=
Revised: If the c ( π ) does not get strictly closer to the ω :=

Original: Proj ( x ) , then we are done, and x is the optimal value.
Revised: Proj ( x ) , then we are done, and x is the optimal value.

Original: Otherwise, the c ( π ) is added into the active set, and the minor cycle is run to eliminate policies whose measurement vectors are afﬁnely dependent.
Revised: Otherwise, the c ( π ) is added into the active set, and the minor cycle is run to eliminate policies whose measurement vectors are afﬁnely dependent.

Revised: Line 6 to line 13 contains the minor cycle, which is the same as the original Wolfe’s method (except in line 6, we ﬁnd afﬁne minimizer with respect to ω ).

Original: The elimination is executed as a series of afﬁne projections.
Revised: The elimination is executed as a series of afﬁne projections.

Original: The minor cycle terminates if active set S c is afﬁnely independent.
Revised: The minor cycle terminates if active set S c is afﬁnely independent.

Original: Though the interleaving of major and minor cycles oscillate the size of active set S c , the minor cycles keep |S c | an afﬁne independent set, and is terminated whenever S c contains a single element.
Revised: Though the interleaving of major and minor cycles oscillate the size of active set S c , the minor cycles keep |S c | an afﬁnely independent set, and is terminated whenever S c contains a single element.

Original: Therefore at the start of any major cycle, the size of the active set satisﬁes |S c | ∈ [0 , m + 1] .
Revised: Therefore at the start of any major cycle, the size of the active set satisﬁes |S c | ∈ [0 , m + 1] .

Revised: More background about the minor cycle in Wolfe’s method is provided in Appendix A.2.

Original: The construction of our RL oracle can use any off-the-shelf RL algorithm that minimizes a scalar reward.
Revised: The construction of our RL oracle can use any off-the-shelf RL algorithm that maximizes a scalar reward.

Original: For any given λ ∈ R m , we deﬁne any algorithm that ﬁnds a policy minimizing the linear function λ T c ( · ) as a RL oracle , that is
Revised: For any given λ ∈ R m , we deﬁne any algorithm that ﬁnds a policy minimizing the linear function λ T c ( · ) as a RL oracle , that is

Original: [Equation]
Revised: [Equation]

Revised: Recall that standard RL algorithm receives a scalar reward after each state transition, instead of the long-term measurement vector c ( π ) ∈ R m .

Original: The construction of RL oracle relies on the following linear property 
Revised: We then use the following linear property to reformulate the right hand side of (12) to a standard RL problem

Original: [Equation]
Revised: [Equation]

Original: which shows that a linear minimizer of λ T c ( · ) can be solved by training any RL algorithms to maximize the  scalar reward  ( − λ T c t ) .
Revised: This shows that if we consider the Markov decision process with the same state, action, and transition probability, and construct a scalar reward r := ( − λ T c t ) , then any policy that maximizes the expected r is a linear minimizer of (12).

Original: Therefore any RL algorithm that  suits the underlying problem can be used to construct a RL oracle.
Revised: Therefore any RL algorithm that best suits the underlying problems can be used to construct a RL oracle.

Original: Certifying constraint satisfaction amounts to evaluate the measurement vector of the current policy.
Revised: Certifying constraint satisfaction amounts to evaluate the measurement vector of the current policy.

Original: This is handy in online settings, where simulations can be used to evaluate the measurement vector of the policy directly.
Revised: This is handy in online settings, where simulations can be used to evaluate the measurement vector of the policy directly.

Original: Otherwise, in batch settings, various off-policy evaluation methods, such as importance sampling (Precup, 2000; Precup et al., 2001) or doubly robust (Jiang & Li, 2016;
Revised: Otherwise, in batch settings, various off-policy evaluation methods, such as importance sampling (Precup, 2000; Precup et al., 2001) or doubly robust (Jiang & Li, 2016;

Original: Dud´ık et al., 2011), can be used to evaluate the policy.
Revised: Dud´ık et al., 2011), can be used to evaluate the policy.

Original: [Equation]
Revised: [Equation]

Original: With a bit abuse of notation, we assume a RL Oracle returns a policy as well as the measurement vector of the policy.
Revised: [Equation] To simplify notation, we assume a RL Oracle returns a policy as well as its measurement vector [Equation]

Original:  The process AffineMinimizer ( S, x ) returns the ( y , α ) the afﬁne minimizer of S with respect to x where y is the afﬁne minimizer and α := { α s |∀ s ∈ S c } is the set of coefﬁcient expressing y as an afﬁne combination of points in S , that is y
Revised: Finding Extended Afﬁne Minimizer The process AffineMinimizer ( S, x ) returns the ( y , α ) the afﬁne minimizer of S with respect to x where y is the afﬁne minimizer and α := { α s |∀ s ∈ S c } is the set of coefﬁcient expressing y as an afﬁne combination of points in S , that is y

Original: = (cid:80) s
Revised: = (cid:80) s

Original: ∈S c α s s , where α s is the weight associated with s .
Revised: ∈S c α s s , where α s is the weight associated with s .

Original: The process AffineMinimizer ( S, x ) can be straightforwardly implemented using linear algebra.
Revised: The process AffineMinimizer ( S, x ) can be straightforwardly implemented using linear algebra.

Original: Wolfe (1976) also provides a more efﬁcient implementation that uses a triangular array representation of the active set.
Revised: Wolfe (1976) also provides a more efﬁcient implementation that uses a triangular array representation of the active set.

Original: 4. C ONVERGENCE AND S PARSITY
Revised: 4. C ONVERGENCE AND S PARSITY

Original:  We ﬁrst show that approximation error of Algorithm 1 strictly decreases among any two major cycle steps (Thm. 4.1). Then, the convergence is studied. We show that after O (1 /(cid:15) ) rounds of iteration,
Revised: In this section, we analyze the convergence and complexity of the proposed C2RL method (Algo. 1). We ﬁrst show that approximation error of C2RL strictly decreases between any two major cycle steps and it converges in O (1 /t ) rate.

Original: Algorithm 1 returns a policy µ ∈ ∆(Π) satisfying err ( µ ) ≤ (cid:15) (Thm. 4.2).
Revised: Then we show our method ensures convergence of arbitrary RL algorithm, including those searching for deterministic policies.

Original: That is, the Algorithm (1) converges in O (1 /t ) rate. Finally we show that maintaining an active policy set of m + 1 is worst case optimal to ensure the convergence of arbitrary RL algorithm, especially those searching for deterministic policies (Thm. 4.5.1).
Revised: Moreover, concerning the memory complexity, we show that maintaining an active policy set of m +1 is worst case optimal to ensure the convergence of arbitrary RL algorithm. Therefore, the proposed C2RL indeed achieves the optimal sparsity for the found policy, making it favorable for large-scale usage.

Original: A major obstacle to analyze the proposed Algorithm 1 is the mixture of  major and minor cycles, which makes the sizes of the active sets S c and S p ﬂuctuate during iterations.
Revised: The main difference between the convergence analysis of C2RL and Wolfe’s method is the addition of the projection step. Intuitively, at each major step, if we are making a signiﬁcant progress toward the projected point, then the distance to the convex set is decreased by at least the same amount. Time Complexity.

Original: In our analysis, we consider the approximation error as deﬁned in (5).
Revised: In our analysis, we consider the approximation error as deﬁned in (5).

Original: We use superscript t to denote the variable in t -th major cycle before executing any minor cycle.
Revised: We use superscript t to denote the variable in t -th major cycle before executing any minor cycle.

Original: To simplify notions, we let x t := c ( µ t ) and s t :
Revised: To simplify notions, we let x t := c ( µ t ) and s t :

Original: = c ( π t ) .
Revised: = c ( π t ) .

Original: Since y is used only within each major cycle, when discussing one step with t ﬁxed,we let y i denote the afﬁne minimizer found in i -th minor cycle (line 6 of Algorithm 1).
Revised: When discussing one step with t ﬁxed, let y i denote the afﬁne minimizer found in i -th minor cycle (line 6 of Algo. 1).

Original: We ﬁrst show that the Algorithm 1 strictly reduces approximation error between two calls of the RL oracle.
Revised: We ﬁrst show that the C2RL method strictly reduces approximation error between two calls of the RL oracle.

Original: Theorem 4.1 (Approximation Error Strictly Decreases) .
Revised: Theorem 4.1 (Approximation Error Strictly Decreases) .

Original: For any non-terminal step t , we have err ( µ t +1 ) <
Revised: For any non-terminal step t , we have err ( µ t +1 ) <

Original: err ( µ t ) .
Revised: err ( µ t ) .

Original: That is, the measurement vector of µ t found by Algorithm 1 gets strictly closer to the convex set Ω after major cycle step.
Revised: That is, the measurement vector of µ t found by the C2RL method gets strictly closer to the convex set Ω after major cycle step.

Original: The proof is provided in Appendix B.
Revised: The proof is provided in Appendix B.

Original: The idea is to consider the distance between x t and ω t .
Revised: The idea is to consider the distance between x t and ω t .

Original: When the major cycle has no minor cycle, the non-terminal condition and the afﬁne minimizer property implies dist  ( x t +1 , ω t ) < dist  ( x t , ω t ) .
Revised: When the major cycle has no minor cycle, the non-terminal condition and the afﬁne minimizer property implies dist 2 ( x t +1 , ω t ) < dist 2 ( x t , ω t ) .

Original: Otherwise we show that the ﬁrst minor cycle strictly reduces the dist  ( x t , ω t ) by moving along the segment joining x and y , and the subsequent minor cycle cannot increase it.
Revised: Otherwise we show that the ﬁrst minor cycle strictly reduces the dist 2 ( x t , ω t ) by moving along the segment joining x and y , and the subsequent minor cycle cannot increase it.

Original: Since ω t ∈ Ω , we conclude err ( x t +1 ) ≤ dist  ( x t +1 , ω t ) < dist  ( x t , ω t ) = err ( x t ) , and the approximation error strictly decreases.
Revised: Since ω t ∈ Ω , we conclude err ( x t +1 ) ≤ dist 2 ( x t +1 , ω t ) < dist 2 ( x t , ω t ) = err ( x t ) , and the approximation error strictly decreases.

Original: Given the approximation error strictly decreases, Wolfe’s method for minimum norm point can be shown to terminate ﬁnitely (Wolfe, 1976).
Revised: Given the approximation error strictly decreases, Wolfe’s method for minimum norm point can be shown to terminate ﬁnitely (Wolfe, 1976).

Original: However, this ﬁnitely terminating property does not hold for our algorithm.
Revised: However, this ﬁnitely terminating property does not hold for our algorithm.

Original: Since a changed ω t may yield a lower distance to the same active set S tc , the active set may stay unchanged across major cycles (see Figure 2 Middle for an example).
Revised: Since a changed ω t may yield a lower distance to the same active set S tc , the active set may stay unchanged across major cycles (see Figure 2 Middle for an example).

Original: We establish the convergencerate of our method in the following theorem.
Revised: Therefore we establish the convergence of the C2RL method by the following theorem.

Original: Theorem 4.2 (Convergence in Approximation Error) .
Revised: Theorem 4.2 (Convergence in Approximation Error) .

Original: For t ≥ 1 , the mixed policy µ t found by 
Revised: For t ≥ 1 , the mixed policy µ t found by the

Original: Algorithm 1 satisﬁes
Revised: C2RL method satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where Q := max µ ∈ ∆( U )
Revised: where Q := max µ ∈ ∆( U )

Original: ||
Revised: ||

Original: c ( µ ) || is the maximum norm of a measurement vector.
Revised: c ( µ ) || is the maximum norm of a measurement vector.

Original: The proof is provided in Appendix C, which relies on the following two lemmas.
Revised: The proof is provided in Appendix C, which relies on the following two lemmas.

Original: We brieﬂy discuss the main idea here.
Revised: We brieﬂy discuss the main idea here.

Original: Deﬁne major cycle steps with at most one minor cycle as ”non-drop step” and major cycle steps with more than one minor cycles as ”drop steps”.
Revised: Deﬁne major cycle steps with at most one minor cycle as ”non-drop step” and major cycle steps with more than one minor cycles as ”drop steps”.

Original: We show that in each non-drop step, Algorithm 1 is guaranteed to make enough progress in the following lemma.
Revised: We show that in each non-drop step, Algorithm 1 is guaranteed to make enough progress in the following lemma.

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: For a non-drop step, we have err [Equation] t [Equation]
Revised: For a non-drop step in C2RL method, we have err ( µ t ) − err ( µ t +1 ) ≥ err 2 ( µ t ) / 8 Q 2 .

Original: [Equation] [Figure] Goal 0. However, does lemma
Revised: Though this does not hold for drop steps, we can bound the frequency of drop steps by the following.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: After t major cycle steps of Algorithm 1, the number of drop steps is less than t/ 2 .
Revised: After t major cycle steps of C2RL method, the number of drop steps is less than t/ 2 .

Original: Since the approximation error strictly decreases (Thm. 4.1), and in more than half of the major cycles steps Algorithm 1 makes signiﬁcantly progress.
Revised: Since the approximation error strictly decreases (Thm. 4.1), and in more than half of the major cycles steps, the C2RL method makes signiﬁcantly progress.

Original: The Thm.
Revised: The Thm.

Original: (4.2) can then be proved using an induction argument (Appendix C).
Revised: (4.2) can then be proved using an induction argument (Appendix C).

Revised: Convergence with Arbitrary RL Algo.

Revised: The convergence of the C2RL method when used with

Revised: RL algorithms that search for deterministic policies, such as DQN, DDPG and variants, is indeed straightforward.

Revised: In (8), though each time the oracle yields a vertex, the FW-type algorithms indeed optimize over the polytope formed by these vertices.

Revised: Then since citetaltman1999constrained shows that any c ( · ) achievable can be achieved by some mixed deterministic policies, we conclude that if the underlying problem is feasible, then our C2RL method is able to ﬁnd a feasible policy.

Original:  We then discuss the sparsity of mixed policy for constrained RL problem, and we give an constructive proof in Appendix D.
Revised: Memory Complexity We then discuss the sparsity of mixed policy for constrained RL problem. We give a constructive proof in Appendix D to show that to ensure convergence for RL algorithms that search for deterministic policies, storing m + 1 policies is required in the worst case. [Figure] Goal 0.

Original: Theorem 4.5 (Memory Complexity Bound) .
Revised: Theorem 4.5 (Memory Complexity Bound) .

Original: For an constrained RL problem with m -dimensional measurement vector, in the worst case, a mixed policy needs to randomize among m + 1 individual policies to ensure convergence of RL oracles that search for deterministic policies.
Revised: For an constrained RL problem with m -dimensional measurement vector, in the worst case, a mixed policy needs to randomize among m + 1 individual policies to ensure convergence of RL oracles that search for deterministic policies.

Revised: Since the minor cycles in the C2RL method eliminate policies with afﬁnely dependent measurement vectors, after the termination of minor cycles, the size of the active set is at most m + 1 .

Revised: That is, the policy found by the C2RL method requires randomization among no more than m + 1 individual policies.

Revised: Therefore the proposed C2RL indeed achieves the optimal sparsity in the worst case, making it favorable for large-scale usage.

Original: Corollary 4.5.1.
Revised: Corollary 4.5.1.

Original: The Algorithm 1 that randomize among at most m + 1 policies is worst-case optimal to ensure convergence of any RL oracle.
Revised: The C2RL method that randomizes among at most m + 1 policies is worst-case optimal to ensure convergence of any RL oracle.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the performance of C2RL in a grid-world navigation task (Figure 1, and demonstrate its ability to efﬁciently ﬁnd sparse policy.
Revised: We evaluate the performance of C2RL in a grid-world navigation task (Fig. 1), and demonstrate its ability to efﬁciently ﬁnd sparse policy.

Revised: In this Risky Mars Rover environment, the agent is required to navigate from the starting point to the goal point, by moving to one of the four neighborhood cells at each step.

Revised: The episodes terminate when the goal point is reached or after 300 steps.

Revised: To enforce robustness, we add a risky area to indicate the dangerous states.

Original: The proposed method is compared with approachabilitybased policy optimization (ApproPO) (Miryooseﬁ et al., 2019) and  with reward constrained policy optimization (RCPO)
Revised: The agent receives a measurement vector to indicate the steps it takes ( 0 . 1 for every step), and whether it stays in the risky area ( 0 . 1 for every risky step, and 0 otherwise), with discount factor γ = 0 . We constrain the agent to reach the goal point with expected cumulative steps measure within 1 . 1 and the expected cumulative risky steps within 0 .

Revised: Note that by design, the shortest path from the starting point to the goal point does not satisfy the constraint.

Revised: This is common in practice, as robustness typically evolves trade-off between the reward and the constraint satisfaction.

Revised: The proposed C2RL method is compared with approachability-based policy optimization (ApproPO) (Miryooseﬁ et al., 2019) and with reward constrained policy optimization (RCPO)

Original: (Tessler et al., 2018).
Revised: (Tessler et al., 2018).

Original: ApproPO solves the same convex constrained RL problem by using an RL oracle to play against a no-regret online learner (Hazan et al., 2008; Zinkevich, 2003).
Revised: ApproPO solves the same convex constrained RL problem by using an RL oracle to play against a no-regret online learner (Hazan et al., 2008; Zinkevich, 2003).

Original: Since ApproPO and C2RL both use a RL oracle, ApproPO is a natural baseline to be compared with our method.
Revised: Since ApproPO and C2RL both use a RL oracle, ApproPO is a natural baseline to be compared with our method.

Original: Besides, we also compare with RCPO, which takes a Lagrangian approach to incorporate the constraints as a penalty signal into the reward.
Revised: Besides, we also compare with RCPO, which takes a Lagrangian approach to incorporate the constraints as a penalty signal into the reward.

Original: Using an advantage actor critic (A2C)
Revised: Using an advantage actor critic (A2C)

Original: Mnih et al.
Revised: Mnih et al.

Original: (2016), RCPO has been shown to converge to a ﬁxed point.
Revised: RCPO has been shown to converge to a ﬁxed point.

Original: For a fair comparison, C2RL and ApproPO uses an A2C agent as the RL oracle, with the same hyperparameter as used in RCPO.
Revised: For a fair comparison, C2RL and ApproPO uses an A2C agent as the RL oracle, with the same hyperparameter as used in RCPO.

Original: The approximation errors are compared after training for the same number of samples.
Revised: The approximation errors are compared after training for the same number of samples.

Original: We evaluate the  method on a Risky Mars Rover grid-world environment.
Revised: Note that the C2RL method does not introduce any extra hyper-parameter.

Original: In this environment, the agent navigates from the starting point to the goal. The episodes terminate when the goal point is reached or after 300 steps. The environment contains a risky area. The agent receives a measurement vector to indicate the steps it takes ( 0 .
Revised: For ApproPO and RCPO, they require extra hyper-parameter for the initialization and learning rate of a variable equivalent to our λ in the outer loop.

Original: 1 for every step), and whether it stays in the risky region ( 0 .for every risky step, and 0 otherwise), with discount factor γ = 0 . We constrain the agent to reach the goal point with expected cumulative steps measure within 1 . 1 and  the expected cumulative risky steps within 0 .
Revised: This is because our approach does not rely on the online learning framework, and therefore there is no need to tune the initialization and learning rate for our λ and ease the usage.

Original: We ﬁrst showcase the consequences of our theoretical results using an optimal RL oracle .
Revised: We ﬁrst showcase the consequences of our theoretical results using an optimal RL oracle .

Original: For any x ∈ R m , an optimal policy can be easily found via Dijkstra’s algorithm.
Revised: For any x ∈ R m , an optimal policy can be easily found via Dijkstra’s algorithm.

Original: If multiple optimal paths exist, one is randomly picked to form a deterministic policy. 
Revised: If multiple optimal paths exist, one is randomly picked to form a deterministic policy. [Figure]

Original: Using this as an optimal RL oracle, the convergence property of C2RL and ApproPo are compared.
Revised: Using this as an optimal RL oracle, the convergence property of C2RL and ApproPo are compared.

Original: Figure 2{+
+} Middle shows the value of policies c ( µ t ) found after each call to the oracle.
Revised: Figure {+
+} Middle shows the value of policies c ( µ t ) found after each call to the oracle.

Original: In Figure 2 Right, when approaching the boundary of the feasible set, the iterations of approachability-based methods start to zigzag.

Original: Lacoste-

Original: [Figure]
Revised: In Figure

Original: Julien & Jaggi (2015) discuss this zig-zagging problem in the context of FW-type algorithms and shows that, intuitively, reducing weights from ”bad atoms” alleviates this problem, such as away-step FW (Wolfe, 1970; Jaggi, 2013), pairwise FW (Mitchell et al., 1974), and Wolfe’s method (Wolfe, 1976).
Revised: Right, when approaching the boundary of the feasible set, the iterations of approachability-based methods start to zigzag.

Original: Since C2RL contains a minor cycle to re-optimize the weights among the active set, C2RL progresses quickly to reach the exact optimal solution.
Revised: Since C2RL contains a minor cycle to re-optimize the weights among the active set, C2RL progresses quickly to reach the exact optimal solution.

Original: In Figure 3 Left, the approximation error is shown for 300 calls of the optimal RL oracle.
Revised: In Figure 3 Left, the approximation error is shown for 300 calls of the optimal RL oracle.

Original: We then compare C2RL, ApproPO and RCPO using the same A2C agent. ApproPO has a parameter to project the convex constraint to a cone constraint and both ApproPO and RCPO has an learning rate in the outer loop. We use the value as suggested by their paper.
Revised: We then compare C2RL, ApproPO and RCPO using the same A2C agent (details of the model structures and hyper-parameters are provided in Appendix E).

Original: We run each algorithm fortimes, and each run for a maximum of 100 thousands of samples.
Revised: We run each algorithm for 50 times, and each run for a maximum of 100 thousands of samples.

Original: The mean and standard deviation of the results are presented in Figure 3.
Revised: The mean and standard deviation of the results are presented in Figure 3.

Original: The original paper of ApproPO suggests using a cache to save memory, and the memory requirement of this variant is also presented.
Revised: The original paper of ApproPO suggests using a cache to save memory, and the memory requirement of this variant is also presented.

Original: Figure 3 demonstrates that C2RL converges to an optimal policy faster than previous methods, and a sparse combination of individual policies are maintained throughout the iteration process.
Revised: Figure 3 demonstrates that C2RL converges to an optimal policy faster than previous methods, and a sparse combination of individual policies is maintained throughout the iteration process.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce C2RL, an algorithm to solve RL problems under  convex constraints.
Revised: In this paper, we introduce C2RL, an algorithm to solve RL problems under orthant or convex constraints.

Original: Our method reduces the constrained RL problem to a distance minimization problem, and a novel variant of Frank-Wolfe type algorithm is proposed to solve this.
Revised: Our method reduces the constrained RL problem to a distance minimization problem, and a novel variant of Frank-Wolfe type algorithm is proposed to solve this.

Original: Our method comes with rigorous theoretical guarantees.
Revised: Our method comes with rigorous theoretical guarantees and does not introduce any extra hyper-parameter.

Original: To ﬁnd an (cid:15) -approximation solution, C2RL takes O (1 /(cid:15) )
Revised: To ﬁnd an (cid:15) -approximation solution, C2RL takes O (1 /(cid:15) )

Original: calls of any RL oracle, which improves from previous O (1 /(cid:15) 2 ) .
Revised: calls of any RL oracle and ensures convergence to work with arbitrary RL algorithm.

Original: Moreover, C2RL strictly reduces the approximation error between consecutive calls of RL oracle. Beside the improved performance, for m -dimensional constraints, the memory requirement is reduce from storing inﬁnitely many policies ( O (1 /(cid:15)2 ) ) to storing at most constantly many ( m + 1 ) polices.
Revised: Moreover, C2RL strictly reduces the approximation error between consecutive calls of RL oracle, and for m -dimensional constraints, the memory requirement is reduced from storing inﬁnitely many policies ( O (1 /(cid:15) ) ) to storing at most constantly many ( m +1 ) polices.

Original: We further show that the constant is worst-case optimal to ensure the convergence of any RL algorithms that search for deterministic policies.
Revised: We further show that the constant is worst-case optimal to ensure the convergence for RL algorithms that search for deterministic policies.

Original: Experimentally, we demonstrate that  C2RL  ﬁnds sparse solution efﬁciently, and outperforms previous methods.
Revised: Experimentally, we demonstrate that the proposed C2RL method ﬁnds sparse solution efﬁciently, and outperforms previous methods.

Original: A M ORE ON F RANK -W OLFE - TYPE A LGORITHMSA.
Revised: A M ORE ON F RANK -W OLFE - TYPE A LGORITHMS

Revised: A.

Original: S TANDARD F RANK -W OLFE
Revised: S TANDARD F RANK -W OLFE

Original:  LGORITHM
Revised: A LGORITHM

Original: Input: obj.
Revised: Input: obj.

Original: f : Y (cid:55)→ R , oracle O ( · ) , init.
Revised: f : Y (cid:55)→ R , oracle O ( · ) , init.

Original: w 0 ∈ Y 1: for t=1, 2, 3 ..., T do 2: v t ← Oracle ( ∇ f ( w t − 1 )) =
Revised: x 0 ∈ Y 1: for t=1, 2, 3 ..., T do 2: s ← Oracle ( ∇ f ( x t − 1 )) =

Original: arg min v ∈Y v T ∇ f ( w t − 1 ) 3: w t ← (1 − η t w t − 1) + η t v t , for η  := 2 t + 4: end for 5: return w T
Revised: arg min s ∈{ s 1 ,..., s n } s T ∇ f ( x t − 1 ) 3: x t ← (1 − η t ) x t − 1 + η t s , for η t := 2 t + 4: end for 5: return x T

Original: For a convex function f : X (cid:55)→ R the Frank-Wolfe algorithm (FW) solves the constrained optimization problem over a compact and convex set X .
Revised: For a convex function f : X (cid:55)→ R the Frank-Wolfe algorithm (FW) solves the constrained optimization problem over a compact and convex set X .

Original: The standard FW is known to have a sublinear convergence rate, and various methods are proposed to improve the performance.
Revised: The standard FW is known to have a sublinear convergence rate, and various methods are proposed to improve the performance.

Original: For example, when the underlying feasible set is a polytope, and the objective function is strongly convex, multiple variants, such as away-step FW (Wolfe, 1970; Jaggi, 2013), pairwise FW (Mitchell et al., 1974), and Wolfe’s method (Wolfe, 1976) are shown to enjoy linear convergence rate.
Revised: For example, when the underlying feasible set is a polytope, and the objective function is strongly convex, multiple variants, such as away-step FW (Wolfe, 1970; Jaggi, 2013), pairwise FW (Mitchell et al., 1974), and Wolfe’s method (Wolfe, 1976) are shown to enjoy linear convergence rate.

Original: Linear convergence under other conditions are also studied (Beck & Shtern, 2017; Garber & Hazan, 2013a;b).
Revised: Linear convergence under other conditions is also studied (Beck & Shtern, 2017; Garber & Hazan, 2013a;b).

Original: A. W OLFE ’ S M ETHOD FOR M INIMUM N ORM P OINT
Revised: A. W OLFE ’ S M ETHOD FOR M INIMUM N ORM P OINT

Original: Algorithm

Original: Initialize x ∈ P , active set S =
Revised: Initialize x ∈ P , active set S =

Original: [ x ] and weight λ = [1] .
Revised: [ x ] and weight λ = [1] .

Original: Output: x ∈ P that has the minimum Euclidean norm.
Revised: Output: x ∈ P that has the minimum Euclidean norm.

Original: 1: while true do //
Revised: 1: while true do //

Original: Major cycle 2: s ← Oracle ( x ) //
Revised: Major cycle 2: s ← Oracle ( x ) //

Original: Potential improving point 3: if || x || 2 ≤ x T s + (cid:15) then break 4:
Revised: Potential improving point 3: if || x || 2 ≤ x T s + (cid:15) then break 4:

Original: S ← S ∪ { s } 5: while true do // Minor cycle 6: y , α ← AffineMinimizer ( S ) //
Revised: S ← S ∪ { s } 5: while true do // Minor cycle 6: y , α ← AffineMinimizer ( S ) //

Original: y = arg min s ∈ aff ( S ) ||
Revised: y = arg min s ∈ aff ( S ) ||

Original: s ||7: if α s > 0 for all s then break // y ∈ conv ( S ) 8: //
Revised: s ||7: if α s > 0 for all s then break // y ∈ conv ( S ) 8: //

Original: If y / ∈ conv ( S ) , then update y to the intersection of conv ( S ) and segment joining x and y .
Revised: If y / ∈ conv ( S ) , then update y to the intersection of conv ( S ) and segment joining x and y .

Original: Then remove points in S unnecessary for describing y .
Revised: Then remove points in S unnecessary for describing y .

Original: 9: θ ← min i : α i ≤λ
Revised: 9: θ ← min i : α i ≤λ

Original: i
Revised: i

Original: λ
Revised: λ

Original: i − α
Revised: i − α

Original: i // Recall λ satisﬁes x = (cid:80) s ∈S λ s s 10: y ← θ y + (1 − θ )
Revised: i // Recall λ satisﬁes x = (cid:80) s ∈S λ s s 10: y ← θ y + (1 − θ )

Original: x , λ i = θα i + (1 − θ )
Revised: x , λ i = θα i + (1 − θ )

Original: λ i 11:
Revised: λ i 11:

Original: S ← { s i | s i ∈ S and λ i > 0 } 12: end while 13:
Revised: S ← { s i | s i ∈ S and λ i > 0 } 12: end while 13:

Original: Update x = y and λ = α .
Revised: Update x = y and λ = α .

Original: 14: end while 15: return x
Revised: 14: end while 15: return x

Original: Wolfe’s method is an iterative algorithm for ﬁnding the point with minimum Euclidean norm in a polytope, which is deﬁned as the convex hull of a set of ﬁnitely many points.
Revised: Wolfe’s method is an iterative algorithm for ﬁnding the point with minimum Euclidean norm in a polytope, which is deﬁned as the convex hull of a set of ﬁnitely many points.

Original: The Wolfe’s method consists of a ﬁnite number of major cycles, each of which consists of a ﬁnite number of minor cycles.
Revised: The Wolfe’s method consists of a ﬁnite number of major cycles, each of which consists of a ﬁnite number of minor cycles.

Original: At the start of each major cycle, let H ( x ) :=
Revised: At the start of each major cycle, let H ( x ) :=

Original: { y T x = x x } be the hyperplane deﬁned by x .
Revised: { y T x = x x } be the hyperplane deﬁned by x .

Original: If H ( x ) separates the polytope from the origin, then the major cycle is terminated.
Revised: If H ( x ) separates the polytope from the origin, then the major cycle is terminated.

Original: Otherwise, it invokes an oracle to ﬁnd any point on the near side of the hyperplane.
Revised: Otherwise, it invokes an oracle to ﬁnd any point on the near side of the hyperplane.

Original: The point is then added into the active set S , and starts a minor cycle.
Revised: The point is then added into the active set S , and starts a minor cycle.

Original: In a minor cycle, let y be the point of smallest norm in of the afﬁne hull aff ( S ) .
Revised: In a minor cycle, let y be the point of smallest norm in of the afﬁne hull aff ( S ) .

Original: If y is in the relative interior of the convex hull conv ( S ) , then x is updated to y and the minor cycle is terminated.
Revised: If y is in the relative interior of the convex hull conv ( S ) , then x is updated to y and the minor cycle is terminated.

Original: Otherwise, y is updated to the nearest point to y on the line segment conv ( S ) ∩
Revised: Otherwise, y is updated to the nearest point to y on the line segment conv ( S ) ∩

Original: [ x , y ] .
Revised: [ x , y ] .

Original: Thus y is updated to a boundary point of conv ( S ) , and any point that is not on the face of conv ( S ) in which y lies is deleted.
Revised: Thus y is updated to a boundary point of conv ( S ) , and any point that is not on the face of conv ( S ) in which y lies is deleted.

Original: The minor cycles are executed repeatedly until S becomes a corral , that is, a set whose afﬁne minimizer lies inside its convex hull.
Revised: The minor cycles are executed repeatedly until S becomes a corral , that is, a set whose afﬁne minimizer lies inside its convex hull.

Original: Since a set of one point is always a corral, the minor cycles is terminated after a ﬁnite number of runs.
Revised: Since a set of one point is always a corral, the minor cycles is terminated after a ﬁnite number of runs.

Original: B P ROOF OF T HEOREM 4. 
Revised: B P ROOF OF T HEOREM 4.

Original: Theorem 4.1 (Approximation Error Strictly Decreases) .
Revised: Theorem 4.1 (Approximation Error Strictly Decreases) .

Original: For any non-terminal step t , we have err ( µ t +1 ) <
Revised: For any non-terminal step t , we have err ( µ t +1 ) <

Original: err ( µ t ) .
Revised: err ( µ t ) .

Original: That is, the measurement vector of µ t found by Algorithm 1 gets strictly closer to the convex set Ω after major cycle step.
Revised: That is, the measurement vector of µ t found by the C2RL method gets strictly closer to the convex set Ω after major cycle step.

Original: Proof.
Revised: Proof.

Original: If the current step is a major cycle with no minor cycle, then x t +1 is the afﬁne minimizer of aff ( S ∪ { s t } ) with respect to ω t .
Revised: If the current step is a major cycle with no minor cycle, then x t +1 is the afﬁne minimizer of aff ( S ∪ { s t } ) with respect to ω t .

Original: Then the afﬁne minimizer property implies ( s t − x t +1 )
Revised: Then the afﬁne minimizer property implies ( s t − x t +1 )

Original: ( x t +1 − ω t )
Revised: ( x t +1 − ω t )

Original: = 0 .
Revised: = 0 .

Original: Since iteration does not terminate at step t , we have ( x t − ω t ) T ( x t − s t ) > 0 , and
Revised: Since iteration does not terminate at step t , we have ( x t − ω t ) T ( x t − s t ) > 0 , and

Original: therefore x t +1 not equal to x t .
Revised: therefore x t +1 not equal to x t .

Original: Then x t +1 is the unique afﬁne minimizer implies f Ω ( x t +1 )
Revised: Then x t +1 is the unique afﬁne minimizer implies f Ω ( x t +1 )

Original: =
Revised: =

Original: min ω ∈ Ω || x t +1 − ω
Revised: min ω ∈ Ω || x t +1 − ω

Original: || 2 ≤ || x t +1 − ω t
Revised: || 2 ≤ || x t +1 − ω t

Original: || 2 <
Revised: || 2 <

Original: || x t − ω t
Revised: || x t − ω t

Original: || 

Revised: ||

Original: = f Ω ( x t ) .
Revised: = f Ω ( x t ) .

Original: Otherwise the current step contains one or more minor cycles.
Revised: Otherwise the current step contains one or more minor cycles.

Original: In this case, we show that the ﬁrst minor cycle strictly reduces the approximation error, and the (possibly) following minor cycles cannot increase it.
Revised: In this case, we show that the ﬁrst minor cycle strictly reduces the approximation error, and the (possibly) following minor cycles cannot increase it.

Original: For the ﬁrst minor cycle, the afﬁne minimizer y 0 of aff ( S ∪ { s t } ) with respect to ω t is outside conv ( S ∪ { s t } ) .
Revised: For the ﬁrst minor cycle, the afﬁne minimizer y 0 of aff ( S ∪ { s t } ) with respect to ω t is outside conv ( S ∪ { s t } ) .

Original: Let z
Revised: Let z

Original: = θ y 

Revised: = θ y

Original: + (1 − θ )
Revised: + (1 − θ )

Original: x t be the intersection of conv ( S ∪ { s t } ) and segment joining x and y .
Revised: x t be the intersection of conv ( S ∪ { s t } ) and segment joining x and y .

Original: Let V 0 :
Revised: Let V 0 :

Original: = S t and V i denotes the active set after the i -th minor cycle.
Revised: = S t and V i denote the active set after the i -th minor cycle.

Original: Then since y 1 is the afﬁne minimizer of V 1 with respect to ω t , we have
Revised: Then since y 1 is the afﬁne minimizer of V 1 with respect to ω t , we have

Original: [Equation]
Revised: [Equation]

Original: where the second step uses the triangle inequality and the last step follows since the segment x t yintersects the interior of conv ( S∪{ s t } ) , and the distance to ω t strictly decreases along this segment.
Revised: where the second step uses the triangle inequality and the last step follows since the segment x t yintersects the interior of conv ( S∪{ s t } ) , and the distance to ω t strictly decreases along this segment.

Original: Therefore the point z found by ﬁrst minor cycle satisﬁes
Revised: Therefore the point z found by ﬁrst minor cycle satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Hence h ( y 1 ) < h
Revised: Hence h ( y 1 ) < h

Original: ( x t ) , and the ﬁrst minor cycle strictly decreases the approximation error.
Revised: ( x t ) , and the ﬁrst minor cycle strictly decreases the approximation error.

Original: By a similar argument, in subsequent minor cycles the approximation error cannot be increased.
Revised: By a similar argument, in subsequent minor cycles the approximation error cannot be increased.

Original: However, after the ﬁrst minor cycle, the iterating point may already at the intersection point and the strict inequality in last step of Eq.
Revised: However, after the ﬁrst minor cycle, the iterating point may already at the intersection point and the strict inequality in last step of Eq.

Original: 14 need to be replaced by non-strict inequality.
Revised: 17 need to be replaced by non-strict inequality.

Original: Therefore any major cycle either ﬁnds an improving point and continue, or enter minor cycles where the ﬁrst minor cycle ﬁnds an improving point, and the subsequent minor cycles does not increase the distance.
Revised: Therefore any major cycle either ﬁnds an improving point and continue, or enters minor cycles where the ﬁrst minor cycle ﬁnds an improving point, and the subsequent minor cycles does not increase the distance.

Original: Adding both side of f Ω ( x t +1 ) <
Revised: Adding both side of f Ω ( x t +1 ) <

Original: f Ω ( x t ) by f Ω ( x ∗ ) and we have the approximation error h
Revised: f Ω ( x t ) by f Ω ( x ∗ ) and we have the approximation error h

Original: ( x t +1 ) <
Revised: ( x t +1 ) <

Original: h ( x t ) strictly decreases.
Revised: h ( x t ) strictly decreases.

Original: C P ROOF OF T HEOREM 4. 
Revised: C P ROOF OF T HEOREM 4.

Original: We ﬁrst prove the Theorem 4.2, using Lemma 4.3 and Lemma 4.4.
Revised: We ﬁrst prove the Theorem 4.2, using Lemma 4.3 and Lemma 4.4.

Original: Then  present the proof of the lemmas.
Revised: Then we present the proof of the lemmas.

Original: Theorem 4.2 (Convergence in Approximation Error) .
Revised: Theorem 4.2 (Convergence in Approximation Error) .

Original: For t ≥ 1 , the mixed policy µ t found by 
Revised: For t ≥ 1 , the mixed policy µ t found by the

Original: Algorithm 1 satisﬁes
Revised: C2RL method satisﬁes

Original: [Equation]
Revised: [Equation]

Original: (16) where Q := max µ ∈ ∆( U ) ||
Revised: (19) where Q := max µ ∈ ∆( U ) ||

Original: c ( µ ) || is the maximum norm of a measurement vector.
Revised: c ( µ ) || is the maximum norm of a measurement vector.

Original: Proof.
Revised: Proof.

Original: Since Lemma 4.4 shows that drop steps are no more than half of total major cycle steps, and
Revised: Since Lemma 4.4 shows that drop steps are no more than half of total major cycle steps, and

Original: Theorem 4.1 guarantees these drop steps reducing the approximation error, 
Revised: Theorem 4.1 guarantees these drop steps reducing the approximation error, we can safely skip these step, and re-index the step numbers to include non-drop steps only using k .

Original: For these non-drop steps, we claim that err ( µ k ) ≤ 8 Q 2 / ( k + 1) .
Revised: For these non-drop steps, we claim that err ( µ k ) ≤ 8 Q 2 / ( k + 1) .

Original: Using Lemma 4.3, we prove the convergence rate using induction.
Revised: Using Lemma 4.3, we prove the convergence rate using induction.

Original: We ﬁrst bound the error of any err ( µ k ) .
Revised: We ﬁrst bound the error of any err ( µ k ) .

Original: For any k ≥
Revised: For any k ≥

Original: [Equation]
Revised: [Equation]

Original: where Eq.
Revised: where Eq.

Original: 18 uses the deﬁnition of our squared Euclidean distance function.
Revised: 21 uses the deﬁnition of our squared Euclidean distance function.

Original: Eq.
Revised: Eq.

Original: 19 follows from triangle inequality, and Eq.
Revised: 22 follows from triangle inequality, and Eq.

Original: 20 is by the contractive property of the Euclidean distance.
Revised: 23 is by the contractive property of the Euclidean distance.

Original: When k = 1 , the Eq.
Revised: When k = 1 , the Eq.

Original: 22 established the based case.
Revised: 25 established the based case.

Original: Now for k ≥ 1 , assume that err ( µ k ) ≤ 8 Q 2 / ( k + 1) for k ≥ 1 , then Lemma 4.3 gives err ( µ k +1 ) ≤ err ( µ k ) − err 2 ( µ k ) / 8 Q 2 .
Revised: Now for k ≥ 1 , assume that err ( µ k ) ≤ 8 Q 2 / ( k + 1) for k ≥ 1 , then Lemma 4.3 gives err ( µ k +1 ) ≤ err ( µ k ) − err 2 ( µ k ) / 8 Q 2 .

Original: Since the quadratic function of the right hand side is monotonically increasing on ( −∞ , 4 Q 2 ] , using the inductive hypothesis
Revised: Since the quadratic function of the right hand side is monotonically increasing on ( −∞ , 4 Q 2 ] , using the inductive hypothesis

Original: [Equation]
Revised: [Equation]

Original: Then since for t steps of major cycle steps, the number of non-drop steps k > t/ 2 , we conclude that err ( µ t ) ≤ 16 Q 2 / ( t + 2) .
Revised: Since for t steps of major cycle steps, the number of non-drop steps k > t/ 2 , we conclude that err ( µ t ) ≤ 16 Q 2 / ( t + 2) .

Original: Then we prove the lemmas.
Revised: Then we prove the lemmas.

Original: Lemma 4.3.
Revised: Lemma 4.3.

Original: For a non-drop step, we have err ( µ t ) − err ( µ t +1 ) ≥ err 2 ( µ t ) / 8 Q 2 .
Revised: For a non-drop step, we have err ( µ t ) − err ( µ t +1 ) ≥ err 2 ( µ t ) / 8 Q 2 .

Original: Proof.
Revised: Proof.

Original: The non-drop step contains either no minor cycle or one minor cycle.
Revised: The non-drop step contains either no minor cycle or one minor cycle.

Original: We ﬁrst consider the no minor cycle case.
Revised: We ﬁrst consider the no minor cycle case.

Original: If a major cycle contains no minor cycle, then x t +1 is the afﬁne minimizer of the S ∪ { s t } . 
Revised: If a major cycle contains no minor cycle, then x t +1 is the afﬁne minimizer of the S ∪ { s t } . [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the equation (28) follows from the afﬁne minimizer property Eq.
Revised: where the equation (31) follows from the afﬁne minimizer property Eq.

Original: For || x t − x t +1 || in the last equation, and ∀ q ∈ aff ( S ∪ { s t } ) , we have
Revised: For || x t − x t +1 || in the last equation, and ∀ q ∈ aff ( S ∪ { s t } ) , we have

Original: [Equation]
Revised: [Equation]

Original: Then it sufﬁces to show that ( x t − ω t )( x t − q ) ≥ err ( µ t ) .
Revised: Then it sufﬁces to show that ( x t − ω t )( x t − q ) ≥ err ( µ t ) .

Original: Since Ω is a convex set, the squared Euclidean distance function dist  ( x , Ω) is convex for x , which implies
Revised: Since Ω is a convex set, the squared Euclidean distance function dist 2 ( x , Ω) is convex for x , which implies

Original: [Equation]
Revised: [Equation]

Original: Putting in ∇ dist  ( x t , Ω) =
Revised: Putting in ∇ dist 2 ( x t , Ω) =

Original: ( x t − Proj Ω ( x t ))
Revised: ( x t − Proj Ω ( x t ))

Original: =
Revised: =

Original: ( x t − ω t ) , we get ( x t − ω t )( x t − q ) ≥ err ( µ t ) , which together with Eq.
Revised: ( x t − ω t ) , we get ( x t − ω t )( x t − q ) ≥ err ( µ t ) , which together with Eq.

Original: 29 and Eq.
Revised: 32 and Eq.

Original: 33 concludes that for non-drop step with no minor cycles, we have err ( µ t ) − err ( µ t +1 ) ≥ err 2 ( µ t ) / 8 Q 2 .
Revised: 36 concludes that for non-drop step with no minor cycles, we have err ( µ t ) − err ( µ t +1 ) ≥ err 2 ( µ t ) / 8 Q 2 .

Original: For non-drop step with one minor cycle, we use the Theorem 6 of (Chakrabarty et al., 2014).
Revised: For non-drop step with one minor cycle, we use the Theorem 6 of (Chakrabarty et al., 2014).

Original: By a linear translation of adding all points with − ω t , it gives
Revised: By a linear translation of adding all points with − ω t , it gives

Original: [Equation]
Revised: [Equation]

Original: Then applying the same argument as Eq. 34, and we ﬁnished our proof.
Revised: Then applying the same argument as Eq. 37, and we ﬁnished our proof.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: After t major cycle steps of Algorithm 1, the number of drop steps is less than t/ 2 .
Revised: After t major cycle steps of C2RL method, the number of drop steps is less than t/ 2 .

Original: Proof.
Revised: Proof.

Original: Recall that at the termination of a minor cycle, the size of the active set |S c | ∈ [1 , m ] .
Revised: Recall that at the termination of a minor cycle, the size of the active set |S c | ∈ [1 , m ] .

Original: Since in each major cycle steps, the size of active set S t increases by one, and each drop step reduces the size of S t by at least one, the number of drop steps is always less than half of total number of the major cycle steps.
Revised: Since in each major cycle steps, the size of active set S t increases by one, and each drop step reduces the size of S t by at least one, the number of drop steps is always less than half of total number of the major cycle steps.

Original: D P ROOF OF T HEOREM 4. 
Revised: D P ROOF OF T HEOREM 4.

Original: Theorem 4.5 (Memory Complexity Bound) .
Revised: Theorem 4.5 (Memory Complexity Bound) .

Original: For an constrained RL problem with m -dimensional measurement vector, in the worst case, a mixed policy needs to randomize among m + 1 individual policies to ensure convergence of RL oracles that search for deterministic policies.
Revised: For an constrained RL problem with m -dimensional measurement vector, in the worst case, a mixed policy needs to randomize among m + 1 individual policies to ensure convergence of RL oracles that search for deterministic policies.

Original: Proof.
Revised: Proof.

Original: We give a constructive proof.
Revised: We give a constructive proof.

Original: Consider a m -dimensional vector-valued MDP with a single state, m + 1 actions, and c ( a i ) :
Revised: Consider a m -dimensional vector-valued MDP with a single state, m + 1 actions, and c ( a i ) :

Original: = e i is the unit vector of i -th dimension for i ∈ [1 , m ] , and c ( a m +1 ) := 0 , and the episode terminates after 1 steps.
Revised: = e i is the unit vector of i -th dimension for i ∈ [1 , m ] , and c ( a m +1 ) := 0 , and the episode terminates after 1 steps.

Original: The constrained RL problem is to ﬁnd a policy whose measurement vector lies in the convex set of a single point { 1 / 2 m } .
Revised: The constrained RL problem is to ﬁnd a policy whose measurement vector lies in the convex set of a single point { 1 / 2 m } .

Original: By linear programming, it is clear that the only feasible mixed deterministic policy is to select a m +1 with 1 /probability, and the rest m actions with 1 / 2 m probability; i.e. the unique feasible policy to this problem has an active set containing m + 1 deterministic policies. 
Revised: By linear programming, it is clear that the only feasible mixed deterministic policy is to select a m +1 with 1 /probability, and the rest m actions with 1 / 2 m probability; i.e. the unique feasible policy to this problem has an active set containing m + 1 deterministic policies. Therefore any method randomize among less than m + 1 individual policies does not ensure convergence when used with RL algorithms searching for deterministic policies. E A DDITIONAL E XPERIMENT D ETAILS All the methods use the same A2C agent.

Revised: The input is the one-hot encoded current position index.

Original: Therefore any method randomize among less than m + 1 individual policies does not ensure convergence when used with RL algorithms searching for deterministic policies.
Revised: The A2C is the standard fully connected multi-layer perceptron with ReLU activation function. The actor and critic share the internal representation and have their only ﬁnal layer. Both actor and critic networks use Adam optimizer with learning rate set to 1 e − 2 . The network is as follows

Revised: Actor

Revised: Critic

Revised: Input layer

Revised: One-hot encoded state index (dim= 54 )

Revised: Hidden layer

Revised: Linear(in= 54 , out= 128 , act=”relu”)

Revised: Output layer

Revised: Linear(in= 128 , out= 4 , act=”relu”)

Revised: Linear(in= 128 , out= 1 , act=”relu”)

Revised: Output name

Revised: Action score

Revised: State value

Revised: For ApproPO, the constant κ for projection convex set to convex cone is set to be 20 .

Revised: The θ is initialized to 0 .

Revised: Following the original paper.

Revised: For RCPO, the learning rate of its λ is set to 2 .

Revised: 5 e − 5 , and its λ is initialized to 0 and updated by online gradient descent with learning rate set to 1 , as used by the original paper.

Revised: The proposed C2RL introduces no extra hyper-parameters, and has nothing to report.