Understanding Curriculum Learning in Policy Optimization for Online Combinatorial Optimization
Abstract
Over the recent years, reinforcement learning (RL) starts to show promising results in tackling combinatorial optimization (CO) problems, in particular when coupled with curriculum learning to facilitate training.
Despite emerging empirical evidence, theoretical study on why RL helps is still at its early stage.
This paper presents the first systematic study on policy optimization methods for online CO problems.
We show that online CO problems can be naturally formulated as latent
Markov Decision Processes (LMDPs), and prove convergence bounds on natural policy gradient (NPG) for solving LMDPs.
Furthermore, our theory explains the benefit of curriculum learning: it can find a strong sampling policy and reduce the distribution shift, a critical quantity that governs the convergence rate in our theorem.
For a canonical online CO problem, Secretary Problem, we formally prove that distribution shift is reduced exponentially with curriculum learning even if the curriculum is randomly generated .
Our theory also shows we can simplify the curriculum learning scheme used in prior work from multi-step to single-step.
Lastly, we provide extensive experiments on Secretary Problem and Online Knapsack to verify our findings.
1. Introduction
In recent years, machine learning has shown promising results in solving combinatorial optimization (CO) problems, including traveling salesman problem (TSP, Kool et al.
[13]), maximum cut [11] and satisfiability problem [17].
While in the worst case some CO problems are NP-hard, in practice, the probability that we need to solve the worst-case problem instance is low [6].
A significant subclass of CO problems is called online CO problems, which has gained much attention [8–10].
Online CO problems entail a sequential decision-making process, which perfectly matches the nature of reinforcement learning (RL).
This paper concerns using RL to tackle online CO problems.
RL is often coupled with specialized techniques including (a particular type of)
Curriculum Learning [12], human feedback and correction (P´erez-Dattari et al.
[15], Scholten et al.
[16]), and policy aggregation (boosting, Brukhim et al.
[5]) to accelerate the training speed.
However, it is unclear when and why they improve the performance.
We particularly focus on
RL with Curriculum Learning (Bengio et al.
[4], also named “bootstrapping” in Kong et al.
[12]): train the agent from an easy task and gradually increase the difficulty until the target task.
Main contributions.
In this paper, we initiate the formal study on using RL to tackle online
CO problems, with a particular emphasis on understanding the specialized techniques developed in this emerging subarea.
Our contributions are summarized below.
• Formalization.
For online CO problems, we want to learn a single policy that enjoys good performance over a distribution of problem instances.
This motivates us to use Latent Markov Decision Process (LMDP) [14] instead of standard MDP formulation.
We give concrete examples, Secretary Problem (SP) and Online Knapsack, to show how LMDP models online CO problems.
• Provable efficiency of policy optimization.
We analyze the performance of NPG for LMDP.
The performance bound is characterized by the number of iterations, the excess risk of policy evaluation, the transfer error and the relative condition number κ that characterizes the distribution shift between the sampling policy and the optimal policy.
• Understanding and simplifying Curriculum Learning.
Using our performance guarantee on NPG for LMDP, we study when and why Curriculum Learning is beneficial to RL for online CO problems.
Our main finding is that the main effect of Curriculum Learning is to give a stronger sampling policy.
Under certain circumstances, Curriculum Learning reduces the relative condition number κ , improving the convergence rate.
For the Secretary Problem, we provably show that Curriculum Learning can exponentially reduce κ compared with using the na¨ıve sampling policy.
Surprisingly, this means even a random curriculum of SP accelerates the training exponentially .
As a direct implication, we show that the multi-step Curriculum Learning proposed in [12] can be significantly simplified into a single-step scheme.
Lastly, to obtain a complete understanding, we study the failure mode of Curriculum Learning.
To verify our theories, we conduct extensive experiments on two classical online CO problems (Secretary Problem and Online Knapsack).
2. Problem setup
In this section, we first introduce LMDP and why it naturally formulates online CO problems.
The next are necessary components required by the algorithm, Natural Policy Gradient.
2.1. Latent Markov Decision Process
Latent MDP [14] is a collection of MDPs M =
{M 1 , M 2 , . . .
, M M } .
All the MDPs share state set S , action set A and horizon H .
Each MDP M m =
( S , A , H, ν m , P m , r m ) has its own initial state distribution ν m ∈ ∆( S ) , transition P m : S ×
A → ∆( S ) and reward r m : S ×
A → [0 , 1] , where ∆( S ) is the probability simplex over S .
Let w 1 , w 2 , . .
, w M be the mixing weights of MDPs such that w m > 0 for any m and
(cid:80) Mm =1 w m = 1 .
At the start of every episode, one MDP M m ∈ M is randomly chosen with probability w m .
Due to the time and space complexities of finding the optimal history-dependent policies, we stay in line with Kong et al.
[12] and care only about finding the optimal history-independent policy.
Let Π =
π : S → ∆( A ) } denote the class of all the history-independent policies.
Log-linear policy.
Let ϕ : S ×
A → R d be a feature mapping function where d denotes the dimension of feature space.
Assume that ∥ ϕ ( s, a ) ∥ 2 ≤ B .
A log-linear policy is of the form: π θ
[Equation]
a | s )
[Equation]
a ))
[Equation]
θ ⊤ ϕ ( s, a ′ )) , where θ ∈ R d .
Entropy regularized value function, Q-function and advantage function.
We incorporate entropy regularization for completeness because prior works used it to facilitate training.
The value function V π,λm,h ( s ) is defined as the sum of future λ -regularized rewards starting from s and executing π for h steps in M m .
The λ -regularized reward is defined as r π,λm ( s, a ) :
= r m ( s, a ) + λ ln 1 π ( a | s ) , which depends on the current policy π .
Denote V π,λ := (cid:80)
Mm =1 w m (cid:80) s 0 ∈S ν m ( s 0 )
V π,λm,H ( s 0 ) .
Under regularization, we seek for π ⋆λ = arg max π ∈ Π V π,λ .
Denote V
⋆,λ =
V π ⋆λ ,λ .
For notational ease, we abuse π ⋆ with π ⋆λ .
The Q-function Q π,λm,h ( s, a ) and advantage function A π,λm,h ( s, a ) :=
Q π,λm,h ( s, a ) − V π,λm,h ( s ) are defined in a similar manner.
Modelling of online CO problems.
Refer to Sec.
A for the formulation of two online CO problems,
Secretary Problem and Online Knapsack (decision version), using LMDPs.
2.2. Algorithm components
In this subsection we will introduce some necessary notions used by our main algorithm.
Definition 1 (State(-action) Visitation Distribution)
The state visitation distribution and stateaction visitation distribution at step h ≥ 0 with respect to π in M m are defined as
[Equation]
We define a grafted distribution (cid:101) d πm,h ( s, a ) :
= d πm,h ( s ) ◦ Unif A ( a ) which can be attained by first acting under π for h steps to get states then sample actions from the uniform distribution Unif A .
Denote d ♣ m,h :=
d π ♣ m,h and d ♣ as short for { d ♣ m,h } 1 ≤ m ≤ M, 0 ≤ h ≤ H − 1 , here ♣ can be any symbol.
In the following definitions, let v be the collection of any distribution, which will be instantiated by d ⋆ , d t , etc.
in the remaining sections.
Definition 2 (Compatible Function Approximation Loss) Let g be the parameter update weight, then NPG is related to finding the minimizer for the following function:
[Equation]
Definition 3 (Generic Fisher Information Matrix)
[Equation]
Particularly, denote F ( θ ) =
Σ θd θ as the Fisher information matrix induced by π θ .
3. Learning procedure
Natural Policy Gradient.
Starting from θ 0 , the algorithm updates the parameter by setting θ t +
=
θ t + ηg t , where η is a predefined constant learning rate, and g t is the update weight.
Denote π t :=
π θ t , V t,λ :=
V π t ,λ and A t,λm,h :=
A π t ,λm,h for convenience.
NPG satisfies g t ∈ arg min g L ( g ; θ t , d θ t ) .
When we only have samples, we use the approximate version of NPG: g t ≈ arg min g ∈G L ( g ; θ t , d θ t ) , where G = { x : ∥ x ∥ 2 ≤ G } for some hyper-parameter G .
We also introduce a variant of NPG: instead of sampling from d θ t using the current policy π t , we sample from (cid:101)
d
π s using a fixed sampling policy π s .
The update rule is g t ≈ arg min g ∈G L ( g ; θ t , (cid:101) d π s ) .
This version makes a closed-form analysis for SP possible.
The main algorithm is shown in Alg.
1 (deferred to App. B).
It admits two types of training: ①
If π s =
None, it samples s, a ∼ d θ t ; ② If π s ̸
=
None, it samples s, a ∼ (cid:101) d π s .
In both cases, we denote d t as the sampling distribution and Σ t as the induced Fisher Information
Matrix used in step t , i.e. d t := d θ t , Σ t := F ( θ t ) if π s =
None; d t := (cid:101) d π s , Σ t :=
Σ θ t (cid:101) d
πs otherwise.
The update rule can be written in a unified way as g t ≈ arg min g ∈G L ( g ; θ t , d t ) .
This is equivalent to solving a constrained quadratic optimization and we can use existing solvers.
Curriculum Learning.
We use Curriculum Learning to facilitate training.
Alg.
3 (deferred to App. B) is our proposed training framework, which first constructs an easy environment E ′ and trains a (near-)optimal policy π s of it.
In the target environment E , we either use π s to sample data while training a new policy from scratch, or simply continue training π s .
To be specific and provide clarity for the results in Sec. 5, we name a few training modes (without regularization) here, and the rest are in Tab. 1.
curl , the standard Curriculum Learning, runs Alg. 3 with sampler always to be the current policy; fix samp curl stands for the fixed sampler Curriculum Learning, running Alg.
3 with π s as the sampler.
direct means directly learning in E without curriculum, i.e., running Alg. 1; naive samp also directly learns in E , while using the na¨ıve random policy to sample data.
4. Performance analysis 4.1. Natural Policy Gradient for Latent MDP
Let
g ⋆t ∈ arg min g ∈G L ( g ; θ t , d t ) denote the true minimizer.
We have the following definitions:
Definition 4 Define for 0 ≤ t ≤ T :
• (Excess risk) ϵ stat :=
max t E [ L ( g t ; θ t , d t ) − L ( g ⋆t ; θ t , d t )] ; • (Transfer error)
ϵ bias := max t E
[ L ( g ⋆t ; θ t , d ⋆ )] ; • (Relative condition number) κ :=
max t E (cid:20) sup x ∈ R d x ⊤ Σ
θtd⋆ x x ⊤ Σ t x (cid:21) .
Note that term inside the expectation is a random quantity as θ t is random.
The expectation is with respect to the randomness in the sequence of weights g 0 , g 1 , . .
. , g T .
ϵ stat is due to that the minimizer g t from samples may not minimize the population loss L .
ϵ bias quantifies the approximation error due to feature maps.
κ characterizes the distribution mismatch between d t and d ⋆ .
Here we introduce a fitting error which depicts the closeness between π ⋆ and any policy π .
Definition 5 (Fitting Error)
Suppose the update rule is θ t +
=
θ t + ηg t , define err
[Equation]
Thm. 6 shows the convergence rate of Alg. 1, and its proof is deferred to Sec.
C.1.
Theorem 6 With Def. 4, 5 and 8, our algorithm enjoys the following performance bound:
[Equation]
where Φ( π 0 ) is the Lyapunov potential function which is only relevant to the initialization.
As a remark: If we can find a policy whose κ is small with a single curriculum, we do not need the
4.2. Curriculum learning for Secretary Problem
For SP, there exists a threshold policy that is optimal [3].
Suppose the threshold is p ∈ (0 , 1) , then the policy is: accept candidate i if and only if in > p and X i = 1 .
For the classical SP where all the n !
instances have equal probability to be sampled, the optimal threshold is 1 / e.
To show that curriculum learning makes the training converge faster, Thm. 6 gives a direct hint: curriculum learning produces a good sampler leading to much smaller κ than that of a na¨ıve random sampler.
Here we focus on the cases where samp =
pi s because the sampler is fixed, while when samp = pi t it is impossible to analyze a dynamic procedure.
We show Thm.
7 to characterize κ in SP.
Its full statement and proof is deferred to Sec.
C.2.
Theorem 7 Assume that each candidate is independent of others and the i -th candidate has a probability P i of being the best so far (Sec. 2.1).
Assume the optimal policy is a p -threshold policy and the sampling policy is a q -threshold policy.
There exists a policy parameterization such that:
[Equation]
where κ curl and κ na¨ıve are κ of the sampling policy and the na¨ıve random policy, respectively.
To understand how curriculum learning influences κ , we apply Thm.
7 to three concrete cases.
They show that, when the state distribution induced by the optimal policy in the small problem is similar to that in the original large problem, then a single-step curriculum suffices.
The classical case: an exponential improvement.
The probability series for the classical SP is P i
= 1 i .
Substituting them into Eq.
1 directly gives:
[Equation]
Except for the corner case where q < 1 n , we have that κ curl =
O ( n ) while κ na¨ıve = Ω(2 n ) .
Notice that any distribution with P i ≤ 1 i leads to an exponential improvement.
A more general case.
Consider the case where P i ≤ 12 for i ≥ 2 .
Eq.
1 now becomes:
[Equation]
Clearly, κ curl ≤ κ na¨ıve always holds.
When q is close to p , the difference is exponential in ⌊ nq ⌋ .
Failure mode of Curriculum Learning.
From Eq.
1, κ na¨ıve ≤ 2 n − 1 .
Similar as Sec.
3 of
Beckmann [3], the optimal threshold p satisfies:
[Equation]
So letting P n > 12 results in p ∈ [ n − 1 n , 1) .
Further, if q < n − 1 n and P j > 1 − 2 −
[Equation]
1 for any ⌊ nq ⌋ + 1 ≤ j ≤ n − 1 , then from Eq. 1, κ curl > 2 n > κ na¨ıve .
This means that Curriculum Learning can always be manipulated adversarially.
Sometimes there is hardly any reasonable curriculum.
5. Experiments
More formulation details and results are presented in Sec.
D. In Curriculum
Learning the entire training process splits into two phases.
We call the training on curriculum (small scale instances) “warm-up phase” and the training on large scale instances “final phase”.
We ran more than one experiments for each problem.
In one experiment there are more than one training processes to show the the effect of different samplers and regularization coefficients.
To highlight the effect of Curriculum
Learning, we omit the results regarding regularization, and they can be found in supplementary files.
All the trainings in the same experiment have the same distributions over LMDPs for final phase and warm-up phase (if any), respectively.
Secretary Problem.
We show one of the four experiments in Fig.
Aside from reward and ln κ , we plot the weighted average of err t according to the proof of Thm. 6: avg ( err t ) =
[Equation]
ti
[Equation]
=
[Equation]
t − i ′ .
All the instance distributions are generated from parameterized series { P n } with fixed random seeds, which guarantees reproducibility and comparability.
There is no explicit relation between the curriculum and the target environment, so the curriculum can be viewed as random and independent .
The experiments clearly demonstrate that curriculum learning can boost the performance by a large margin and curriculum learning indeed dramatically reduces κ , even the curriculum is randomly generated.
Online Knapsack (decision version).
We show one of the three experiments in Fig.
ln κ and avg( err t ) are with respect to the reference policy, a bang-per-buck algorithm, which is not the optimal policy.
Thus, they are only for reference.
The curriculum generation is also parameterized, random and independent of the target environment.
The experiments again demonstrate the effectiveness of curriculum learning and curriculum learning indeed dramatically reduces κ .
Appendix A. Motivating Online Combinatorial Optimization problems
Online CO problems are a natural class of problems that admit constructions of small-scale instances, because the hardness of online CO problems can be characterized by the input length, and instances of different scales are similar.
This property simplifies the construction of curricula and underscores curriculum learning.
In this section we introduce two motivating online CO problems.
We are interested in these problems because they have all been extensively studied and have closed-form, easy-to-implement policies as references.
Furthermore, they were studied in Kong et al.
[12], the paper that motivates our work.
They also have real-world applications, e.g., auction design [2].
A.1. Secretary Problem
In SP, the goal is to maximize the probability of choosing the best among n candidates, where n is known.
All candidates have different scores to quantify their abilities.
They arrive sequentially and when the i -th candidate shows up, the decision-maker observes the relative ranking X i among the first i candidates, which means being the X
i th-best so far.
A decision that whether to accept or reject the i -th candidate must be made immediately when the candidate comes, and such decisions cannot be revoked .
Once one candidate is accepted, the game ends immediately.
The ordering of the candidates is unknown.
There are in total n ! permutations, and an instance of SP is drawn from an unknown distribution over these permutations.
In the classical SP, each permutation is sampled with equal probability.
The optimal solution for the classical SP is the wellknown 1 / e -threshold strategy : reject all the first ⌊ n/ e ⌋ candidates, then accept the first one who is the best so-far.
In this paper, we also study some different distributions.
Modeling SP.
For SP, each instance is a permutation of length n , and in each round an instance is drawn from an unknown distribution over all permutations.
In the i -th step for i ∈ [ n ] , the state encodes the i -th candidate and relative ranking so far.
The transition is deterministic according to the problem definition.
A reward of 1 is given if and only if the best candidate is accepted.
We model the distribution as follows: suppose for candidate i , he/
she is the best so far with probability
P i and is independent of other i ′ .
The classical SP satisfies P i = 1 i .
A.2. Online Knapsack (decision version)
In Online Knapsack problems the decision-maker observes n (which is known) items arriving sequentially, each with value v i and size s i revealed upon arrival.
A decision to either accept or reject the i -th item must be made immediately when it arrives, and such decisions cannot be revoked .
At any time the accepted items should have their total size no larger than a known budget B .
The goal of standard Online Knapsack is to maximize the total value of accepted items.
In this paper, we study the decision version, denoted as OKD, whose goal is to maximize the probability of total value reaching a known target V .
We assume that all values and sizes are sampled independently from two fixed distributions, namely v 1 , v 2 , . .
. , v n i.i.d.
∼ F v and s 1 , s 2 , . .
. , s n i.i.d.
∼ F s .
In Kong et al.
[12] the experiments were carried out with F v = F s = Unif [0 , 1] , and we also study other distributions.
Modeling OKD.
For OKD, each instance is a sequence of items with values and sizes drawn from unknown distributions F v and F s .
In the i -th step for i ∈ [ n ] , the state encodes the information of i -th item’s value and size, the remaining budget, and the remaining target value to fulfill.
The transition is also deterministic according to the problem definition, and a reward of 1 is given if and only if the agent obtains the target value for the first time.
F v = F s = Unif [0 , 1] in Kong et al.
Appendix B. Skipped algorithms
In this section, we present the algorithms skipped in the main text.
Alg. 1 is the NPG for LMDP.
Alg. 2 is the sampling function.
Alg.
3 is the Curriculum Learning framework.
Algorithm
1: Input: Environment E ; learning rate η ; episode number T ; batch size N ; initialization θ 0 ; sampler π s ; regularization coefficient λ ; entropy clip bound U ; optimization domain G .
2: for t ← 0 , 1 , . . .
, T − 1 do 3:
Initialize (cid:98) F t ← 0 d × d , (cid:98) ∇ t ← 0 d .
4: for n ← 0 , 1 , . . .
, N − 1 do 5: for h ← 0 , 1 , . . .
, H − 1 do 6: if π s is not None then 7: s h , a h , (cid:98)
A H − h ( s h , a h ) ← Sample ( E, π s , True , π t , h, λ, U ) (see Alg. 2).
// s, a ∼ (cid:101) d π s m,h , estimate A t,λm,H − h
( s, a ) .
8: else 9: s h , a h , (cid:98)
A H − h ( s h , a h ) ← Sample ( E, π t , False , π t , h, λ, U ) .
// s, a ∼ d θ t m,h , estimate A t,λm,H − h
( s, a ) .
10: end if 11: end for 12:
Update: (cid:98) F t ← (cid:98) F t +
H − 1 (cid:88) h
= ∇ θ ln π θ t ( a h | s h ) ( ∇ θ ln π θ t
( a h | s h ))
⊤ , (cid:98) ∇ t ← (cid:98) ∇ t +
H − 1 (cid:88) h
= (cid:98)
A H − h ( s h , a h ) ∇ θ ln π θ t ( a h | s h ) .
13: end for 14:
Call any solver to get (cid:98) g t ← arg min g ∈G g ⊤ (cid:98) F t g − 2 g ⊤ (cid:98) ∇ t .
Update θ t +1 ← θ t + η (cid:98) g t .
16: end for 17: Return: θ T .
Appendix C. Proof of the main results C.1. Performance of Natural Policy Gradient for LMDP
First we give the skipped definition of the Lyapunov potential function Φ , then prove Thm. 6.
M ) a ∼ Unif A if unif =
True and a ∼ π samp ( ·| s ) otherwise, and estimate of A t,λm,H − h ( s, a ) .
2: E .reset().
3: for i ← 0 , 1 , . . .
, h − 1 do 4: s i ← E .get state().
Sample action a i ∼ π samp ( ·| s i ) and E .execute( a i ).
6: end for 7: s h ← E .get state().
8: if unif =
True then 9: a h ∼ Unif A .
10: else 11: a h ∼ π samp ( ·| s h ) .
12: end if 13: ( s, a ) ← ( s h , a h ) .
14: Get a random number p ∼ Unif [0 , 1] .
15: if p < 12 then 16:
Override a h ∼ π t ( ·| s h ) .
Set importance weight C ← − 2 .
18: r h ← E .execute( a h ).
Initialize cumulative reward R ← r h + λ H ( π t ( ·| s h )) .
20: else 21:
C ← 2 .
22: r h ← E .execute( a h ).
R ← r h + λ min { ln 1 π t ( a h | s h ) , U } .
24: end if 25: for i ← h + 1 , h + 2 , . .
, H − 1 do 26: s i ← E .get state().
27: a i ∼ π t ( ·| s i ) and r h ← E .execute( a i ).
R ← R + r i + λ H ( π t ( ·| s i )) .
29: end for 30: Return: s, a, (cid:98)
A t,λ H − h ( s, a ) = CR .
Definition 8 (Lyapunov Potential Function [7])
We define the potential function Φ : Π → R as follows: for any π ∈ Π ,
[Equation]
Theorem 6 (Restatement of Thm. 6) With Def. 4, 5 and 8, our algorithm enjoys the following performance bound:
[Equation]
Proof Here we make shorthands for the sub-optimality gap and potential function: ∆ t := V ⋆,λ −
V t,λ and Φ t :=
Φ( π t ) .
From Lem.
15 we have
[Equation]
Taking expectation over the update weights
[Equation]
we have
[Equation]
Thus,
[Equation] [Equation]
where the last step uses the fact that Φ( π ) ≥ 0 .
This is a weighted average, so by normalizing the coefficients,
[Equation]
where the last step comes from Lem.
16 and Jensen’s inequality.
This completes the proof.
C.2. Curriculum learning and the constant gap for Secretary Problem
Theorem 7 (Formal statement of Thm. 7) For SP, set samp =
pi s in Alg.
Assume that each candidate is independent from others and the i -th candidate has probability P i of being the best so far (see formulation in Sec.
2.1 and D.1).
Assume the optimal policy is a p -threshold policy and the sampling policy is a q -threshold policy.
There exists a policy parameterization and quantities k curl = (cid:40) (cid:81) ⌊ np ⌋
[Equation]
such that k curl ≤ κ curl ≤ 2 k curl and k na¨ıve ≤ κ na¨ıve ≤ 2 k na¨ıve .
Here κ curl and κ na¨ıve correspond to κ induced by the q -threshold policy and the na¨ıve random policy respectively.
Proof We need to calculate three state-action visitation distributions: that induced by the optimal policy, d ⋆ ; that induced by the sampler which is the optimal for the curriculum, (cid:101) d curl ; and that induced by the na¨ıve random sampler, (cid:101) d na¨ıve .
This then boils down to calculating the state(-action) visitation distribution under two types of policies: any threshold policy and the na¨ıve random policy.
For any policy π , denote d π (cid:0) in (cid:1) as the probability for the agent acting under π to see states in with arbitrary x i .
We do not need to take the terminal state g into consideration, since it stays in a zero-reward loop and contributes 0 to L ( g ; θ, d ) .
We use the LMDP distribution described in Sec. 5.
Denote π p as the p -threshold policy, i.e. accept if and only if in > p and x i = 1 .
Then
[Equation]
reject all previous i − 1 states | π p
[Equation]
Denote π na¨ıve as the na¨ıve random policy, i.e., accept with probability 12 regardless of the state.
Then
[Equation]
previous
[Equation]
For any π , we can see that the state visitation distribution satisfies
d π (cid:0) in , 1 (cid:1) =
P i d π (cid:0) in (cid:1) and d π
[Equation]
in ,
[Equation]
To show the possible largest difference, we use a parameterization that for each state s , ϕ ( s ) =
One-hot ( s ) .
The policy is then
[Equation]
Denote π θ ( s ) =
π θ ( accept | s ) , we have
[Equation]
Now suppose the optimal threshold and the threshold learned through curriculum are p and q , then
[Equation]
Denote κ ♣ ( θ ) = sup x ∈ R d x ⊤ Σ θd⋆ x x ⊤ Σ θ (cid:101)
d ♣ x .
From parameterization we know all ϕ ( s ) are orthogonal.
Abusing π q with π curl , we have
[Equation]
We can separately consider each s ∈ S because of the orthogonal features.
Observe that π p ( s ) ∈ { 0 , 1 } , so for s ∈ S , its corresponding term in κ ♣ ( θ ) is maximized when π θ ( s ) = 1 − π p ( s ) and is equal to 2 d πp ( s ) d ♣ ( s ) .
By definition, κ ♣ = max 0 ≤ t ≤ T E [ κ ♣ ( θ t )] .
Since θ 0 =
d , we have κ ♣ ≥ κ ♣ (0 d ) where π θ ( s )
=
12 and the corresponding term is d πp ( s ) d ♣ ( s ) .
So
[Equation]
We now have an order-accurate
result k ♣ = max s ∈S d πp ( s ) d ♣ ( s ) for κ ♣ .
Direct computation gives
[Equation]

[Equation]
, max
[Equation]
  .
This completes the proof.
Appendix D. Full experiments
Here are all the experiments not shown in Sec. 5.
All the experiments were run on a server with
CPU AMD Ryzen 9 3950X, GPU NVIDIA GeForce
Super and 128G memory.
For legend description please refer to the caption of Fig.
For experiment data (code, checkpoints, logging data and policy visualization) please refer to the supplementary files.
Policy parameterization.
Since in all the experiments there are exactly two actions, we can use ϕ ( s ) =
ϕ ( s, accept ) − ϕ
( s, reject ) instead of ϕ ( s, accept ) and ϕ ( s, reject ) .
Now the policy is π θ
[Equation]
accept
[Equation]
and π
[Equation] [Equation]
s
[Equation]
=
[Equation]
Training schemes.
We ran seven experiments in total, three for Secretary Problem and four for
Online Knapsack (decision version).
The difference between the experiments of the same problem lies in the distribution over instances (i.e., { w m } ).
In the following subsections, we will introduce how we parameterized the distribution in detail.
In a single experiment, we ran eight setups, each representing a combination of sampler policies, initialization policies of the final phase, and whether we used regularization.
For visual clarity, we did not plot setups with entropy regularization, but the readers can plot it using plot.py (comment L55-58 and uncomment L59-62) in the supplementary files.
We make a detailed list of the training schemes in Tab. 1.
D.1. Secretary Problem
State and action spaces.
States with X i > 1 are the same.
To make the problem “scale-invariant”, we use in to represent i .
So the states are ( in , x i =
[ X i = 1]) .
There is an additional terminal state g = (0 , 0) .
For each state, the agent can either accept or reject.
Transition and reward.
Any action in g leads back to g .
Once the agent accepts the i -th candidate, the state transits into g , and reward is 1 if i is the best in the instance.
If the agent rejects, then the state goes to ( i +1 n , x i +1 ) if i < n
and g if i = n .
For all other cases, rewards are 0 .
Feature mapping.
Recall that all states are of the form ( f, x ) where f ∈ [0 , 1] , x ∈ { 0 , 1 } .
We set a degree d 0 and the feature mapping is constructed as the collection of polynomial bases with degree less than d 0 ( d = 2 d 0 ):
[Equation]
LMDP distribution.
We model the distribution as follows: for each i , we can have x i = 1 with probability P i and is independent from other i ′ .
By definition, P 1 = 1 while other P i can be arbitrary.
The classical SP satisfies P i = 1 i .
We also experimented on three other distributions (so in total there are four experiments), each with a series of numbers p 2 , p 3 , . .
, p n i.i.d.
∼ Unif [0 , 1] and set P i = 1 i 2 pi +0 .
For each experiment, we run eight setups, each with different combinations of sampler policies, initialization policies of the final phase, and the value of regularization coefficient λ .
For the warmup phases we set n = 10 and for final phases n = 100 .
Results.
Fig.
3 (with its full view Fig. 4), Fig. 5, Fig. 6, along with Fig.
1 (with seed 2018011309) show four experiments of SP.
They shared a learning rate of 0 . 2 , batch size of 100 per step in horizon, final n = 100 and warm-up n = 10 (if applied curriculum learning).
The experiment in Fig. 3 was done in the classical SP environment, i.e., all permutations have probability 1 n ! to be sampled.
Experiments Fig.
1, Fig. 5 and Fig.
6 were done with other distributions (see LMDP distribution of Sec. 5): the only differences are the random seeds, which we fixed and used to generate P
i s for reproducibility.
The experiment of classical SP was run until the direct training of n = 100 converges, while all other experiments were run to a maximum episode of 30000 (hence sample number of T Hb = 30000 × 100 × 100 = 3 × 10 8 ).
The optimal policy was derived from dynamic programming.
State and action spaces.
The states are represented as
[Equation]
where x j = 1 [ item j was successfully chosen ] for 1 ≤ j ≤ i − 1 (in the instance).
There is an additional terminal state g = (0 , 0 , 0 , 0 , 0) .
For each state (including g for simplicity), the agent can either accept or reject.
Transition and reward.
The transition is implied by the definition of the problem.
Any action in terminal state g leads back to g .
The item is successfully chosen if and only if the agent accepts and the budget is sufficient.
A reward of 1 is given only the first time (cid:80) ij =1 x i v i ≥ V , and then the state goes to g .
For all other cases, reward is 0 .
Feature mapping.
Suppose the state is ( f, s, v, r, q ) .
We set a degree d 0 and the feature mapping is constructed as the collection of polynomial bases with degree less than d 0 ( d = d 50 ): ϕ
( f, s, v, r, q )
=
( f i f s i s
v i v r
i r q i q ) i f ,i s ,i v ,i r ,i q where i ♣ ∈ { 0 , 1 , . . .
, d 0 − 1 } .
LMDP distribution.
In Sec.
A.2 the values and sizes are sampled from F v ans F s .
If F v or F s is not Unif [0 , 1] , we model the distribution as: first set a granularity gran and take gran numbers p 1 , p 2 , . .
, p gran i.i.d.
∼ Unif [0 , 1] .
p i represents the (unnormalized) probability that x ∈ ( i − 1 gran , i gran ) .
To sample, we take i ∼ Multinomial ( p 1 , p 2 , . .
. , p gran ) and return x ∼ i − 1+ Unif [0 , 1] gran .
For each experiment, we ran four setups, each with different combinations of sampler policies and initialization policies of the final phase.
For the warm-up phases n = 10 and for final phases we set n = 100 in all experiments, while B and V vary.
In one experiment it satisfies that Bn are close for warm-up and final, and VB increases from warm-up to final.
Results.
Fig.
7, Fig. 8, along with Fig.
2 (with F v = F s = Unif [0 , 1] ) show three experiments of
OKD.
They shared a learning rate of 0 . 1 , batch size of 100 per step in horizon, final n = 100 and warm-up n = 10 (if applied curriculum learning).
Experiments Fig.
7 and Fig.
8 were done with other value and size distributions (see LMDP distribution of Sec. 5): the only differences are the random seeds, which we fixed and used to generate F v and F s for reproducibility.
All experiments were run to a maximum episode of 50000 (hence sample number of T Hb = 50000 × 100 × 100 = 5 × 10 8 ).
The reference policy is a bang-per-buck algorithm (Sec. 3.1 of Kong et al.
[12]): given a threshold r , accept i -th item if v i s i ≥ r .
We searched for the optimal r with respect to Online Knapsack because we found that in general the reward is unimodal to r and contains no “plain area”, so we can easily apply ternary search (the reward of OKD contains “plain area”).
Appendix E. Technical details and lemmas E.1. Natural Policy Gradient for LMDP
This section is a complement to Sec. 3.
We give details about the correctness of Natural Policy
Gradient for LMDP.
Thm.
11 is the finite-horizon Policy Gradient Theorem for LMDP, which takes the mixing weight { w m } into consideration.
According to Agarwal et al.
[1], the unconstrained, full-information NPG update weight satisfies
F ( θ t ) g t = ∇ θ V t,λ .
Lem.
12 and Lem.
13 together show that: it is equivalent to finding a minimizer of the fitting compatible function approximation loss (Def. 2).
Theorem 11 (Policy Gradient Theorem for LMDP)
For any policy π θ parameterized by θ , and any 1 ≤ m ≤ M ,
[Equation]
As a result,
[Equation]
Proof For any 1 ≤ h ≤ H and s ∈ S , since V π θ
[Equation]
λ m
[Equation]
h ( s ) =
(cid:80) a ∈A π θ ( a | s ) Q π θ ,λ m,h ( s, a ) , we have
[Equation]
Hence
H (cid:88) h
= (cid:88) s ∈S
[Equation]
Next we focus on the second term.
From the Bellman equation,
[Equation]
Particularly, ∇ θ Q π,λi, 1 ( s, a ) = − λ ∇ θ ln π θ ( a | s ) .
So
H (cid:88) h
= (cid:88) s ∈S
[Equation]
=
H
[Equation]
h
[Equation] [Equation]
where we used the definition of d and ν m .
So by rearranging the terms, we complete the proof.
Lemma 12 Suppose Γ ∈ R n × m , D = diag
( d 1 , d 2 , . . .
, d m ) ∈
R m × m where d i ≥ 0 and q ∈ R m , then x = (Γ D Γ ⊤ ) † Γ Dq is a solution to the equation Γ D Γ ⊤ x = Γ Dq .
Proof Denote D 1 / 2 = diag ( √ d 1 , √ d 2 , . .
√ d m ) , P = Γ D 1 / 2 , p
= D 1 / 2 q , then the equation is reduced to P P ⊤ x = P p .
Suppose the singular value decomposition of P is U Σ V ⊤ where U ∈
R n × n , Σ ∈ R n × m , V ∈ R m × m where U and V are unitary, and singular values are σ 1 , σ 2 , . . .
, σ k .
So P P ⊤ =
U (ΣΣ ⊤ ) U ⊤ and ( P P ⊤ ) † = U (ΣΣ ⊤ ) † U ⊤ .
Notice that
[Equation]
we can then derive the pseudo-inverse of this particular diagonal matrix as
[Equation]
It is then easy to verify that (ΣΣ ⊤ )(ΣΣ ⊤ ) † Σ = Σ .
Finally
[Equation]
This completes the proof
[Equation]
Lemma 13 (NPG Update Rule)
The update rule θ ← θ + ηF ( θ ) † ∇ θ V π θ ,λ where
[Equation]
is equivalent to θ ← θ + ηg ⋆ , where g ⋆ is a minimizer of the function
[Equation]
Proof
[Equation]
Suppose g ⋆ is any minimizer of L ( g ) , we have ∇ g L ( g ⋆ )
= 0 , hence
M
[Equation]
Since ( u ⊤ v ) v = ( vv ⊤ ) u , then
[Equation]
Now we assign 1 , 2 , . . .
, MHSA as indices to all ( m, h, s, a ) ∈ { 1 , . .
, M }×{ 1 , . . .
, H }×S ×A , and set
[Equation]
where j is the index assigned to ( m, h, s, a ) .
Then F ( θ ) =
Φ D Φ ⊤ and ∇ θ V θ = Φ Dq where
[Equation]
We now conclude the proof by utilizing Lem.
E.2. Auxiliary lemmas used in the main results
Lemma 14 (Performance Difference Lemma)
For any two policies π 1 and π 2 , and any 1 ≤ m ≤
M ,
[Equation]
As a result,
[Equation]
Proof First we fix s 0 .
By definition of the value function
[Equation]
we have
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
, where the last step uses law of iterated expectations.
Since
E
[Equation]
, we have
[Equation]
By taking expectation over s
[Equation]
we have
[Equation]
The proof is completed by reversing the order of h .
Lemma 15 (Lyapunov Drift) Recall definitions in Def. 8 and 5.
We have that:
[Equation]
Proof Denote Φ t :=
Φ( π t ) .
This proof follows a similar manner as in that of Lem.
6 in Cayci et al.
By smoothness (see Rem.
6.7 in Agarwal et al.
[Equation] [Equation]
By the definition of Φ ,
[Equation]
By the definition of err t
[Equation]
Lem.
14 and again the definition of Φ
[Equation]
we finally have
[Equation]
which completes the proof.
Lemma 16 Recall that g ⋆t is the true minimizer of L ( g ; θ t , d t ) in domain G .
err t defined in Def.satisfies
[Equation]
Proof The proof is similar to that of Thm.
6.1 in Agarwal et al.
We make the following decomposition of err t :
[Equation]
Since (cid:80) Mm =1 w m (cid:80) H − 1 h =0 (cid:80) ( s,a ) ∈S×A d ⋆m,h ( s, a ) =
H , normalize the coefficients and apply Jensen’s inequality, then ① ≤ (cid:118)(cid:117)(cid:117)(cid:116) M (cid:88) m =
w m
H
[Equation]
(cid:88) h
= (cid:88) ( s,a )
[Equation]
d
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
Similarly,
[Equation]
where in (i)
[Equation]
for vector v
[Equation]
denote ∥ v ∥ A = √ v ⊤ Av for a symmetric positive semi-definite matrix A .
Due to that g ⋆t minimizes L ( g ; θ t , d t ) over the set G
[Equation]
the first-order optimality condition implies that
[Equation]
for any g .
Therefore,
[Equation]
g ; θ t
[Equation]
d t
[Equation]
So finally we have
[Equation]
This completes the proof.
E.3.
Bounding ϵ stat
Lemma 17 (Hoeffding’s Inequality) Suppose X 1 , X 2 , . . .
, X n are i.i.d.
random variables taking values in [ a, b ] , with expectation µ .
Let ¯ X denote their average, then for any ϵ ≥ 0 ,
[Equation]
Lemma
For any policy π , any state s ∈ S and any U ≥ ln |A| − 1 ,
[Equation]
Proof The first inequality is straightforward, so we focus on the second part.
Set A ′ =
{ a ∈ A : ln 1 π ( a | s ) > U } = { a ∈ A : π ( a | s ) < 1 e U } and p = (cid:80) a ∈A ′ π ( a | s ) , then (cid:88) a ∈A
[Equation]
where the penultimate step comes from concavity of ln x and Jensen’s inequality.
Let f ( p ) =
p ln |A| p − pU , then f ′ ( p ) =
ln |A| − U − 1 − ln p .
Recall that U ≥ ln |A| − 1 , so f ( p ) increases when p ∈ (0 , |A| e U +1 ) and decreases when p ∈ ( |A| e U +1 , 1) .
Since f ( |A| e U +1 )
=
|A| e U +1 we complete the proof.
Lemma 19 (Loss Function Concentration)
If set π s =
None and U ≥ ln |A| − 1 , then with probability 1 − 2( T + 1) exp (cid:16) − 2 Nϵ 2 C 2 (cid:17) , the update weight sequence of Alg.
1 satisfies: for any 0 ≤ t ≤ T ,
[Equation]
where
[Equation]
If π s ̸
=
None and λ = 0 , then with probability 1 − 2( T + 1) exp (cid:16) − 2 Nϵ 2 C 2 (cid:17) , the update weight sequence of Alg.
1 satisfies: for any 0 ≤ t ≤ T ,
[Equation]
where
[Equation]
Proof We first prove the π s =
None case.
For time step t , Alg.
1 samples HN trajectories.
Abusing the notation, denote
[Equation]
Notice that ① is a constant.
From Alg. 1, (cid:98) g t is the minimizer of ② (hence (cid:98) L ( g ) ) inside the ball G .
From ∇ θ ln π θ ( a | s ) =
ϕ ( s, a ) − E
a ′ ∼ π θ ( ·| s )
[ ϕ ( s, a ′ )] , ∥ ϕ ( s, a ) ∥ 2 ≤ B, ∥ g ∥ 2 ≤ G , we know that (cid:12)(cid:12)
g ⊤ ∇ θ ln π θ ( a | s ) (cid:12)(cid:12) ≤ 2 GB .
So 0 ≤ g ⊤ (cid:98) F t g ≤ 4 HG 2 B 2 .
From Alg. 2, we know that any sampled (cid:98) A satisfies | (cid:98) A | ≤ 2[1 + λU + H (1 + λ ln |A| )] .
So | g ⊤ (cid:98) ∇ t | ≤ 4 HGB [1 + λU + H (1 + λ ln |A| )] .
We first have that
[Equation]
To apply any standard concentration inequality, we next need to calculate the expectation of ② .
According to Monte Carlo sampling and Lem.
18, for any 1 ≤ m ≤ M, 1 ≤ h ≤ H and ( s, a ) ∈ S × A , we have
[Equation]
Denote ∇ t as the exact policy gradient at time step t , then
[Equation]
Since Monte Carlo sampling correctly estimates state-action visitation distribution, E (cid:104) (cid:98) F t (cid:105)
=
F ( θ t ) .
Notice that g ⊤ (cid:98) F t g is linear in entries of (cid:98) F t , we have E (cid:104) g ⊤ (cid:98) F t g (cid:105) =
g ⊤ F ( θ t ) g .
Now we are in the position to show that
[Equation]
Hoeffding’s inequality (Lem. 17) gives
[Equation]
where from Eq. 2,
[Equation]
After applying union bound for all t , with probability 1 − 2( T + 1) exp (cid:16) − 2 Nϵ 2 C 2 (cid:17) the following holds for any g ∈ G :
[Equation]
Hence
[Equation]
For π s ̸
=
None and λ = 0 , we notice that | (cid:98) A | ≤ 2 H and hence − 8 H 2 GB ≤ ② ≤ 8 H 2 GB +
HG 2 B 2 .
Moreover, E (cid:104) (cid:98)
A t,λm,h ( s, a ) (cid:105) =
A t,λm,h ( s, a ) .
So by slightly modifying the proof we can get the result.
Abbreviation
Detailed setup
Script fix samp curl
Fix ed samp ler cur riculum l earning.
In the warm-up phase, train a policy π s from scratch (with zero initialization in parameters) using a small environment E ′ .
In the final phase, change to the true environment
E , use π s as the sampler policy to train a policy from scratch.
Run
Alg.with samp = pi s and λ = 0 .
fix samp curl reg The same as fix samp curl , but add entropy reg ularization to both phases.
Run
Alg.with samp = pi s and λ ̸ = 0 .
direct
Direct learning.
Only the final phase.
Train a policy from scratch directly in E .
Run
Alg.with θ 0 =
d , π s =
None and λ = 0 .
direct reg
The same as direct , but add entropy reg ularization.
Run
Alg.with θ 0 =
d , π s =
None and λ ̸ = 0 .
naive samp
Learning with the na¨ıve samp ler.
Only the final phase.
Use the na¨ıve random policy as the sampler to train a policy from scratch in E .
Run
Alg.with θ 0 =
d , π s = na¨ıve random policy and λ = 0 .
naive samp reg
The same as naive samp , but add entropy reg ularization.
Run
Alg.with θ 0 =
d , π s = naive random policy and λ ̸ = 0 .
curl
Cur riculum l earning.
In the warm-up phase, train a policy π s from scratch in E ′ .
In the final phase, change to E and continue on training π s .
Run
Alg.with samp = pi t and λ = 0 .
curl reg
The same as curl , but add entropy reg ularization.
Run
Alg.with samp = pi t and
[Equation]
reference
This is the reference policy.
For SP, it is exactly the optimal policy since it can be calculated.
For OKD, it is a bang-per-buck policy and is not the optimal policy (whose exact form is not clear).
N/A