L AGRANGIAN M ETHOD FOR E PISODIC L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper considers the problem of learning optimal value functions for ﬁnitetime decision tasks via saddle-point optimization of a nonlinear Lagrangian function that is derived from the Q -form Bellman optimality equation .
Despite a long history of research on this topic in the literature, previous works on this general approach have been focusing on a linear special case known as the linear programming approach to RL/MDP .
Our paper brings new perspectives to this general approach in the following aspects: 1) Inspired by the usually-used linear V -form Lagrangian, we proposed a nonlinear Q -form Lagrangian function and proved that it enjoys strong duality property in spite of its nonlinearity.
The Lagrangian duality property immediately leads to a new imitation learning algorithm, which we applied to Machine Translation and obtained favorable performance on standard MT benchmark.
We pointed out a fundamental limit in existing works which exclusively seeks to ﬁnd the minimax-type saddle points of the Lagrangian function.
We however proved that another class of saddle points, the maximin-type ones, turn out to have better optimality property.
In contrast to most previous works, our theory and algorithm are oriented to the undiscounted episode-wise reward, which is practically more relevant than the usually considered discounted-MDP setting, thus have ﬁlled a gap between theory and practice on the topic.
I NTRODUCTION
Episodic learning Terry (2017) is a general learning paradigm in which the agent learns based on data collected from a sequence of episodes of environmental interactions.
Each episode consists of a ﬁnite number of decision steps, and the goal is to maximize an episode-wise performance.
With one-step episodes, episodic learning subsumes supervised learning and bandit problems as special cases; for sequential decision making problems, an episode may contain multiple (and possibly varying number of) steps, in which case episode learning encompasses most reinforcement learning and imitation learning problems encountered in current AI practice.
A pervasive idea of episodic learning, often called the value-based approach Sutton and Barto (2018), seeks to ﬁnd a value function such that greedily selecting the actions with the best values (as prescribed by this function) gives good performance.
For example, in most classiﬁcation tasks, most deep neural networks are learned (and used) to represent such a value function.
Finding good value functions can be highly challenging, especially for tasks with multi-step episodes.
In this case, the classic Bellman optimality equation Bertsekas (2019) gives a sufﬁcient condition of optimal value function (i.e. value functions with which greedy decisions have optimal performance).
In this paper, we study a Lagrangian method which learns optimal value functions by characterizing them as saddle points of the Lagrangian function of a variational (re-)formulation of the Bellman equation.
The method reveals an elegant thoery on policy-value duality in machine learning.
A restricted form of this high-level idea has been known as the Linear Programming (LP) approach in the literature of dynamic programming Puterman (1994); Si et al.
; Chen and Wang (2016);
Wang (2017), which has also been introduced to reinforcement learning in recent years Dai et al.
; Lee and He (2018); Nachum et al.
; Yang et al.
; Nachum and Dai (2020).
This paper provides a new perspective to the foundation of this elegant idea through the following contributions:
Firstly, previous works are mostly limited to the discounted-reward setting, which bypass some technical difﬁculties Nachum and Dai (2020), but deviate from the practical settings of many (if not most) real-world tasks.
This paper ﬁlls this well-known gap between theory and practice by establishing a value-based theory (and algorithm) that targets at directly learning optimal value functions and policies for the undiscounted episode-wise reward (see (1) in Section 2).
Secondly, previous works focused on linear formulations, which enjoys the strong duality property and generic LP techniques, but are consequently limited to treatments for the state-value functions (i.e. V-functions) in policy optimization setting.
However, the optimal state-value function cannot be directly used to derive optimal policy in model-free learning.
In contrast, we proved a minimax theorem for the nonlinear Lagrangian functions associated with the action-value functions (i.e. Qfunctions).
The theorem opens a door to nonlinear variational treatment to the Bellman equation, which now enjoys the strong duality property too (see Section 3).
In particular, we developed a simple imitation learning algorithm based on the Lagrangian duality thus established, and applied the algorithm to Machine Translation (MT) as case study.
Transformer models trained by our algorithm achieves beam-search-level performance Vaswani et al.
(2017) with only greedy decoding, and leads to 1 . 4 -BLEU improvement when also equipping with beam search (see Section 4).
Last but not the least, previous works have exclusively focused on the primal-form optimization formulation of the Bellman equation, thus are limited to solving the minimax-type saddle points of the corresponding Lagrangian function.
We however showed that the minimax saddle points are not necessarily optimal value functions in learning settings.
Instead, another class of the Lagrangian saddle points – the maximin-type saddle points which are derived from the dual-form optimization formulation – turn out to have rigorous guarantee on optimality.
This observation points to new directions for this decades-old topic (see Section 5).
E PISODIC L EARNING P ROCESS AND B ELLMAN O PTIMAL V ALUE
We start with introducing the mathematical formulation of episode learning problems, as well as the formal deﬁnitions of related concepts.
An inﬁnite-horizon MDP is a tuple ( S , A , R, P, ρ 0 ) , where S is the state space,
A the action space,
R ( s ) ∈
[ r min , r max ] is a bounded reward (possibly negative) associated to each state s ∈ S ,
P ( s (cid:48) | s, a ) speciﬁes action-conditioned transition probabilities between states, and ρ 0 is the initial distribution with S 0 ∼ ρ 0 .
A MDP is ﬁnite if both S and A are ﬁnite sets.
A policy function π : S ×
A → [0 , 1] speciﬁes the action selection probabilities under each state, which induces a markov chain
P
π
[ S t +1 =
s (cid:48) | S t = s ] .
= (cid:80) a ∈A P ( s (cid:48) | s, a ) · π ( a | s ) .
Let Π denote the policy space , i.e., the set of all policies.
Without loss of generality, we assume every state s in the state space S is reachable from the initial state under at least one policy, 2 where reachable means ∃ π ∈ Π , (cid:80)
∞ t =
P π [ S t = s ] > 0 .
An Episodic Learning Process (ELP) White (2017); Bojun (2020) is an inﬁnite-horizon MDP that repeatedly encounters into, and is reset by, a group of terminal states S ⊥ (despite its name, a terminal state does not terminate the ELP).
Formally, an inﬁnite-horizon MDP is an ELP if there is a non-empty subset S ⊥ ⊆ S such that (1) all terminal states have homogeneous and action-agnostic outbound probabilities: P ( s (cid:48) | s 1 , a 1 ) =
P ( s (cid:48) | s 2 , a 2 ) , ∀ s 1 , s 2 ∈ S ⊥ , ∀ a 1 , a 2 ∈ A , ∀ s (cid:48) ∈ S , (2) the initial state is a terminal state:
ρ 0 ( s ) = 0 , ∀ s (cid:54)∈ S ⊥ , and (3) the average episode length is ﬁnite under any policy: ∀ π ∈ Π , E π [ T ] < ∞ , where T .
= inf { t ≥ 1 : S t ∈ S ⊥ } is called the termination time .
Bojun (2020) proved that in any ELP, every policy π ∈ Π has a unique stationary distribution ρ π ( s, a ) such that P π [ S t +
= s, A t +1 = a ]
=
ρ π ( s, a ) if P π [ S t = s, A t = a ]
=
ρ π ( s, a ) .
Following White (2017) and Bojun (2020), we use the ELP formalism to study ﬁnite-time decision tasks, which arguably account for most AI tasks encountered in current practice.
The ELP model formulates the learning process of such task, which consists of an inﬁnite number of concatenated episodes, each starting and ending at an terminal state.
In general, an episode can be arbitrarily long, but will terminate in ﬁnite steps with probability 1 .
The learning agent may choose to use a different behavior policy β t for each step t of the episodic learning process (based on data collected from previous steps), and the goal is to have β t converge (as t goes to inﬁnity) towards an optimal policy that maximizes the expected episode-wise cumulative reward:
[Equation]
where ζ = ( S 0 , S 1 , S 2 , . . . ) is an inﬁnite trajectory of the markov chain induced by policy π .
Note that the termination time T is a random variable whose value may vary for different trajectory ζ .
See Section D.3 for the ELP formulation of machine translation as an example.
A value function Q : S ×
A → R assigns a real number to each state-action pair ( s, a ) as the “perceived beneﬁt” of doing action a under state s .
We call the set of all possible value functions, the value space , denoted by Q .
A value function Q ∈ Q is called an optimal value if (any of the)
Q -greedy policy with π Q ( a | s ) > 0 ⇒ Q ( s, a ) =
max ¯ a Q ( s, ¯ a ) is an optimal policy.
Bellman optimal value is a special optimal value function that is characterized by the Bellman optimality operator.
In its general form Degris et al.
; Sutton et al.
(2011; 2014; 2016), a generalized Bellman optimality operator B γ : Q → Q transforms a value function Q ∈ Q into another value function B γ Q ∈ Q such that for any ( s, a ) ∈ S × A ,
[Equation]
where γ : S → [0 , 1] is a discounting function over the states.
As a classic result, if the discounting function equals a constant γ c < 1 on every state, the corresponding Bellman optimality operator B γ c has a unique ﬁxed point in the value space of any MDP.
However, the ﬁxed point of B γ c is generally not an optimal value with respect to the undiscounted objective (1).
In the following, we present a new theorem that guarantees the uniqueness of Bellman ﬁxed point for a more general class of discounting functions, among which a particular “episodic discounting function” gives optimal value w.r.t.
objective (1).
Both the uniqueness and the optimality are fundamentally rooted from an inherent graph property of ELP-MDPs.
See the proof in
Appendix A for detailed elaboration.
Theorem 1.
In any ﬁnite Episodic Learning Process ( S , A , P, R, ρ 0 ) , let γ be any discounting function such that γ ( s ) < 1 for all terminal state s ∈ S ⊥ , then
(1) B γ has a unique ﬁxed point, i.e., the equation Q = B γ Q has a unique solution.
The ﬁxed point of B γ is the limiting point of repeatedly applying B γ to any Q ∈ Q .
The ﬁxed point of B γ is an optimal value w.r.t.
objective (1) if γ is the following episodic discounting function :
[Equation]
Since we focus on optimizing objective (1), in the rest of the paper: Bellman optimality operator , denoted by B .
=
B γ epi , refers to the speciﬁc operator (2) that uses the particular episodic discounting function (3); accordingly, Bellman optimal value , denoted by Q ∗ , refers to the unique ﬁxed point of B ; and similarly, Bellman optimality equation refers to the ﬁxed-point equation Q = B Q under episodic discounting, or more explicitly, refers to the following system of non-linear equations:
[Equation]
It is worthwhile noting that although the Bellman optimal value function is unique, there can be many optimal value functions in an episodic learning problem.
In particular, any value function that gives the same preference order with the Bellman optimality value is an optimal value function.
L AGRANGIAN D UALITY AND M INIMAX T HEOREM
In this section we study a minimization-based variational treatment to the Bellman optimality equation (4).
We will derive a practical episodic learning algorithm based on the theory presented here, and will apply the algorithm to standard machine translation benchmarks in the next section.
Our idea is inspired by the long-known linear programming re-formulation of the closely related optimality equation for state-value functions V : S → R ,
[Equation]
Both the Q -form optimality equation (4) and the V -form optimality equation (5) are nonlinear due to the max operation inside, but the V -form equation (5) admits a linear re-formulation Puterman (1994):
[Equation]
Thanks to its linearity, (6) enjoys the standard LP duality properties, and in particular has minimax equality for its corresponding Lagrangian function, which is the basis for a recently revived thread of research on the LP approach to MDP and RL Chen and Wang (2016);
Wang (2017); Cho and
Wang (2017); Dai et al.
; Nachum and Dai (2020).
In model-free learning settings, however, the agent does not know the transition function P , thus cannot directly use a state-value function V to compare the values of different actions (as we do not know which states an action would lead to).
For this reason, most value-based algorithms focus on learning the optimal Q functions Watkins (1989);
Hasselt (2010); Mnih et al. (2015); Haarnoja et al.
It is thus natural to ask if we can develop a variational approach directly for the Q -form optimality equations (4), similar to what we did to the V -form optimality equations.
Indeed, the Q -form optimality equations (4) can be similarly recast into a constrained optimization problem as follows:
[Equation]
where π ∈ Π can be an arbitrary policy, and T .
= inf { t ≥ 1 : S t ∈ S ⊥ } is the termination time.
Unfortunately, unlike the V -form optimality equation for which the nonlinear operation max a can be “unpacked” into |A| linear constraints (cf. (5) and (6)), the variational formulation (7) of the
Q -form optimality equation is still nonlinear, due to the max a (cid:48) “wrapped” inside (cid:80) s (cid:48) .
As a result, although (7) can still be written into the Lagrangian form
[Equation]
it is unclear if the nonlinear Q-form Lagrangian still enjoys strong duality, a key property for designing effective and principled learning algorithms Wang (2017); Dai et al.
In the following, we give an afﬁrmative answer to this open question by proving a minimax theorem for the nonlinear Lagrangian of the Q -form Bellman optimality equation (4).
As with Theorem 1, the strong Lagrangian duality is also rooted from inherent structures of the episodic learning process.
Deﬁnition.
Given a ﬁnite Episodic Learning Process ( S , A , P, R, ρ 0 ) , a function L π : Q ×
R |S|·|A|≥→ R is called a Lagrangian function with conjugate policy π if
[Equation]
where π ∈ Π can be an arbitrary policy, and T .
= inf { t ≥ 1 : S t ∈ S ⊥ } is the termination time.
Note that the conjugate policy π only determines the distribution of terminal states (and actions) in the ﬁrst term of L π .
With any conjugate policy π , the second term of L π always uses the Bellman optimality operator B .
Let us ﬁrst conﬁrm that Q ∗ is a minimax solution of the Lagrangian function (9), as Lemma 2.1 states.
The proof can be found in Appendix B.1.
Lemma 2.1.
In any ﬁnite ELP, for any conjugate policy π , Q ∗ ∈ arg min Q ∈Q max λ ≥ 0 L π ( Q, λ ) .
Now, we observe that the Lagrangian L π has an equivalent form when the Lagrangian multiplier vector is a special vector λ π that is proportional to the stationary distribution of the conjugate policy π (and proportional to the average episode length of π too).
Lemma 2.2.
In any ﬁnite Episodic Learning Process ( S , A , P, R, ρ 0 ) , for any conjugate policy π , let L π be the corresponding Lagrangian, and let λ π be the particular Lagrangian multipliers with λ π ( s, a )
=
ρ π ( s, a ) · E ζ ∼ π
[ T ] , where ρ π is the stationary distribution of π , then
[Equation]
Proof idea:
Applying a known ergodic theorem of ELP, we can transform the ﬁrst term of (9) from an average over trajectories to an average over the state-action space (see Appendix B.2 for details):
[Equation]
Then substituting the above equation to (9), yields
[Equation]
In above, we have again used the ergodic theorem of ELP to transform E π [ T ] · E S ∼ ρ π [ R ( S )] back to the trajectory space, so as to derive J ( π ) .
See Appendix B.2 for the complete proof.
The ﬁrst term in the dual form of the Lagrangian, i.e. in (10), is the true performance of the conjugate policy π (in terms of objective (1)).
Utilizing this fact, we can prove the strong duality property for
Lagrangians conjugating with optimal policies, as the following theorem states.
Theorem 2 (ELP Minimax Theorem) .
In any ﬁnite Episodic Learning Process ( S , A , P, R, ρ 0 ) , if µ ∈ Π is an optimal policy, then its conjugate Lagrangian L µ has strong duality property, with
[Equation]
Proof idea: Let π ∗ denote a Q ∗ -greedy policy, which is thus an optimal policy.
For any conjugate policy π , since Q ∗ is a minimax solution of L π (Lemma 2.1), we have [-min Q max λ ≥ 
-][Equation]
[-L π ( Q, λ ) = max λ ≥ 
 L π ( Q ∗ , λ ) = E π [ Q ∗ ( S T , A T )] = E π [ Q π ∗ ( S T , A-] T )] .
Because J ( π ∗ ) =
Q π ∗ ( s ⊥ , a ) for any terminal state s ⊥ ∈ S ⊥ and any action a ∈ A (see Appendix
A.3), we have E π [ Q π ∗ ( S T , A T )]
=
J ( π ∗ ) , and so min Q max λ ≥
L π ( Q, λ ) =
J ( π ∗ ) .
Again for any conjugate policy π , due to Lemma 2.2, the Lagrangian dual L π ( Q, λ π ) for the particular multiplier λ π attains its minimum when Q achieves complementary slackness with λ π in the second term of (10), in which case min Q ∈Q L π ( Q, λ π ) =
J ( π ) .
So now, when the conjugate policy π is an optimal policy µ , as assumed in the theorem, we have
[Equation]
The inequality above must actually be an equality because of the universally-held weak duality of the Lagrangian.
See Appendix B.3 for a more detailed proof.
From the proof above we can also see that ( Q ∗ , π ∗ ) , as the ﬁxed point of Bellman optimality operator
B and the corresponding Q ∗ -greedy policy (resp.), forms a minimax saddle point of the Lagrangian
L µ .
In fact, we can derive general conditions for all saddle points of this kind by combining the complementary slackness conditions in (9) and (10).
See the proof in Appendix B.4.
Proposition 3.
In any ﬁnite ELP, for any value function Q ∈ Q and any policy π ∈ Π , let λ π ( s, a ) =
ρ π ( s, a ) · E ζ ∼ π
[ T ] , then L π ( Q, ¯ λ ) ≤ L π ( Q, λ π ) ≤ L π ( ¯ Q, λ π ) , ∀ ¯ Q, ¯ λ , if and only if
[Equation]
≤, ∀ ( s, a ) ∈ S
× A (2) ρ π ( s, a ) · (cid:16) B Q ( s, a ) − Q ( s, a ) (cid:17) =, ∀ ( s, a ) ∈ S × A (3) ρ π ( s, a ) · (cid:16) max ¯ a Q ( s, ¯ a ) − Q ( s, a ) (cid:17) =, ∀ s (cid:54)∈ S ⊥ , a ∈
A  A LGORITHMIC A PPLICATION TO M ACHINE T RANSLATION
From last section we know that the Q -form Lagrangian L µ has strong duality where µ is optimal policy, and that the special Lagrangian multiplier λ µ forms a minimax saddle point (or minimax equilibrium) with the solution of (7).
Formally,
[Equation]
A simple idea to ﬁnd such minimax saddle point is to minimize the rightmost side of the above equation, that is, to minimize L µ ( Q, λ µ ) over the value space Q .
Although the closed form of the objective function L µ ( Q, λ µ ) depends on an optimal policy µ (which might appear paradoxical at a ﬁrst glance as coming up with such an optimal policy was our original goal of learning), note that estimating the gradient of L µ ( Q, λ µ ) only requires knowledge about a “trace” of the optimal policy µ , instead of explicit knowledge about how µ is constructed.
Speciﬁcally, for parameterized value function Q ( s, a ; w ) , we want to estimate ∇ w L µ ( Q ( w ) , λ µ ) , that is,
[Equation]
[ Q [Equation]
[Equation] [Equation] ] + E µ [ [Equation]
[Equation] [Equation] in which the optimal policy µ only plays a role in data weighting – it determines the population distribution of the terminal state-actions ( S T , A T ) , of the terminal time T , and of the transition variable ( S, A, S (cid:48) ) .
This observation inspires a general imitation learning approach, named LAgrangian MINimization (LAMIN) here, in which we try to collect some demonstration data from an optimal policy, based on which we construct an estimator of the Lagrangian gradient (12), then apply standard stochastic gradient procedures to approximate the value function that minimizes L µ ( Q ( w ) , λ µ ) , which corresponds to a minimax point of the original Lagrangian function (9). One may design different estimators of (12) to form different algorithms under the LAMIN idea. In the following we discuss two, and they both perform reasonably well in our validation experiment.
LAMIN1 : A technical challenge of estimating (12) is to deal with the max operator in it.
One idea is to relax the Lagrangian function into a smoothed version:
[Equation] w [Equation] where π βQ ( w ) ( a | s ) .
=
[Equation] Q ( s,b ; w ) /β (cid:1) is the Boltzmann distribution with temperature β , and δ ( s, a, s (cid:48) , a (cid:48) ; w ) . = R ( s (cid:48) )
+ γ epi ( s (cid:48) ) Q ( s (cid:48) , a (cid:48) ; w ) − Q ( s, a ; w ) is the temporal-difference error. The smoothed Lagrangian L βµ subsumes the original Lagrangian L µ as the limiting case β → 0 , while is readily differentiable. Thus, the LAMIN1 algorithm seeks to minimize the smoothed Lagrangian L βµ via SGD with unbiased gradient estimator of (13).
See the pseudo-code as well as more nuanced discussion on this algorithm in Appendix D.1.
Among others, it turns out that the β -smoothing trick used in LAMIN1 is more than an approximation heuristic, but may potentially serve as a practical correction to an inherent bias of the minimax points of the Lagrangian that we will discuss in Section 5.
LAMIN2 : Another idea is to stick to the exact Lagrangian L µ but seek to construct “local” estimator of its gradient in a per-step manner.
Speciﬁcally, let w t be the parameter vector of Q that is to be updated at a gradient step t , then we can estimate the value of ∇ w L µ ( Q ( w ) , λ µ ) (cid:12)(cid:12) w = w t , for this particular paramter w t , with
[Equation]
where π Q ( w ∗ ) is the Q ( w ∗ ) -greedy policy (see Section 2).
Note that the greedy policy π Q ( w ∗ ) in (14) does not depend on w and thus is invariant to ∇ w ; in contrast, the Boltzmann policy π βQ ( w ) used by LAMIN1 (see (13)) will take part in the gradient computation.
The consistency of the estimator (14) is characterized by the following mathematical fact.
Proposition 4.
Let A be a ﬁnite action space, Q ( s, a ; w ) a parameterized value function.
If with a given parameter vector w ∗ , Q ( w ∗ ) suggests a unique best action a max ( s ; w ∗ ) . = arg max a ∈A Q ( s, a ; w ∗ [Table] See the proof in Appendix D.2.
In practice, the condition in Proposition 4 – i.e. Q ( w ∗ ) gives unique best actions – may be expected for large-scale continuously-valued Q functions, as it is unlikely that the real-numbered values of two actions happen to be identical under stochastically generated w . Moreover, the local gradient estimator (14) can be combined with the β -smoothing trick by replacing the greedy policy π Q ( w ∗ ) in (14) with Boltzmann policy π βQ ( w ∗ ) , forming the smoothed local gradient estimator that the LAMIN2 algorithm will use. See the pseudo-code in Appendix D.2. Empirically, we found that higher temperatures help slightly improve the performance of LAMIN2.
We now apply LAMIN algorithms to Machine Translation (MT) as a case study.
Learning to translate is a highly impactful AI application Wu et al.
(2016), and is also an excellent example of episodic learning problem: A translation episode starts with a given sentence in a source language, and the agent takes actions to generate translation tokens one by one in a sequential manner.
The episode terminates when the agent outputs a special end-of-sentence (EOS) token, at which point the translation quality is evaluated based on the source sentence and the generated translation sentence.
Note that in MT, the episode length (i.e. the length of the translation sentence) is a variable controlled by the agent policy, and the episode-wise reward (i.e. the translation quality) is generally not discounted for long episodes. Just the opposite, standard MT metrics such as BLEU Papineni et al.
(2002) uses brevity penalty factor to discount the reward of short (instead of long) translations/episodes. See Appendix D.3 for the exact episodic learning formulation of MT.
Given a source sentence X , most MT metrics measure the quality of a generated translation Y by comparing the similarity between Y and some reference translation Z of X , where Z is typically provided by human expert given X .
It follows that the translation policy used by the human expert (which maps X to Z ) must be an optimal policy under such metric.
Importantly, a trace of the optimal translation policy, in the form of a collection of source-reference sentence pairs, is indeed widely available in standard MT benchmarks, which can be readily used to power LAMIN algorithms.
In general, the same idea applies to all machine learning problems with imitation reward where the performance metric is based on similarity comparison to reference/ground-truth outputs.
We tested our algorithmic ideas using the WMT English → German (en2de) dataset, one of the most inﬂuential MT benchmarks.
We parameterized value function Q with the standard TransformerBase neural network Vaswani et al.
and trained the Transformer-based Q-function using LAMIN and LAMIN2, with varying temperature β , then tested the Q-greedy policy with standard BLEU metric using the WMT’14 NewsTest data.
See Appendix D.4 for details in experimentation setup.
Figure 1 illustrates the performance of LAMIN1 and LAMIN2 under a spectrum of Boltzmann temperatures, from 0 .
01 (in which case the algorithm is effectively minimizing the exact Lagrangian function) to 1 . {+
+}
(which corresponds to the softmax distribution). Table 1 and 2 in Appendix D. reports the numerical scores. We see that LAMIN1 has slightly better performance than LAMIN  under most temperatures. Interesting, higher temperatures tend to help the performance of LAMIN but hurt that of LAMIN1 (albeit with limited margins in both cases).
Notably, LAMIN1 with β = 0 . 01 attains a BLEU score of 27 .
4 with a greedy policy .
In comparison, when standard supervised learning is used, the same neural network famously attains 27 .
3 only if further combined with a systematic beam search Vaswani et al.
So, our algorithm achieves comparable performance with the state-of-the-art result without time-consuming search procedure at decision/translation time.
On the other hand, when incorporating with the same beam search procedure, the value function trained by LAMIN1 (with β = 0 . 01 ) attains 28 . 7 , or 1 . 4 BLEU score higher.
Figure 2 shows the learning curves of the algorithm, with and without beam search.
M INIMAX V ALUES VS M AXIMIN V ALUES
The Lagrangian method we explored so far has focused on a particular class of Lagrangian saddle points, the minimax saddle points arg min Q max λ L π , which correspond to exactly the set of optimal solutions for the minimization-form (or primal-form ) variational problem (7).
Let’s call such a value function, a minimax value function .
It seems that existing research on the Lagrangian method have been exclusively focusing on studying minimax value functions (or minimax points of the Lagrangian); see Puterman (1994); Cho and Wang (2017); Dai et al.
; Nachum and Dai (2020) for examples.
Indeed, as Lemma 2.1 conﬁrmed, the Bellman-optimal value Q ∗ , as an optimal value function, is a minimax value function.
However, a minimax value function, or equivalently, a minimax solution of L π (for any π ), is not necessarily an optimal value function.
Figure 3 shows an example, where state 4 and 5 are terminal states, from which any action leads to [Equation]
, 𝒓 =𝒓 =𝒓 =𝒓 = 𝒓 =
𝒓
= 𝝆 state 0 . Choosing action 1 , 2 , 3 under state 0 deterministically transits to state 1 , 2 , 3 , respectively.
All actions under state 1 lead to state 4 , and all actions under state 2 and 3 lead to state 5 .
The agent only receives non-zero rewards at terminal states, with R (4)
= 1 , R (5)
= 2 .
The initial state is set to state 4 .
Clearly, an optimal policy of this ELP should only choose action 2 or 3 (or both), but not action 1 , under state 0 .
One can verify that for this ELP, the constant value function Q min ( s, a ) ≡ 2 is a minimax value function.
In particular, we have 2 = Q min (0 , 1) =
+ 1 · Q min (1 , a ) > 1 + 0 · Q min (4 , a )
= 1 , which does satisfy the constraints of (7) (that Q ≥ B Q ).
But the constant value function Q min is certainly not optimal as it assigns the same value to all actions under state 0 .
Moreover, because Q min (1 , a ) > B Q min (1 , a ) = 1 for all a , the λ corresponding to Q min needs to have λ (1 , a ) =
for all a .
Such a λ cannot encode any policy.
The problem with minimax value functions is that they only guarantee optimality for optimal actions, but do not enforce sub-optimality for sub-optimal actions (such as the action 1 under state 0 ).
This inadequacy of minimax value functions may explain our experimental observation in last section that adding a softmax temperature in the Lagrangian leads to better value functions (because the softmax averaging motivates, to some extent, the optimizer to downgrade sub-optimal actions to further lower the Lagrangian).
Moreover, we remark that this is not only a problem for Q -functions or for episodic discounting, but the minimax solutions of the more popular V -form Lagrangian in discounted-MDPs Dai et al.
(2018) suffer from the same issue too (see Appendix C.4 for a counterexample for the minimax V -functions in discounted setting).
Interestingly enough, it turns out that value optimality is indeed guaranteed for another class of saddle points of the Lagrangian – the maximin saddle points .
Speciﬁcally, with the same argument as in Lemma 2.1 one can show that Q ∗ , the ﬁxed point of B , is also the optimal solution of the following “mirrored” problem to (7).
See Appendix C.1 for the proof.
Lemma 5.1.
In any ﬁnite ELP, Q ∗ is an optimal solution of
[Equation]
for any conjugate policy π ∈ Π .
Equivalently, Q ∗ ∈ arg max Q ∈Q min λ ≥
L π ( Q, λ ) .
We call an optimal solution of (15), a maximin value function (with respect to the Lagrangian L π ).
Different from minimax value functions, a maximin value function enforces sub-optimality for all (truly) sub-optimal actions, and at the same time can guarantee optimality for at least one (truly) optimal action.
Consequently, a maximin value function always and only induces optimal policy.
Theorem 5.
In any ﬁnite ELP, for any conjugate policy π , let Q max be an maximin value function of the Lagrangian L π – i.e. let Q max be an optimal solution of (15) – then Q max is an optimal value function, in the sense that Q max -greedy policy maximizes the episodic-reward objective (1) .
Proof idea: First observe that Q max ( s, a ) ≤ Q ∗ at all ( s, a ) .
This is because Q max , as a feasible solution of (15), has Q max ≤ B Q max ≤ BB Q max · · · ≤ Q ∗ .
Now Q max ≤ Q ∗ implies that for all actions sub-optimal to Q ∗ (under a state), they can only have even lower values in Q max .
So, it is enough to prove that max a Q max ( s, a ) =
max a Q ∗ ( s, a ) at every non-terminal state s that is reachable by Q max -greedy policy – in that case the footprint of a Q max -greedy policy will be a subset of the footprint of a Q ∗ -greedy policy, in which only Q ∗ -optimal actions are selected.
Note that Q max ’s values on terminal states do not affect its footprint, due to the ELP conditions.
For an arbitrary state-action pair ( s, a ) , let ( S × A ) next denote the set of all the non-terminal ( s (cid:48) , a (cid:48) )
pairs that can directly follow ( s, a ) under the Q max -greedy policy.
We can prove that
[Equation] ( s, a ) ⇒ Q max ( s (cid:48) , a (cid:48) )
=
Q ∗ ( s (cid:48) , a (cid:48) ) and max ¯ a Q max ( s (cid:48) , ¯ a ) =
max ¯ a Q ∗ ( s (cid:48) , ¯ a ) , ∀ ( s (cid:48) , a (cid:48) ) ∈ ( S × A ) next which would enable a proof-by-induction, starting from some terminal state where Q max and Q ∗ equal to each other (such terminal state exists because both Q max and Q ∗ are optimal solutions of (15)).
See Appendix C.2 for the complete proof.
In light of the symmetry breaking between the minimax-type and the maximin-type Lagrangian saddle points (in terms of optimality), we believe the latter, as well as minimax value functions and the corresponding dual-form variational problem (15), deserve more attentions from the community.
R ELATED WORKS
A main challenge of this work comes from the non-linearity in both the Bellman optimality equation and the associated Q -form Lagrangian being studied.
In contrast, most related research focused on linear treatments to related objects.
For a linear policy-speciﬁc Bellman operator B π (which replaces max a with a linear average), White (2017) proved that B π has unique ﬁxed point in episodic learning setting. The linear operator B π leads to a LP (re-)formulation, based on which the “DICE” family of policy evaluation algorithms were developed Nachum et al.
; Yang et al.
On the policy optimization side, an active thread of research used saddle-point optimization to solve the linear Vform Lagrangian, again relying on the generic LP duality inherited from the linear treatment Chen and Wang (2016);
Wang (2017); Cho and Wang (2017); Dai et al.
; Chen et al.
; Serrano and Neu (2020); Si et al.
The underlying techniques in these linear settings are not directly applicable to our nonlinear treatment in this work.
The disparity between discounted formalism and episodic learning practice is well recognized in reinforcement learning literature.
The RL textbook Sutton and Barto (2018) devoted its Section 10.4 to the issue of deprecating the discounted formalism.
The DP textbook Bertsekas and Tsitsiklis (1996) subsumed the discounted setting as a special case of an ﬁnite-termination setting.
A special case of Theorem 1 dedicated to episodic discounting was proved by Bertsekas and Tsitsiklis (1991).
The WMT machine translation benchmark used in this paper is a fruitful driver of the rapid technical advances in Neural Machine Translation in recent years Wu et al.
; Koehn and Knowles (2017); Vaswani et al.
MDP-based techniques have been actively studied as a promising method for this problem Ranzato et al.
; Edunov et al.
; Bahdanau et al. ; Wu et al. (2018), but with relatively limited effectiveness observed so far Choshen et al. To our best knowledge, the LAMIN algorithm is one of the ﬁrst MDP-based solutions that is able to train
Transformer-scale neural networks independently (without pretraining or ensemble learning with the aid of other techniques) to attain competitive performance on the WMT benchmark.
R EPRODUCIBILITY S TATEMENT
Every theorem, lemma, proposition in the main text or in the appendix of this paper has been given complete proof (unless already proved in other papers in which case the reference was given).
For the experiment part, detailed experimentation setting was described in Appendix D.4.
Experimentation code was attached in Supplementary Material
A P ROOFS OF B ELLMAN O PTIMALITY
In this section we prove the uniqueness and optimality of the solution of generalized Bellman equation in episodic learning setting.
A. P ROPERTIES OF E PISODIC L EARNING P ROCESS
This section presents a series of mathematical properties of ELP; the ﬁrst three is known, the rest are new.
Most of our theoretical results in this paper are based on these properties.
Proposition 6.
For any policy π in any ELP, the Markov chain induced by policy π is irreducible: [Equation]
[Equation]
Proposition 7.
For any policy π in any ELP, the Markov chain induced by policy π is positive recurrent: ∀ s ∈ S π , E π [ T s ] < ∞ , where T s is the recurrent time of s .
(Bojun (2020), Lemma 1.2)
Proposition 8.
For any policy π in any ELP, let f : S
→ R be a real-valued function over the states, we have (Bojun (2020), Theorem 4)
[Equation]
We write Q 1 ≥ Q 2 for two value functions Q 1 , Q 2 ∈ Q iff Q 1 ( s, a ) ≥ Q 2 ( s, a ) , ∀ ( s, a ) ∈ S × A .
First observe that the generalized Bellman optimality operator (2) is monotonic .
Proposition 9.
For any discounting function γ : S → [0 , 1] , the generalized Bellman optimality operator B γ is a monotonic operator: Q 1 ≥ Q 2 ⇒ B
γ Q 1 ≥ B γ Q 2 , for all Q 1 , Q 2 ∈ Q .
Proof.
Rewrite (2) as B γ Q ( s, a ) =
(cid:80) s (cid:48)
(cid:16) P ( s (cid:48) | s, a ) · γ ( s (cid:48) ) (cid:17) · max a (cid:48) Q ( s (cid:48) , a (cid:48) )
+ (cid:80) s (cid:48)
P ( s (cid:48) | s, a ) R ( s (cid:48) ) .
As Q 1 ( s (cid:48) , a (cid:48) ) ≥ Q 2 ( s (cid:48) , a (cid:48) ) for all ( s (cid:48) , a (cid:48) ) , we have max a (cid:48) Q 1 ( s (cid:48) , a (cid:48) ) ≥ max a (cid:48)
Q 2 ( s (cid:48) , a (cid:48) )
for all s (cid:48) , and thus (cid:80) s (cid:48) (cid:16) P ( s (cid:48) | s, a ) · γ ( s (cid:48) ) (cid:17) · max a (cid:48)
Q 1 ( s (cid:48) , a (cid:48) ) ≥ (cid:80) s (cid:48)
(cid:16) P ( s (cid:48) | s, a ) · γ ( s (cid:48) ) (cid:17) · max a (cid:48)
Q 2 ( s (cid:48) , a (cid:48) )
because P ( s (cid:48) | s, a ) · γ ( s (cid:48) ) ≥ 0 for all s (cid:48) .
Now, as a well-known special case, if the discounting function is a constant less than 1 , the corresponding Bellman optimality operator is a contraction mapping with respect to the maximum-norm distance over the value space Q , which guarantees, by the Banach ﬁxed-point theorem, that there is a special value function Q ∗ c which is both the unique ﬁxed point ( Q ∗ c = B c Q ∗ c ) and the unique limiting point of the Bellman optimality operator ( Q ∗ c = lim n →∞ ( B c ) n Q , ∀ Q ∈ Q ).
Unfortunately, B γ loses the above contraction property under general discounting.
Proposition 10.
In any ELP where there is a single state s ∗ and a single action a ∗ such that doing a ∗ under s ∗ only goes to non-terminal states s (cid:48) , i.e. (cid:80) s (cid:48)
(cid:54)∈S ⊥ P ( s (cid:48) | s ∗ , a ∗ )
= 1 , the Bellman operator
B with episodic discounting (3) is not a contraction mapping with respect to the maximum-norm distance: For some Q 1 , Q 2 ∈ Q , max s,a | Q 1 ( s, a ) − Q 2 ( s, a ) | = max s,a |B Q 1 ( s, a ) − B Q ( s, a ) | .
Proof.
Consider two value functions with constant difference everywhere: Q 1 ( s, a ) ≡ Q 2 ( s, a )+ δ , for some δ > 0 .
Clearly, max s,a | Q 1 ( s, a ) − Q 2 ( s, a ) | = δ .
On the other hand, for any ( s, a ) we have |B Q 1 ( s, a ) − B Q 2 ( s, a ) | = (cid:80) s (cid:48) ∈S P ( s (cid:48) | s, a ) · γ ( s (cid:48) ) · δ ≤ δ , which equals δ at the particular ( s ∗ , a ∗ ) because (cid:80) s (cid:48) ∈S P ( s (cid:48) | s ∗ , a ∗ ) · γ ( s (cid:48) ) · δ = (cid:80) s (cid:48)
(cid:54)∈S ⊥ P ( s (cid:48) | s ∗ , a ∗ ) · 1 · δ = δ .
Proposition 10 seems to be a negative result for most non-trivial ELPs – it says B is not a contraction mapping unless we always has a chance to immediately terminate an episode no matter where we are (even when we are at terminal states, at which point the episode has not effectively started yet).
But interestingly, it turns out that the episodic Bellman optimality operator B still has unique ﬁxed and limiting point in all ﬁnite ELPs, not because of the contraction property as in discounted-MDPs, but because of a graph property dedicated to the family of ELPs, as will be shown next.
Lemma 2.1.
In any Episodic Learning Process ( S , A , P, R, ρ 0 ) , for any subset of states Ω ⊆ S and any policy π , let C π (Ω) .
= { s (cid:48) :
∃ s ∈ Ω , P π [ S t +1 =
s (cid:48) | S t = s ] > 0 } be the set of all successor states that are one-step reachable from Ω under π , and let S π be the set of states that are ever reachable under π (from initial states, in ﬁnite steps), then C π (Ω) ⊆
Ω only if S π ⊆ Ω .
Proof.
For contradiction suppose C π (Ω) ⊆
Ω and yet there is a π -reachable state s ∗ ∈ S π that is outside the given subset Ω .
We will show that in this case, it is possible to construct a policy µ (that is possibly different from π ) such that s ∗ is also reachable under µ , and that µ admits an inﬁnite trajectory that passes through s ∗ and never return back to s ∗ (see Figure 4 below).
This would contradict with Proposition 7 above which asserts that a µ -reachable state s ∗ must have ﬁnite mean recurrence time under µ .
[Figure]
Speciﬁcally, ﬁrst observe that C π (Ω) ⊆
Ω means Ω is a absorbing subset under π so that once we get into Ω we would never get out.
Then observe that the existence of a π -reachable state s ∗ (cid:54)∈ Ω entails that Ω cannot contain all initial states, as otherwise from any of the initial states (in Ω ) we cannot go outside the absorbing subset Ω to reach s ∗ .
Let s 0 be such an initial state that is outside Ω , from which we can reach s ∗ under π (as assumed) without reaching any state in Ω in the middle (otherwise we never reach s ∗ ).
Now, pick an arbitrary state s + in Ω , there must be some policy β under which we can reach s + from s 0 (as states unreachable under any policy should not show up in S in the ﬁrst place, see Section 2).
Without loss of generality we can again assume that we can reach s + from s 0 under β without going through any other state in Ω in the middle because otherwise we can simply re-deﬁne s + to be the ﬁrst state in Ω that we have encountered on the path (from s 0 to the “old s + ”).
So far we have obtained an initial state s 0 outside Ω , from which there is an admissible path s
π → s ∗ for policy π , and an admissible path s 0 β
→ s + for policy β .
Both paths only contain states outside Ω (except s + ).
Now we construct policy µ as follows: we ask µ to copy π on states in the path s
π → s ∗ , and ask µ to copy β on states in the path s 0 β
→ s + .
If a state shows up in both paths, we ask µ to be a (probability) mixture of both π and β on that state.
Clearly, we can reach both s ∗ and s + from s 0 under µ (note that the policy mixing only decreases the probabilities to reach s ∗ and s + but does not change their reachability).
By Proposition 6, since both s ∗ and s + are in S µ , we must be able to go from s ∗ to s + under µ .
Note that so far we have only prescribed µ ’s behavior outside Ω .
Our ﬁnal step is to ask µ to copy π for all states in Ω , so that Ω is also an absorbing subset for µ , meaning that once we reach s + (from s ∗ ), we will be stuck in Ω without going back to s ∗ (which is outside Ω by our assumption at the beginning of the proof).
In this way, we have constructed a policy µ , under which we can ﬁrst go from s 0 to s ∗ , then go from s ∗ to s + , and then be stuck in Ω forever without returning to s ∗ .
A possibility of such an inﬁnite trajectory under µ directly contradicts with Proposition 7.
Lemma 2.1 says that in ELPs, any absorbing subset of S under a policy must contain all reachable states of this policy.
In particular, there cannot exist an absorbing subset outside the reachable set S π .
Utilizing this fact, we can prove the ﬁrst two statements of Theorem 1.
(Theorem 1 (1) (2)) .
In any ELP ( S , A , P, R, ρ 0 ) with ﬁnite state space S and ﬁnite action space
A , let γ be any discounting function such that γ ( s ) < 1 for all terminal state s ∈ S ⊥ , then
(1) B γ has a unique ﬁxed point,
i
e
, the equation Q =
B
γ Q has a
unique solution.
The ﬁxed point of B γ is also the limiting point of repeatedly applying B γ to any Q ∈ Q .
Proof.
For any two value functions Q 1 and Q 2 , consider their L ∞ -distance
[Equation]
where
[Equation]
As usual, we have
[Equation]
Traditionally, it was assumed that γ ( s (cid:48) ) ≡ γ c < 1 for all states, thus the γ ( s (cid:48) ) term in (17) can be readily moved out of the sum, immediately yielding d ( B Q 1 , B Q 2 ) ≤ γ c · d ( Q 1 , Q 2 ) .
When γ ( s (cid:48) ) is not constant and is allowed to be 1 for non-terminal states, applying the operator B γ to Q 1 and Qcannot guarantee to reduce d ( Q 1 , Q 2 ) , as discussed in Proposition 10.
However, by utilizing the graph property as proved in Lemma 2.1, we can show that B γ guarantees to reduce the per-state distance d s ( Q 1 , Q 2 ) at some “support dimension” s , so that if we repeatedly apply B γ , the set of “support states” will become smaller and smaller and eventually become empty at which point the overall L ∞ -distance gets reduced (by the composite operator of repeatedly applying B γ ).
Speciﬁcally, for given Q 1 , Q 2 ∈ Q , we will identify a sequence of proper subsets of states [Equation]
[Equation]
such that
[Equation]
for all k ≥ 0 .
The construction of the subsets is by induction, and is based on the following insight:
Lemma 2.2.
Under the condition of Theorem 1 , if (19) holds for k − 1 , then ∃
[Equation]
support ( k
[Equation]
, such that ∀ i ≥ k
[Equation]
d s
[Equation]
(cid:16)
[Equation]
γ
[Equation]
d
[Equation]
[Equation]
Proof.
We ﬁrst refactor (17) a little bit, which actually holds in a per-state sense, so
[Equation]
Recursively applying (20) gives
[Equation]
where a
[Equation]
Now we prove that the inequality (21) must be strict for some support-state s ∗ ∈ d-support ( k ) .
In particular, we will prove
[Equation]
Case 1 : There is an s ∗ ∈
d-support ( k ) with (cid:80) s (cid:48)
∈ d-support ( k ) P
[Equation]
s
[Equation]
s
[Equation]
a max ( s
[Equation] [Equation]
< 1 .
In this case it’s possible to go from s ∗ to some s (cid:48) outside the subset of d-support ( k ) .
For such s (cid:48) we have γ ( s (cid:48) ) ·
d
[Equation]
γ
[Equation]
Q
[Equation]
(cid:17) <
d ( Q 1 , Q 2 ) (because (19) holds for k − 1 as assumed), thus (cid:88)
[Equation]
k )
[Equation]
≤
[Equation]
Case 2 :
For all s ∈ d-support ( Q 1 , Q 2 ) , (cid:80) s (cid:48)
∈ d-support ( Q 1 ,Q 2 ) P (cid:0) s (cid:48) | s, a max ( s ) (cid:1)
= 1 .
This is equivalent to say that there exists a policy – which would choose a max ( s ) under the corresponding s – such that it is impossible to move from any state in d-support ( k ) to a state outside d-support ( k ) under this policy.
In other words, let µ be such a policy, we have
[Equation]
which, by Lemma 2.1, entails
[Equation]
(23) (23) literally says that the set of support-states d-support ( k ) , as an absorbing subset under µ as assumed in case 2, must contain all reachable states under µ .
By the deﬁnition of ELP, these µreachable states must in turn contain at least one terminal state (otherwise we would not have ﬁnite episode under µ at all).
Let s ⊥ ∈ S µ ⊆
d-support ( k ) be such a reachable terminal state under µ .
Since s ⊥ is reachable under µ (and µ chooses a max ( s ) under each s ), there must also be an s ∗ ∈ S µ such that P ( s ⊥ | s ∗ , a max ( s ∗ )) > 0 .
Because γ ( s ⊥ ) < 1 as assumed as the general condition of
Theorem 1, we have
[Equation]
Now we have proved that ∃ s ∗ ∈ d-support ( k ) , d s ∗ (cid:16) ( B γ ) k Q 1 , ( B γ ) k Q 2 (cid:17) <
d ( Q 1 , Q 2 ) .
It is straightforward to verify that the same proof idea applies to all i > k too (in case 1, we still have γ ( s (cid:48) )
· d s (cid:48) (cid:16) ( B γ )
i − 1 Q 1 , ( B γ )
i − 1 Q 2 (cid:17) <
d ( Q 1 , Q 2 ) because (19) holds for k − 1 ; in case 2, the existence of s ⊥ is a graph property that is independent of how many times B γ is applied).
With Lemma 2.2, we can construct each subset d-support ( k ) in (18) by removing the s ∗ from d-support ( k − 1) .
It’s clear that (19) will hold for the sequence of support subsets thus constructed.
In particular, note that Lemma 2.2 holds for k = 0 without the inductive condition (that (19) holds for k − 1 ) because in this case it’s impossible to go outside d-support (0)
=
S as in
Case 1, so only Case 2 is possible (and in this case the proof does not need the inductive condition).
Now, (18) implies that d-support ( |S| ) is empty set.
Substituting this observation into (19), yields d s (cid:16) ( B γ ) |S| Q 1 , ( B γ ) |S| Q 2 (cid:17) <
d ( Q 1 , Q 2 ) for all s ∈ S , which means d (cid:16) ( B γ ) |S| Q 1 , ( B γ ) |S| Q 2 (cid:17) <
d ( Q 1 , Q 2 ) .
In other words, the existence of the d-support sequence satisfying (18) and (19) means that the composite operator of “repeatedly applying B γ for |S| times” (i.e. ( B γ ) |S| ) guarantees to reduce the L ∞ -distance of every value function pair in Q .
Moreover, note that the initial values of Q 1 and Q 2 do not determine how much percentage the distance between them will reduce – the values of Q 1 and Q 2 only affect what a max is in (22),
There are |S| · |A| · |S| possible transition probabilities in total, and |S| possible γ -values in (22), so no matter how a max (which is a policy) and arg max d change over iterations, they just select a different subset from the |S| 3 · |A| possible terms.
There are a ﬁnite number of such subsets, thus when the sum of the subset is less than 1 , there must be an absolute upper bound 1 − r min for the subset sum.
This upper bound ratio of distance reduction can be extremely close to 1 , especially after repeating the process for |S| times, but still, it is a ﬁxed number smaller than 1 , and thus the operator ( B γ ) |S| is a contraction mapping, and thus it has a unique ﬁxed point (by the Banach ﬁxed point theorem).
Our last step is to prove that the unique ﬁxed point of the composite operator ( B γ ) |S| is also the unique ﬁxed (and limiting) point of the original Bellman operator B γ .
It is easy to see that B γ cannot have two ﬁxed points, as otherwise their distance could not get reduced by repeatedly applying B γ for |S| times.
To show that B γ does have a ﬁxed point, we prove the following slightly stronger result (in the following lemma we use B to denote the generalized Bellman operator for brevity).
Lemma 2.3.
Under the condition of Theorem 1, let Q ∗ be the unique ﬁxed point of ( B ) |S| ,
[Equation]
(24) and
[Equation] [Equation]
Proof.
We ﬁrst prove that (24) holds for all Q − with Q − ≤ B Q − .
3 Because B is monotonic operator, for such Q − we have
[Equation]
where Q ∗ being the limit of the above sequence is guaranteed by the contraction property of B |S| .
Recall that Q 1 ≤ Q 2 means ∀ s, a Q 1 ( s, a ) ≤ Q 2 ( s, a ) and
that d ( Q 1 , Q 2 ) = max s,a | Q 1 ( s, a ) − Q 2 ( s, a ) | , we see that the value functions in the above (monotonic) sequence must have non-increasing distances to Q ∗ .
On the other hand, as the subsequence
Q − , ( B ) |S| Q − , ( B ) 2 ·|S| Q − , ( B ) 3 ·|S| Q − , . .
converges to Q ∗ , we know that for any (cid:15) > 0 , there exists an i ∗ such that d (cid:16) ( B ) i ∗ ·|S| Q − , Q ∗ (cid:17) < (cid:15) , and thus d (cid:16) ( B ) i Q − , Q ∗ (cid:17) < (cid:15) for all integer i >
i ∗ due to the monotonicity, which literally means that the overall sequence (26) also converges to Q ∗ .
Now because Q ∗ = lim n →∞ ( B ) n Q − , we must have B Q ∗ =
B lim n →∞ ( B ) n Q − =
lim n →∞ ( B ) n Q − =
Q ∗ , thus (25) holds, and that Q ∗ is a ﬁxed point of B .
Finally, because Q ∗ = B Q ∗ , for any Q ∈ Q (not necessarily with Q ≤ B Q this time), we have d ( B Q, Q ∗ ) =
d ( B Q, B Q ∗ ) ≥ d ( Q, Q ∗ ) , where the inequality is by (17).
Again, d ( B Q, Q ∗ ) ≥ d ( Q, Q ∗ ) means the sequence (26) has non-increasing distances to Q ∗ , this time for all Q ∈ Q .
So, by the same logic as above, Q ∗ must be the limit of the sequence Q, B Q, ( B ) 2 Q, . . .
– again, for all
Q ∈ Q this time (not necessarily with Q ≤ B Q ) .
Note that the speciﬁc form of the γ -function does not play a role in the proof of Lemma 2.3 as presented above, so (25) and (24) apply to all B γ that complies with the condition of Theorem 1.
We thus have completed the proof of statement (1) and (2) in Theorem 1.
In comparison, the classic Bellman optimality property requires γ ( s ) < 1 at every state s ∈ S , while
Theorem 1 only requires γ ( s ) < 1 at terminal states.
On the other hand, the classic result applies to all MDPs, while Theorem 1 is fundamentally based on unique structures of episodic learning process.
Importantly, the episodic discounting function that we use in ELP – i.e. (3) – does satisfy the condition in Theorem 1.
A. P ROOF OF T HEOREM 1 (3)
Finally, we prove statement (3) of Theorem 1 which asserts that Q ∗ , as the unique ﬁxed point of the
Bellman operator under episodic discounting, is indeed an optimal value function, in the sense that a Q ∗ -greedy policy is an optimal policy with respect to the undiscounted objective (1).
(Theorem 1 (3)) .
In any ﬁnite ELP, let Q ∗ be the ﬁxed point of B (i.e. the solution of (4) ), let π ∗ be a policy such that π ∗ ( a | s ) >
only if Q ∗ ( s, a ) = max ¯ a Q ∗ ( s, ¯ a ) , then J ( π ∗ ) = max π ∈ Π J
( π ) , where J is the episodic-reward objective (1) .
Proof.
Every policy π is coupled with a (unique) on-policy value function Q π which assigns values to ( s, a ) pairs according to the conditional expectations of the episode-wise cumulative reward under the policy π .
Formally,
[Equation]
Comparing (27) with the deﬁnition of the episodic-reward objective J (i.e. with (1)), and by the
ELP conditions, we see that for any policy π , its performance score equals its on-policy terminal values :
[Equation]
We will prove that for the Q ∗ -greedy policy π ∗ , we have Q π ∗ ≥ Q π for all π ∈ Π , which would entail that J ( π ∗ ) ≥ J ( π ) .
First observe that Q π ∗ =
Q ∗ , that is, the value function Q ∗ which induced the greedy policy π ∗ is also the on-policy value function of π ∗ .
Speciﬁcally, with the episodic γ -function (3), for any ( s, a ) , by the deﬁnition of π ∗ we have
[Equation]
[Equation]
Connecting (29) and (30) and generalizing to arbitrary π yields that for any policy π ∈ Π , we obtain
Q π ≤ B Q π ; speciﬁcally,
[Equation]
In other words, the set { Q : ∃ π ∈ Π , Q = Q π } is a subset of the set { Q : Q ≤ B Q } .
Now, because Q ∗ is a maximum value-function of the set { Q ≤ B Q } which contains all on-policy value functions as a subset, and because Q ∗ itself is also an on-policy value function (as Q ∗ =
Q π ∗ )
, it must follow that Q ∗ is also a maximum value-function of the subset { Q π } .
Thus we have proved that Q π ∗ =
Q ∗ ≥ Q π for all π ∈ Π , as desired.
B P ROOFS OF M INIMAX D UALITY B. P ROOF OF L EMMA 2.
(Lemma 2.1) .
In any ﬁnite ELP, for any conjugate policy π , Q ∗ ∈ arg min Q ∈Q max λ ≥ 0 L π ( Q, λ ) .
Q ∗ is an optimal solution of (7) because B is monotonic, so for Q with Q ≥ B Q we have Q ≥
B Q ≥ ( B ) 2 Q · · · ≥ Q ∗ .
The objective function of (7) is some probabilistic average over the values at terminal states (and actions), which clearly attains minimum for the per-state-action minimum value function Q ∗ .
Due to standard Lagrangian duality, a value function is an optimal solution of (7) if and only if it is a minimax solution of (8).
In particular, only value functions with Q ≥ B Q can prevent max λ L π ( Q, λ ) from being arbitrarily large, and for those Q ’s, max λ L π ( Q, λ ) =
E ζ ∼ π (cid:104) Q ( S T , A T ) (cid:105) under the complementary slackness condition, whose value attains minimum at Q ∗ as proved.
B. P ROOF OF L EMMA 2.
(Lemma 2.2) .
In any ﬁnite Episodic Learning Process ( S , A , P, R, ρ 0 ) , for any conjugate policy π , let L π be the corresponding Lagrangian, and let λ π be the particular Lagrangian multipliers with λ π ( s, a )
=
ρ π ( s ) ·
π ( a | s ) · E ζ ∼ π
[ T ] , where ρ π is the stationary distribution of π , then
[Equation]
By Proposition 8, with f ( s ) =
E A ∼ π ( s ) (cid:104)
[ s ∈ S ⊥ ] · Q ( s, A ) (cid:105) , we have
E ζ ∼ π
[Equation]
So, for any Q ∈ Q , we have
[Equation]
Note that in above for E ζ ∼ π
[ T ] · E S ∼ ρ π (cid:104) R ( S ) (cid:105) =
J ( π ) we have again used the transformation of
Proposition 8, with f ( s ) =
R ( s ) .
B. P ROOF OF T HEOREM
(ELP Minimax Theorem) .
In any ﬁnite Episodic Learning Process ( S , A , P, R, ρ 0 ) , if µ ∈ Π is an optimal policy, then its conjugate Lagrangian L µ has strong duality property, with
[Equation]
Let π ∗ be a Q ∗ -greedy policy, which is thus an optimal policy.
For any conjugate policy π , since Q ∗ is a minimax solution of L π (Lemma 2.1), we have min Q max
[Equation]
L
[Equation]
( Q, λ ) =
max
[Equation]
[-≥ 
-]0 L π ( Q ∗ , λ ) =
E π [ Q ∗ ( S T , A T )] .
By (30) in A.3 we have
E π [ Q ∗ ( S T , A T )]
=
E π [ Q π ∗ ( S T , A T )] .
By (28) in A.3 we further have E π [ Q π ∗ ( S T , A T )] =
J ( π ∗ ) .
Connecting these results together gives
[Equation]
Again for any conjugate policy π , due to Lemma 2.2, the Lagrangian function has the dual form (10) under the particular multiplier λ π , as copied below
[Equation]
The ﬁrst term above is a constant that does not change with Q , and in the second term both λ π ( s, a ) and max ¯ a Q ( s, ¯ a ) − Q ( s, a ) are nonnegative (for any Q and π ), so the sum of the two terms, i.e. L π ( Q, λ π ) , attains its minimum when Q achieves complementary slackness with λ π , in which case
[Equation]
So now, when the conjugate policy π is an optimal policy µ , as assumed in the theorem, we have
[Equation]
Due to weak minimax duality , max λ ≥ 0 min Q ∈Q L µ ( Q, λ ) ≤ min Q max λ ≥ {+
+} L µ ( Q, λ ) , which universally holds, thus the inequality above must actually be an equality, as desired.
B. P ROOF OF P ROPOSITION
(Proposition 3) .
In any ﬁnite ELP, for any value function Q ∈ Q and any policy π ∈ Π , let λ π ( s, a ) =
ρ π ( s, a ) · E ζ ∼ π
[ T ] , where ρ π ( s, a ) .
=
ρ π ( s ) ·
π ( a | s ) , then
L π ( Q, ¯ λ ) ≤ L π ( Q, λ π )
[Equation]
L π ( ¯ Q, λ π ) , ∀ ¯ Q,
[Equation]
if and only if
[Equation]
The “if” part is straightforward:
Condition (1) and (2) immediately gives L π ( Q, λ π ) =
E ζ ∼ π
[ Q ( S T , A T )]
= max ¯ λ ≥
L π ( Q, ¯ λ ) .
On the other hand, condition (3) entails that the second term in dual-form Lagrangian (10) is zero, so L π ( Q, λ π ) =
J ( π ) .
By (33) in B.3 we have min ¯ Q L π ( ¯ Q, λ π ) =
J ( π ) , thus L π ( Q, λ π ) =
min ¯ Q L π ( ¯ Q, λ π ) .
Now we prove the “only if” part, for which we resort to the general saddle-point condition:
Under given conjugate policy π , a ( Q, λ ) pair is a minimax equilibrium (or minimax saddle-point) of function L π ( Q, λ ) if and only if
(i)
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
(ii)
[Figure]
∈
[Figure] [Figure]
(iii) λ ∈ arg max ¯ λ ≥
[Figure]
¯ Q, ¯ λ )
[Figure]
By condition (ii), if ( Q, λ π ) form a minimax equilibrium, then Q must be a minimax value function.
From B.1 we know that such minimax value function must have Q ≥ B Q (condition (1)).
By condition (i) and by (32) in B.3 we know that L π ( Q, λ π ) =
min ¯ Q ∈Q max ¯ λ ≥
L π ( ¯ Q, ¯ λ ) =
J ( π ∗ ) =
E ζ ∼ π [ Q ( S T , A T )] .
In other words, (cid:80) s,a λ π ( s, a ) · (cid:0) B Q ( s, a ) − Q ( s, a ) (cid:1) , as the second term in L π ( Q, λ π ) , needs to be zero.
Because λ π ( s, a ) =
ρ π ( s, a ) · E ζ ∼ π
[ T ] where E ζ ∼ π [ T ] > 0 , and because B Q ≤ Q as proved, the only way to make the term zero is to have ρ π ( s, a ) · (cid:0) B Q ( s, a ) −
Q ( s, a ) (cid:1) = 0 for each and every ( s, a ) (condition (2)).
Moreover, since L π ( Q, λ π ) = min ¯ Q L π ( ¯ Q, λ π ) as assumed, by (33) in B.3 we have L π ( Q, λ π ) =
J ( π ) , which means (cid:80)
[Equation]
Q ( s, a ) (cid:1) , as the second term of the dual-form Lagrangian (10), must be zero.
Again, because E ζ ∼ π [ T ] > 0 and because max ¯ a Q ( s, ¯ a ) − Q ( s, a ) ≥ 0 for all ( s, a ) , the only possibility is to have ρ π ( s, a ) · (cid:0) max ¯ a Q ( s, ¯ a ) −
Q ( s, a ) (cid:1) =
for each ( s, a ) ∈ S \ S ⊥ × A (condition (3)).
C M AXIMIN SADDLE POINTS VS M INIMAX SADDLE POINTS C. P ROOF OF L EMMA 5.
(Lemma 5.1) .
In any ﬁnite ELP, Q ∗ is an optimal solution of max Q
E
[Equation]
S
[Equation]
) for any conjugate policy π ∈ Π .
Equivalently, Q ∗ ∈ arg max Q ∈Q min λ ≥
L π ( Q, λ ) .
The proof is by symmetric argument with the one in B.1: Because B is monotonic, for Q with Q ≤
B Q we have Q ≤ B Q ≤ ( B ) 2 Q · · · ≤ Q ∗ , thus Q ∗ maximizes the objective E ζ ∼ π [ Q ( S T , A T )] in a per-state-action manner.
C. P ROOF OF T HEOREM
(Theorem 5) .
In any ﬁnite ELP, for any conjugate policy π , let Q max be an maximin value function of the Lagrangian L π – i.e. let Q max be an optimal solution of (15) – then Q max is an optimal value function, in the sense that Q max -greedy policy maximizes the episodic-reward objective (1) .
As described in the proof idea, ﬁrst observe that Q max ( s, a ) ≤ Q ∗ for all ( s, a ) ∈ S
× A because
Q max , as a feasible solution of (15), has Q max ≤ B Q max ≤ BB Q max · · · ≤ Q ∗ .
Let µ be a Q max -greedy policy, so µ ( a | s ) > 0 only if Q max ( s, a ) =
max ¯ a Q max ( s, ¯ a ) .
We will prove that
[Equation]
Note that (34) would necessarily imply that
[Equation]
This is because Q max ≤ Q ∗ guarantees that for any action a sub-optimal to Q ∗ , it can only have even lower value in Q max , so Q max ( s, a ) ≤ Q ∗ ( s, a ) < max ¯ a Q ∗ ( s, ¯ a ) =
max ¯ a Q max ( s, ¯ a ) , in which Q max ( s, a ) < max ¯ a Q max ( s, ¯ a ) literally says that such an a cannot be Q max -optimal either.
(35) guarantees that µ , a Q max -greedy policy, will only choose Q ∗ -optimal actions at every nonterminal state it may encounter since time t ≥ 1 .
Such a µ is clearly an optimal policy in terms of episodic reward.
Note that µ ’s choices on terminal states does not matter here as state-transitions in terminal steps are action-agnostic, due to the ELP condition.
Now, to prove (34), we ﬁrst prove the following induction rule:
Proposition 11.
Under the context of Theorem 5, let ( s, a ) be an arbitrary state-action pair, and let
[Equation]
denote the set of all the non-terminal ( s (cid:48) , a (cid:48) )
pairs that can directly follow ( s, a ) under the Q maxgreedy policy µ , then
[Equation]
and max
[Equation]
Proof.
Because Q max ≤ Q ∗ and Q max ≤ B Q max , we have
[Equation]
So, the premise Q max ( s, a ) =
Q ∗ ( s, a ) entails that the two inequalities in above must be equality, among which the second one – i.e. (36) – can be equality only if (cid:88)
[Equation]
where s (cid:48) ∈ ( S × A ) next is a slight abuse of notation which means s (cid:48) shows up in ( S × A ) next with some coupled a (cid:48) (or equivalently, s (cid:48)
(cid:54)∈ S ⊥ and P ( s (cid:48) | s, a ) > 0 ).
Now observe that for (37) to hold, the only possibility is that
[Equation]
(38) as otherwise for those s (cid:48) on which (38) do not hold, it can only be max ¯ a ∈A Q max ( s (cid:48) , ¯ a ) < max ¯ a ∈A Q ∗ ( s (cid:48) , ¯ a ) (because Q max ≤ Q ∗ ); those s (cid:48) must all have positive weights in (37) (by deﬁnition of ( S × A ) next ), and thus will cause a real loss at the LHS of (37) (and importantly, no other state in ( S × A ) next could claim a “gain” to compensate this loss, again because Q max ≤ Q ∗ ).
Next, to prove Q max ( s (cid:48) , a (cid:48) )
=
Q ∗ ( s (cid:48) , a (cid:48) ) for all ( s (cid:48) , a (cid:48) ) ∈ ( S × A ) next (given the premise), notice that for any of such ( s (cid:48) , a (cid:48) )
we have
[Equation]
[Equation] in which max ¯ a ∈A Q max ( s (cid:48) , ¯ a )
=
Q max ( s (cid:48) , a (cid:48) ) is because a (cid:48) is by deﬁnition a Q max -greedy action under s (cid:48) , and Q max ( s (cid:48) , a (cid:48) )
≤ Q ∗ ( s (cid:48) , a (cid:48) ) is (once again) because Q max ≤ Q ∗ .
By (38) we know that the two ends of (39) actually equal to each other, so the inequalities in between must also be equality, and in particular Q max ( s (cid:48) , a (cid:48) )
=
Q ∗ ( s (cid:48) , a (cid:48) ) .
Proposition 11 as proved above enables us to prove (34) by induction (which is enough to prove the whole theorem, as argued above).
Speciﬁcally, because both Q max and Q ∗ are optimal solutions of (15), and because the objective in (15) is a distribution over only the terminal states (see B.1), it follows that Q max and Q ∗ must be equal on at least one terminal state s ⊥ .
Starting from this terminal state s ⊥ – as well as an arbitrary action a ⊥ under it – we have Q max ( s ⊥ , a ⊥ ) =
Q ∗ ( s ⊥ , a ⊥ ) , thus by the induction rule of Proposition 11 we obtain max ¯ a Q max ( s (cid:48) , ¯ a ) =
max ¯ a Q ∗ ( s (cid:48) , ¯ a ) and
Q max ( s (cid:48) , a (cid:48) )
=
Q ∗ ( s (cid:48) , a (cid:48) ) for all ( s (cid:48) , a (cid:48) ) in the ( S × A ) next set with respect to ( s, a ) =
( s ⊥ , a ⊥ ) ; the latter enables us to expand the induction proof to all non-terminal states that are reachable by µ .
[Figure] [Equation]
[Figure]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
C. A N COUNTER - EXAMPLE FOR THE Q - FORM MINIMAX VALUES IN ELP S
In this subsection we elaborate more about the counter-example as illustrated by Figure 3 in Section 5 (which is copied above).
In this ELP, S = { 0 , 1 , 2 , 3 , 4 , 5 } , A
= { 1 , 2 , 3 } .
State 4 andare terminal states, from which any action leads to state 0 .
Choosing action 1 , 2 , 3 under statedeterministically transits to state 1 , 2 , 3 , respectively.
All actions under state 1 lead to state 4 , and all actions under state 2 and 3 lead to state 5 .
The agent only receives non-zero rewards at terminal states, with R (4)
= 1 , R (5)
= 2 .
The initial state is set to state 4 (i.e. ρ 0 ( s ) >
only if s = 4 ).
The Bellman ﬁxed-point Q ∗ for this ELP is as follows:
• Q ∗ (0 , 1) = 1 , Q ∗ (0 , 2) = 2 , Q ∗ (0 , 3) =
• Q ∗ (1 , a ) = 1 , ∀ a • Q ∗ (2 , a ) =
Q ∗ (3 , a ) = 2 , ∀ a • Q ∗ (4 , a ) =
Q ∗ (5 , a ) = 2 , ∀ a
An optimal policy of this ELP should only choose action 2 or 3 , but not action 1 , under state 0 .
For minimax value functions, denoted by Q min , as they are optimal solutions of (7), we have
Q min (4 , a ) =
[Equation]
max  
[Equation]
≥
[Equation]
≥
The above minimax condition only imposes tight bounds for Q ∗ -optimal actions (e.g. action 2 and 3 under state 0 ), but leaves “ﬂexibility” for actions sub-optimal to Q ∗ (e.g. action 1 under state 0 ) as well as for all state-action pairs that follow an sub-optimal action (e.g. all actions under state 1 ).
The consequence is that even constant value function Q min ( s, a ) ≡ 2 can be a minimax value function in this example, which is clearly sub-optimal, as discussed in Section 5.
In contrast, for maximin value functions, denoted by Q max , they are optimal solutions of (15), thus
[Equation]
max
[Equation]
We see that the maximin condition manages to imposes tight bounds for at least one Q ∗ -optimal action while at the same time can enforce all Q ∗ -sub-optimal actions to be still suboptimal to Q max .
For example under state 0 , Q max (0 , 1) cannot exceed 1 , while either Q max (0 , 2) or Q max (0 , 3) needs to be tight (i.e. Q max (0 , 3)
= 2 , or Q max (0 , 2)
= 2 , or both) so as to keep the maximum of the three no less than 2 , as required.
Note that for both Q min and Q max , the Lagrangian multiplier λ that forms equilibrium/saddle points with each of them (resp.) may not encode a policy, in general.
In this example, for the constant
Q min , we have 2 = Q min (0 , 1) =
+ 1 · Q min (1 , a ) > 1 + 0 · Q min (4 , a )
= 1 , so Q min (1 , a ) >
B Q min (1 , a ) = 1 for all a , in which case its equilibrium multiplier λ has to have λ (1 , a ) = 0 for all a , due to the equilibrium condition (2) proved in Proposition 3.
Such an “all-zero” λ (on state 1 ) cannot be normalized into a policy.
Similarly, for the following maximum value function
• Q max (4 , a ) =
Q max (5 , a ) =
• Q max (0 , 1) =
Q max (1 , a ) =
• Q max (0 , 2) =
Q max (2 , a ) =
• Q max (0 , 3) =
• Q max (3 , a )
= 1 .
we have 1 = Q max (0 , 3) < 0 + 1 · Q max (3 , a ) < 2 + 0 · Q max (5 , a )
= 2 , so Q max (3 , a ) <
B Q max (3 , a ) = 2 again for all a , so the multiplier corresponding to this Q max still needs to be all zero at state 3 due to the complementary slackness condition, thus cannot be normalized at state 3 .
C. A COUNTER - EXAMPLE FOR THE V - FORM MINIMAX VALUES IN DISCOUNTED -MDP S
Moreover, the problems with minimax value functions as demonstrated above are not limited to Qfunctions or to ELPs only, but seem to be fundamental issues rooted from the minimax structure.
To see this, consider the discounted-MDP as shown in Figure 6 below.
To be strictly align with the related literature Dai et al.
; Cho and Wang (2017), the rewards are assigned to state-action pairs in this example.
In this discounted-MDP, S = { 0 , 1 , 2 , 3 } , A = { 1 , 2 } .
The initial state is set to state
(i.e. ρ 0 ( s ) = 1 if s = 0 , otherwise ρ 0 ( s ) = 0 ).
From state 0 , taking
[Figure]
action 1 , 2 will deterministically goes to state 1 , 2 , respectively, with zero reward obtained in this step.
From state 1 , any action leads to the absorbing state 3 , with reward R (1 , a ) = 1 obtained (for all a ).
From state 2 , any action leads to the same absorbing state 3 , but with reward R (2 , a ) =obtained (for all a ).
The absorbing state 3 will loop into itself forever, with zero reward obtained.
An optimal policy in this discounted-MDP should choose action 2 , not action 1 , under state 0 .
The discounting constant γ is set to 0 .
5 , so the optimal discounted-reward performance is V ∗ (0)
= 1 .
The V -form Lagrangian of the discounted-MDP above is: Dai et al.
; Cho and Wang (2017)
[Equation]
A minimax state-value function V min of the Lagrangian is an optimal solution of the Linear Programming problem (6), which has inspired some recently proposed RL algorithms (see Section 3).
In this example, the LP can be more explicitly written as
[Equation]
for which a possible optimal solution is: V min (0) = 1 , V min
[Equation]
=
[Equation]
, V min
[Equation]
= 2 , V min [Equation] = 0 , which assigns the same value to action 1 and 2 under state 0 , thus is not an optimal value function.
D L AGRANGIAN M INIMIZATION FOR M ACHINE T RANSLATION D. T HE LAMIN1 A LGORITHM
As mentioned in Section 4, the idea of LAMIN1 is to minimize the smoothed Lagrangian L βµ (with a small yet deﬁnite β ) based on an unbiased gradient estimator of (13). For convenience, we copy (13) below:
[Equation]
where π βQ ( w ) ( a | s ) . = exp (cid:0) Q ( s,a ; w ) /β (cid:1) (cid:80) b exp (cid:0) Q ( s,b ; w ) /β (cid:1) is the Boltzmann distribution with temperature β , and δ ( s, a, s (cid:48) , a (cid:48) ; w ) . = R ( s (cid:48) )
+ γ epi ( s (cid:48) )
Q ( s (cid:48) , a (cid:48) ; w ) − Q ( s, a ; w ) is the temporal-difference error.
Algorithm 1 gives the psuedo-code of such an algorithm.
Algorithm 1: The LAMIN1 algorithm.
Input: A piece of rollout data made by an optimal policy, in the form of { s t , a t , r t } 0 , 1 ,...,n , where t = T 1 , T 2 , . .
, T k are termination steps; A parameterized value function Q ( w ) with initial weight vector w 0 ; Learning rate α ; Boltzmann temperature β .
for gradient update i = 0 , 1 , 2 , . . .
{+do ∆ w ← 1 k (cid:80) t = { T 1 ...T k } ∇ w Q ( s t , a t ; w ) (cid:12)(cid:12)(cid:12) w = w i + nk · 1 n n − 1 (cid:80) t =
+}
γ epi ( s t +1 ) ∇ w (cid:16) (cid:80) a π β w ( s t +1 , a ) · Q ( s t +1 , a ; w ) (cid:17) −∇ w Q ( s t , a t ; w ) (cid:12)(cid:12)(cid:12) w = w i w
i
+1 ← w i − α · ∆ w
Output: A Q ( w ) -greedy policy.
Note that Algorithm 1 samples the stationary distribution ρ µ in (13) by averaging over the rollout data of a bunch of episodes, which is unbiased thanks to the ergodicity of episodic learning Bojun
The overall LAMIN1 algorithm is thus a standard unbiased SGD procedure, which shares the generic convergence property of all SGD procedures (i.e. convergence to local minimum of (13) is guaranteed under properly annealed learning rate; see Goodfellow et al.
As an implementation trick, the gradient computation in Algorithm 1 can utilize the following fact (for brevity and clarity, we omit the argument s t +1 in Q and π in the following):
[Equation]
and so
[Equation]
Finally, we remark that the β -smoothing trick used in LAMIN1 is more than just an approximation heuristic, but may potentially play a role in correcting (to some extent) the sub-optimality bias of minimax value functions as discuss in Section 5.
Speciﬁcally, the original Lagrangian function L µ ( Q, λ µ ) fundamentally cannot distinguish optimal minimax-values from sub-optimal minimax-values, as the function attains its global minimum in both cases. In contrast, the smoothed Lagrangian L βµ ( Q, λ µ ) tends to reach lower value at optimal minimax-values than at sub-optimal minimax-values. As an illustration, consider the special case where there is only one non-terminal state, under which the agent can only choose between two actions, 1 and 2 , and suppose action 1 is the truly optimal.
In this simple case, a value function can be represented by a tuple ( Q 1 , Q 2 ) , which indicating the value of action 1 and 2 respectively.
A sub-optimal minimax-value function may have Q 1 = Q 2 , as we showed in Section 5; in this case the smoothed Lagrangian L β =1 µ ( Q, λ µ ) would still equal J ( µ ) , which can be seen from the dual form of the smoothed Lagrangian:
[Equation]
On the other hand, an optimal minimax-value can make L β =1 µ ( Q, λ µ ) lower than J ( µ ) .
For example, when J ( µ ) = Q 1 = 0 , we have L β =1 µ ( Q, λ µ ) =
e Q 2 1+ e Q 2 · Q 2 , which attains − 0 .
28 at Q 2 = − 1 . 3 .
In other words, L β =1 µ ( Q, λ µ ) < J ( µ ) = 0 under the optimal minimax-value (0 , − 1 . 3) , which is thus distinguished from the sub-optimal minimax-value (0 , 0) in smoothed Lagrangian minimization as LAMIN1 does.
D. T HE LAMIN2 A LGORITHM
The idea of LAMIN2 is to minimize the original Lagrangian function L µ based on the “local” gradient estimator (14).
The consistency of the LAMIN2 estimator (with respect to ∇L µ ) is characterized by Proposition 4, whose proof is given below.
(Proposition 4) .
Let A be a ﬁnite action space, and let Q ( s, a ; w ) be a parameterized and differentiable value function that suggests a single best action a max ( w ∗ ) .
= arg max a ∈A Q ( s, a ; w ∗ ) when evaluating the actions under a given state s with a given parameter vector w ∗ , then
[Equation]
where π max ( w ∗ ) denote the Q ( w ∗ ) -greedy policy.
Proof.
We will prove that for any component of w ,
[Equation]
which readily gives (41).
To prove (42), notice that
[Equation]
When there are a ﬁnite number of possible actions, there must be a non-zero gap between the best action and the second best action under w ∗ .
On the other hand, since Q is differentiable, the change of action-values becomes inﬁnitely small as ∆ → 0 , thus the order between the two best actions will remain the same for small enough ∆ , that is, a max ( w ∗ +∆)
= a max ( w ∗ ) when ∆ → 0 .
Therefore,
[Equation]
Note that when Q ( w ) is a sophisticated real-valued function, such as a deep neural network with scalar output, the chance that two actions have precisely the same value under Q ( w ) should be rare.
Also, continuous action space can be densely quantized into a ﬁnite action space with arbitrarily small quantizing error, thus (41) should still approximately hold even for continuous action spaces. Algorithm 2: The LAMIN2 algorithm.
Input: A piece of rollout data made by an optimal policy, in the form of { s t , a t , r t } 0 , 1 ,...,n , where t = T 1 , T 2 , . .
, T k are termination steps; A parameterized value function Q ( w ) with initial weight vector w 0 ; Learning rate α ; Boltzmann temperature β .
for gradient update i = 0 , 1 , 2 , . . .
do [Equation] Output: A Q ( w ) -greedy policy. Algorithm 2 gives the pseudo-code of LAMIN2 in a further generalized form, where the greedy policy π w i is replaced with the Boltzmann policy π β w i .
As mentioned in Section 4, higher temperatures, such as β = 1 . 0 , can slightly improve performance in our experiment results (for 0 .
5 − 0 .
8 BLEU score).
D. A N E PISODIC L EARNING F ORMULATION OF M ACHINE T RANSLATION
Many AI tasks are sequence generation problems, where we are given a context X , and are then asked to generate a sequence Y =
( bos , y 1 . . .
y L , eos ) using tokens chosen from a given token space.
Machine Translation (MT) is an example of such tasks, where the token space is the vocabulary of a target language, and the context X is a sentence (or a sequence of sentences) in a source language.
The choice of each token y t is conditioned on X and on the partial output
Y <t .
= ( bos , y 1 . . .
y t − 1 ) .
In particular, Y < 1 = ( bos ) , and Y <L +
=
( bos , y 1 . . .
y L ) , conditioned on which the algorithm will generate the ﬁrst token y 1 and the last token y L
+1 .
= eos , respectively.
The ELP formulation exactly captures the real-world MT tasks as described above.
In the MT context, an episode is the translation of a given sentence.
The ﬁrst episode effectively starts with
S 1 =
( X (1) , bos ) where X (1) is a full source sentence.
A learning agent then chooses a token
A 1 = y (1)1 ∈ Σ target ∪ { eos } , after which the environment state transits, deterministically, to S 2 = ( X (1) , Y (1) < 2 ) =
( X (1) , bos , y (1)1 ) .
The agent keeps generating actions A t =
y (1) t under each S t =
( X (1) , Y (1) <t ) until it outputs A T − 1 = eos at some step T − 1 , leading to terminal state S T =
( X (1) , Y (1) ) =
( X (1) , bos , y (1)1 . . .
y (1) T − 2 , eos ) .
The agent will then make a normal action A T as in previous steps, which however makes no effect other than resetting the environment into S T
+1 =
( X (2) , bos ) from which the second translation episode begins.
The process goes on episode after episode, generating a (theoretically inﬁnite) sequence of translations ( X (1) , Y (1) ) , ( X (2) , Y (2) ) , . . . , which collectively serve as the training data for the agent to learn better translation policies.
An episode of length T as above results in a translation sentence Y =
( bos , y 1 . . .
y L , eos ) which contains L = T − 2 “normal” tokens from Σ target .
As a common and necessary practice, most realworld MT systems impose a maximum translation length H so that if an eos action did not show up after H steps, the environment will transit to the terminal state S H +2 =
( X, bos , y 1 . . .
y H , eos ) even if the agent continues to output normal token
A H +1 ∈ Σ target at step H + 1 .
When maximum translation length is applied, the corresponding episodic learning model of MT has bounded episode length, thus satisﬁes the ELP Condition (3) above.
Such a formulation considers the mechanism of maximum translation length as a fundamental part of learning-based MT task speciﬁcation that is essential in helping learning agents (which may not master when to output eos ) to escape from long and meaningless translation episodes which may otherwise lead to ill-conditioned training data.
To comply with ELP Condition (2), we prescribe ρ 0 to be an arbitrary distribution over the set of terminal states, i.e. over all source-target sentence pairs ( X, Y ) where Y is complete sentence ending with eos .
As with other terminal steps, no matter what S 0 and A 0 are, the next state S 1 will follow the same distribution, denoted as ρ 1 , which speciﬁes the distribution of the source sentences that the agent will receive in each episode (ELP Condition (1)).
Finally, the agent receives a scalar reward R ( X, Y ) ∈ [0 , 100] at each terminal state, based on the translation quality of Y (with respect to X ).The
reward is zero at all non-terminal states (which has only partial translations).
With this reward function, the expected episodic-reward objective J ( π ) of a translation policy π corresponds exactly to its average scores under the chosen translation metric.
The ELP formulation of MT as discussed above can be formally summarized as follows:
• S =
(Σ source ) H × { bos } × (Σ target )
H × { , eos } • A = Σ target ∪ { eos } • R ( s ) = (cid:26) metric (cid:0) X ( s ) , Y ( s ) (cid:1) s ∈ S ⊥ s (cid:54)∈ S ⊥ , where S ⊥
= { ( X, Y ) : Y ends with eos } • P ( s (cid:48) | s, a ) =
  ρ 1 ( s (cid:48) )
if s ∈ S ⊥ 1 [ s (cid:48)
=
( s, a ) ] if s (cid:54)∈ S ⊥ and | Y ( s ) | <
H
[ s (cid:48)
=
( s, eos ) ] if s (cid:54)∈ S ⊥ and | Y ( s ) | =
H • ρ 0 ( s ) >
only if s ∈ S ⊥
Note that in above both S and A are ﬁnite sets, in which case the model is a ﬁnite episodic learning process.
For real-world machine translation, we typically have |S| < 40000 2048 × 40000 2048 ×and |A| < 40000 + 1 .
The experimentation code in Supplementary Material contains a faithful implementation in Python of the formulation presented here.
D. E XPERIMENTATION D ETAILS
We tested our algorithmic idea using the WMT’14 NewsTest English → German (en2de) dataset 4 .
The data was pre-processed and post-processed using the BPE tokenizer provided by YouTokenToMe 5 , with shared vocabulary of size 37000 .
We used SacreBLEU Post (2018) as the translation metric and the standard TransformerBase neural network Vaswani et al.
(2017), which is known to achieve a BLEU score of 27 .
3 on the WMT’14 dataset under the state-of-the-art method of MLEbased learning Vaswani et al.
We trained the model on the same 4.5 millions sentence pairs in the WMT’14 data set for 100 ,gradient updates on a V100 GPU, with the same mini-batch size (and token-padding strategy) and learning rate schedule as recommended by Vaswani et al.
A dropout rate of 0 .
1 is applied.
The learned model is then used as search heuristic in the vanilla-beam-search decoding (e.g. see Algorithm 1 in Stahlberg and Byrne (2019)), with a beam size of 4 .
Empirically, we found that some more performance gain can be obtained by adding more tricks, such as modestly increasing the beam size (say, to 10 ), adding length penalty factor in the search heuristic (see Wu et al.
(2016)), and model averaging (see Vaswani et al.
, but we chose to exclude these tricks in our performance report so as to keep our algorithm simple and easy to implement.
Our experimentation code and scripts can be found in Supplementary Material for reproduction.
The following tables give the numerical values of the performance scores shown in Figure 1.
[Table]
.e.
Algorithm 1) under different temperature β
[Table]
[Table]