R ESIDUAL
L OSS
P REDICTION : R EINFORCEMENT L EARNING WITH NO I NCREMENTAL F EEDBACK
Anonymous authors
Paper under double-blind review
A BSTRACT
We consider reinforcement learning and bandit structured prediction problems with very sparse loss feedback: only at the end of an episode.
We introduce a novel algorithm, R ESIDUAL L OSS P REDICTION (R ESLOPE ), that solves such problems by automatically learning an internal representation of a denser reward function.
R ESLOPE operates as a reduction to contextual bandits, using its learned loss representation to solve the credit assignment problem, and a contextual bandit oracle to trade-off exploration and exploitation.
R ESLOPE enjoys a no-regret reductionstyle theoretical guarantee and outperforms state of the art reinforcement learning algorithms in both MDP environments and bandit structured prediction settings.
I NTRODUCTION
Current state of the art learning-based systems require enormous, costly datasets on which to train supervised models.
To progress beyond this requirement, we need learning systems that can interact with their environments, collect feedback (a loss or reward), and improve continually over time.
In most real-world settings, such feedback is sparse and delayed: most decisions made by the system will not immediately lead to feedback.
Any sort of interactive system like this will face at least two challenges: the credit assignment problem (which decision(s) did the system make that led to the good/bad feedback?); and the exploration/exploitation problem (in order to learn, the system must try new things, but these could be bad).
We consider the question of how to learn in an extremely sparse feedback setting: the environment operates episodically, and the only feedback comes at the end of the episode, with no incremental feedback to guide learning.
This setting naturally arises in many classic reinforcement learning problems ( § 4): a barista robot will only get feedback from a customer after their cappuccino is ﬁnished 1 .
It also arises in the context of bandit structured prediction (Sokolov et al., 2016;
Chang et al., 2015) ( § 2.2), where a structured prediction system must produce a single output (e.g., translation) and observes only a scalar loss that output.
We introduce a novel reinforcement learning algorithm, R ESIDUAL L OSS P REDICTION (R ESLOPE ) ( § 3), which aims to learn effective representations of the loss signal , effectively a representationlearning approach to the credit assignment problem .
Intuitively, R ESLOPE attempts to learn a decomposition of the episodic loss into a sum of per-time-step losses.
This process is akin to how a person solving a task might realize before the task is complete when and where they are likely to have made suboptimal choices.
In R ESLOPE , the per-step loss estimates are conditioned on all the information available up to the current point in time, allowing it to learn a highly non-linear representation for the episodic loss (assuming the policy class is sufﬁciently complex; in practice, we use recurrent neural network policies).
When the system receives the ﬁnal episodic loss, it uses the difference between the observed loss and the cumulative predicted loss to update its parameters.
Algorithmically, R ESLOPE operates as a reduction ( § 3.3) to contextual bandits (Langford & Zhang, 2008), allowing the bandit algorithm to handle exploration/exploitation and focusing only on the credit assignment problem.
R ESIDUAL L OSS P REDICTION is theoretically motivated by the need for variance reduction techniques when estimating counterfactual costs (Dud´ık et al., 2014) and enjoys a no-regret bound ( § 3.3) when the underlying bandit algorithm is no-regret.
Experimentally, we show the efﬁcacy of R ESLOPE on four benchmark reinforcement problems and three bandit structured prediction problems ( § 5.1), comparing to several reinforcement learning algorithms: Reinforce,
Proximal Policy Optimization and Advantage Actor-Critic.
P ROBLEM F ORMULATION AND B ACKGROUND
We focus on ﬁnite horizon, episodic Markov Decision Processes (MDPs) in this paper, which captures both traditional reinforcement learning problems ( § 4) and bandit structured prediction problems ( § 2.2).
Our solution to this problem, R ESIDUAL L OSS P REDICTION ( § 3) operates in a reduction framework.
Speciﬁcally, we assume there exists “some” machine learning problem that we know how to solve, and can treat as an oracle.
Our reduction goal is to develop a procedure that takes the reinforcement learning problem described above and map it to this oracle, so that a good solution to the oracle guarantees a good solution to our problem.
The speciﬁc oracle problem we consider is a contextual bandit learning algorithm, relevant details of which we review in § 2.1.
Formally, we consider a (possibly virtual) learning agent that interacts directly with its environment.
The interaction between the agent and the environment is governed by a restricted class of ﬁnitehorizon Markov Decision Processes (MDP), deﬁned as a tuple {S , s 0 , A , P , L , H } where: S is a large but ﬁnite state space, typically S ⊂ R d ; s 0 ∈ S is a start state; A is a ﬁnite action spaceof size K ; P = { P ( s (cid:48) | s, a ) : s, s (cid:48) ∈ S , a ∈ A } is the set of Markovian transition probabilities;
L ∈ R |S| is the state dependent loss function, deﬁned only at terminal states s ∈ S ; H is the horizon (maximum length of an episode).
The goal is to learn a policy π , which deﬁnes the behavior of the agent in the environment.
We consider policies that are potentially functions of entire trajectories 3 , and potentially produce distributions over actions: π ( s ) ∈ ∆ A , where ∆ A is the A -dimensional probability simplex.
However, to ease exposition, we will present the background in terms of policies that depend only on states; this can be accomplished by simply blowing up the state space.
Let d πh denote the distribution of states visited at time step h when starting at state s 0 and operating according to π : d πh +1 ( s (cid:48) )
=
E s h ∼ d πh ,a h ∼ π ( s h ) P ( s (cid:48) | s = s h , a = a h )
The quality of the policy π is quantiﬁed by its value function or q-value function: V π ( s ) ∈ R associates each state with the expected future loss for starting at this state and following π afterwards; Q π ( s, a ) ∈ R associates each state/action pair with the same expected future loss: V π ( s h ) =
E s H ∼ d πH | s h L ( s H ) and Q π ( s h , a h ) =
E s H ∼ d πH | s h ,a h L ( s H )
The learning goal is to estimate a policy π from a hypothesis class of policies Π with minimal expected loss: J ( π ) =
V π ( s 0 ) .
2. C ONTEXTUAL B ANDITS
The contextual bandit learning problem (Langford & Zhang, 2008) can be seen as a tractable special case of reinforcement learning in which the time horizon H = 1 .
In particular, the world operates episodically.
At each round t , the world reveals a context x t ∈ X ; the system chooses an action a t ; the world reveals a scalar loss (cid:96) t ( x t , a t ) ∈ R + , a loss only for the selected action that may depend stochastically on x t and a t .
The total loss for a system over T rounds is the sum of losses: (cid:80) Tt =1 (cid:96) t ( x t , a t ) .
The goal in policy optimization is to learn a policy π : x → A from a policy class Π that has low regret with respect to the best policy in this class.
Assuming the learning algorithm produces a sequence of policies π 1 , π 2 , . .
, π T , its regret is:
Regret (cid:0) (cid:104)
π t (cid:105)
Tt =1 (cid:1)
=
(cid:80) Tt =1 (cid:96) ( x t , π t ( x t ))
− min π ∗ ∈ Π (cid:80)
Tt =1 (cid:96) ( x t , π ∗ ( x t )) .
The particular contextual bandit algorithms we will use in this paper perform a second level of reduction: they assume access to an oracle supervised learning algorithm that can optimize a cost-sensitive loss (Appendix C), and transform the contextual bandit problem to a cost-sensitive classiﬁcation problem.
Algorithms in this family typically vary along two axes: how to explore (faced with a new x how does the algorithm choose which action to take); and how to update (Given the observed loss (cid:96) t , how does the algorithm construct a supervised training example on which to train).
More details are in Appendix A.
2. B ANDIT S TRUCTURED P REDICTION VIA L EARNING TO S EARCH
In structured prediction, we observe structured input sequences x ∈ X and the goal is to predict a set of correlated output variables y ∈ Y .
For example, in machine translation, the input x is a sentence in an input language (e.g., Tagalog) and the output y is a sentence in an output language (e.g., Chippewa).
In the fully supervised setting, we have access to samples ( x , y ) from some distribution D over input/output pairs.
Structured prediction problems typically come paired with a structured loss (cid:96) ( y ,
ˆ y ) ∈
R + that measures the ﬁdelity of a predicted output ˆ y to the “true” output y .
The goal is to learn a function f : X → Y with low expected loss under D : E ( x , y )
∼D (cid:96) ( y , f ( x )) .
Recently, it has become popular to solve structured prediction problems incrementally using some form of recurrent neural network (RNN) model.
When the output y contains multiple parts (e.g., words in a translation), the RNN can predict each word in sequence, conditioning each prediction on all previous decisions.
Although typically such models are trained to maximize cross-entropy with the gold standard output (in a fully supervised setting), there is mounting evidence that this has similar drawbacks to pre-RNN techniques, such as overﬁtting to gold standard preﬁxes (the model never learns what to do once it has made an error) and sensitivity to errors of different severity (due to error compounding).
In order to achieve this we must formally map from the structured prediction problem to the MDP setting; this mapping is natural and described in detail in Appendix B.
Our focus in this paper is on the recently proposed bandit structured prediction setting (Chang et al., 2015; Sokolov et al., 2016), at training time, we only have access to input x from the marginal distribution D X .
For example, a Chippewa speaker sees an article in Tagalog, and asks for a translation.
A system then produces a single translation ˆ y , on which a single “bandit” loss (cid:96) (ˆ y | x ) is observed.
Given only this bandit feedback, without ever seeing the “true” translation, the system must learn.
P ROPOSED A PPROACH
Our goal is to learn a good policy in a Markov Decision Process ( § 2) in which losses only arrive at the end of episodes.
Our solution, R ESIDUAL L OSS P REDICTION (R ESLOPE ), automatically deduces per-step losses based only on the episodic loss.
To gain an intuition for how this works, suppose you are at work and want to meet a colleague at a nearby coffee shop.
In hopes of ﬁnding a more efﬁcient path to the coffee shop, you take a different path than usual.
While you’re on the way, you run into a friend and talk to them for a few minutes.
You then arrive at the coffee shop and your colleague tells you that you are ten minutes late.
To estimate the value of the different path, you wonder: how much of this ten minutes is due to taking the different path vs talking to a friend.
If you can accurately estimate that you spent seven minutes talking to your friend (you lost track of time), you can conclude that the disadvantage for the different path is three minutes.
R ESLOPE addresses the problem of sparse reward signals and credit assignment by learning a decomposition of the reward signal, essentially doing automatic reward shaping (evaluated in § 5.3).
Finally, it addresses the problem of exploration vs exploitation by relying on a strong underlying contextual bandit learning algorithm with provably good exploration behavior.
3. K EY I DEA : R ESIDUAL L OSS P REDICTION
Akin to the coffee shop example, R ESLOPE learns a decomposition of the episodic loss (i.e total time spent from work to the coffee shop) into a sum of per-time-step losses (i.e. timing activities along the route).
R ESLOPE operates as a reduction from reinforcement learning with episodic loss to contextual bandits.
In this way, R ESLOPE solves the credit assignment problem by predicting residual losses, and relies on the underlying contextual bandit oracle to solve explore/exploit. R ESLOPE operates online, incrementally updating a policy π learn once per episode. In the structured contextual bandit setting, we assume access to a reference policy , π ref , that was perhaps pretrained on supervised data, and which we wish to improve; a hyperparameter β controls how much we trust π ref . As π learn improves, we replace π ref with π learn .
In the RL setting, we set β = 0 .
We initially present a simpliﬁed variant of R ESLOPE that mostly follows the learned policy (and the reference policy as appropriate), except for a single deviation per episode. This algorithm closely follows the bandit version of the Locally Optimal Learning to Search (LOLS) approach of Chang et al. (2015), with three key differences: (1) residual loss prediction; (2) alternative exploration International
Conference
For
Learning
Representations
[Figure]
strategies; (3) alternative parameter update strategies.
We assume access to a contextual bandit oracle CB that supports the following API:
CB.
ACT ( π learn , x ) , where x is the input example; this returns a tuple ( a, p ) , where a is the selected action, and p is the probability with which that action was selected.
CB.
COST ( π learn , x , a ) returns the estimated cost of taking action a in the context.
CB.
UPDATE ( π learn , x , a, p, c ) , where x is the input example, a ∈ [ K ] is the selected action, p ∈ (0 , 1] is the probability of that action, and c ∈ R is the target cost.
The requirement that the contextual bandit algorithm also predicts costs (CB. COST ) is somewhat non-standard, but is satisﬁed by many contextual bandit algorithms in practice, which often operate by regressing on costs and picking the minimal predicted cost action.
We describe the speciﬁc contextual bandit approaches we use in § 3.2.
Algorithm 1 shows how our reduction is constructed formally.
To learn a good policy, R ESLOPE reduces long horizon trajectories to single-step contextual bandit training examples.
In each episode,
R ESLOPE picks a single time step to deviate.
Prior to the deviation step, it executes π learn as a rollin policy and after the deviation step, it executes a β mixture of π learn and π ref .
At the deviation step, it calls CB.
ACT to handle the exploration and choose a (possibly) non-greedy action.
At every step, it calls CB.
COST to estimate the cost of that particular step.
Finally, it constructs a single contextual bandit training example for the deviation step, whose input was the observation at that step, whose action and probability are those that were selected by CB. ACT , and whose cost is the observed total cost minus the cost of every other action taken in this trajectory. This example is sent to CB. UPDATE .
When the contextual bandit policy is an RNN (as in our setting), this will then compute a loss which is back-propogated through the RNN.
Interaction between R ESLOPE and the environment takes place via the R ESLOPE P OLICY procedure, which is given state and returns the action a predicted by R ESLOPE .
Internally, R ESLOPE P OLICY performs a standard learning-to-search prediction step.
A mixture policy is used to generate the roll-out trajectories (Figure 5).
The mixture policy either executes π ref with probability β or π learn with probability 1 − β .
At the deviation step, the contextual bandit exploration algorithm is used to pick an action ( § 3.2).
Regardless of how the action is chosen, the action’s cost is estimated using the contextual bandit oracle.
Require:
Reference policy π ref , mixture parameter β ∈ [0 , 1] , contextual bandit oracle CB,
M AKE
1: Initialize a policy π learn{ either randomly or from a pretrained model } 2: for all episodes t = 1 . . .
T do 3: env ← M AKE E NVIRONMENT ( t ) 4:
Initialize variables: example x dev , action a dev , probability p dev 5:
Initialize cost vector ˆ c {+dev ( h ) = 
+}
for h = 1 . . .
env .H 6:
Choose deviation step h dev ← U NIFORM ( env .H ) 7:
Choose rollout policy π mix to be π ref with probability β or π learn t − 1 with probability 1 − β 8: for all time steps h = 1 . . .
env .H do 9: x ← .S
TATE F EATURES { RNN } 10: if h (cid:54) =
h { no deviation } then 11: a ← (cid:26) π t − 1 ( x ) if h h dev π mix ( x ) if h h dev 12: else if h = h dev { deviation } then 13: ( a , p dev ) ←
, x ) 14: x dev ← x 15: a ← a dev 16: end if 17: c ←
, a ) 18: env.
S a) {
} 19: end for 20: (cid:96) residual ← =
t
: end
23: Return
[Table]
3. C ONTEXTUAL B ANDIT O RACLE
The contextual bandit oracle receives examples where the cost for only one predicted action is observed, but no others.
It learns a policy for predicting actions minimizing expected loss by estimating the unobserved target costs for the unpredicted actions and exploring different actions to balance the exploitation exploration trade-off ( § 3.2).
The contextual bandit oracle then calls a cost-sensitive multi-class oracle (Appendix C) given the target costs and the selected action.
CB. UPDATE : Cost Estimation Techniques.
The update procedure for our contextual bandit oracles takes an example x , action a , action probability p and cost c as input and updates its policy.
We do this by reducing to a cost-sensitive classiﬁcation oracle (Appendix C), which expects an example x and a cost vector y ∈ R K that speciﬁes the cost for all actions (not just the selected one).
The reduction challenge is constructing this cost-sensitive classiﬁcation example given the input to CB. UPDATE .
We consider three methods: inverse propensity scoring (Horvitz & Thompson, 1952), doubly robust estimation (Dud´ık et al., 2014) and multitask regression (Langford & Agarwal, 2017).
Inverse Propensity Scoring (IPS): IPS uses the selected action probability p to correct for the shift in action proportions predicted by the policy
π learn .
IPS estimates the target cost vector y as: y ( i ) =
cp 1 [ i = a ] , where 1 is an indicator function and where a is the selected action and c is the observed cost.
While IPS yields an unbiased estimate of costs, it typically has a large variance as p → 0 .
Doubly Robust Cost Estimation (DR): The doubly robust estimator uses both the observed cost c as well as its own predicted costs ˆ y ( i ) for all actions, forming a target that combines these two sources of information.
DR estimates the target cost vector y as: y ( i )
=
ˆ y ( i ) +
[ i = a ]( c − ˆ y
( i ))
/p .
The DR estimator remains unbiased, and the estimated loss y helps decrease its variance.
Multitask Regression (MTR): The multitask regression estimator functions somewhat differently from IPS and MTR.
Instead of reducing to to cost-sensitive classiﬁcation, MTR reduces directly to importance-weighted regression.
MTR maintains K different regressors for predicting costs given input/action pairs.
Given x , a, c, p , MTR constructs a regression example, whose input is ( x , a ) , whose target output is c and whose importance weight is 1 /p .
CB. ACT : Exploration Strategies.
We experiment with three exploration strategies:
Uniform: explores randomly with probability (cid:15) and otherwise acts greedily (Sutton & Barto, 1998).
Boltzmann: varies action probabilities as a graded function of estimated value; action a is chosen with probability proportional to exp[ − c ( a ) / temp ] , where “temp” is a positive parameter called the temperature, and c ( a ) is the predicted cost of action a .
Bootstrap Exploration: In Bootstrap Exploration (Agarwal et al., 2014), a bag of multiple policies is trained simultaneously.
Each policy in the bag votes once on its predicted action, and an action can be sampled from this induced distrubtion.
To train the policies, each example gets passed to each policy Poisson ( λ = 1) -many times.
The randomness introduced by sampling from the Poisson distribution ensures diversity among the policies.
Bootstrap sampling can operate in “greedy update” and “greedy prediction” mode.
In greedy update, we always update the ﬁrst policy in the bag exactly once.
In greedy prediction, we always predict the action from the ﬁrst policy during exploitation.
3. T HEORETICAL A NALYSIS
For simplicity, we ﬁrst consider the case where we have access to a good reference policy π ref but do not have access to good Q-value estimates under π ref .
The only way one can obtain a Q-value estimate is to do a roll-out, but in a non-resettable environment, we can only do this once.
We will subsequently consider the case of suboptimal (or missing) reference policies, in which the goal of the analysis will change from competing with π ref to competing with both π ref and a local optimality guarantee.
We can show the following (proof in Appendix D).
Theorem 1.
If β = 1 , running R ESLOPE for N episodes with a contextual bandit algorithm that has regret (cid:15) CB ( N ) , the average returned policy ¯ π = E n π n has regret equal to the suboptimality of π ref , namely: Regret (¯ π ) ≤ Regret ( π ref ) + 1 N (cid:15) CB ( N ) .
In the case that π ref is not known to be optimal, or not available, we follow the LOLS analysis and obtain a regret to a convex combination of π ref and the learned policy’s one-step deviations (a form of local optimality) and can additionally show the following (proof in Appendix E):
Theorem 2.
For arbitrary β , deﬁne the combined regret of ¯ π as: Regret β (¯ π )
= β [ J (¯ π ) − J ( π ref )]
+ (1 − β ) (cid:80) h
[ J (¯ π ) − min π ∈ Π E s ∼ d h ¯ π Q ¯ π
( s, π )] .
The ﬁrst term is suboptimality to π ref ; the second term is suboptimality to the policy’s own realizable one-step deviations.
Given a contextual bandit learning algorithm, the combined regret of ¯ π satisﬁes: Regret β (¯ π ) ≤ 1 N (cid:15) CB ( N ) .
If the contextual bandit algorithm is no regret, then (cid:15) CB /N
→ 0 as N → ∞ .
3. M ULTI - DEVIATION R ESIDUAL L OSS P REDICTION
Finally, we present the multiple deviation variant of R ESLOPE .
Algorithm 2 shows how R ESLOPE operates under multiple deviations.
The difference between the single and multiple deviation mode is twofold: 1.
Instead of deviating at a single time step, multi- dev R ESLOPE performs deviations at each time step in the horizon; 2.
Instead of generating a single contextual bandit example per episode, multi- dev R ESLOPE generates H examples, where H is the length of the time horizon, effectively updating the policy H times.
These two changes means that we update the learned policy π learn multiple times per episode.
Empirically, we found this to be crucial for achieving superior performance.
Although, the generated samples for the same episode are not independent, this is made-up for by the huge increase in the number of available samples for training (i.e. T × H samples for multiple deviations versus only T samples in the single deviation mode).
The theoretical analysis that precedes still holds in this case, but only makes sense when β = 0 because there is no longer any distinction between roll-in and roll-out, and so the guarantee reduces to a local optimality guarantee.
E XPERIMENTAL S ETUP
We conduct experiments on both reinforcement learning and structured prediction tasks.
Our goal is to evaluate how quickly different learning algorithms learn from episodic loss.
We implement our
Require: Contextual bandit oracle CB
[Table] AKE E NVIRONMENT to build new enviornments [Table] models on top of the dynet neural network optimization package (Neubig et al., 2017).
The code will be made available upon publication of this work.
Reinforcement Learning Environments We perform experiments in four standard reinforcement learning environments: Blackjack (classic card game), Hex (two-player board game), Cartpole (aka “inverted pendulum”) and Gridworld.
Our implementations of these environments is described in
Appendix F and largely follows the AI Gym (Brockman et al., 2016) implementations.
We report results in term of cumulative loss , where loss is 0 − reward for consistency with the loss-based exposition above and the loss-based evaluation of bandit structured prediction ( § 2.2).
Bandit Structured Prediction Environments
We also conduct experiments on structured prediction tasks.
The evaluation framework we consider is the fully online setup described in ( § 2.2), measuring the degree to which various algorithms can effectively improve by observing only the episodic loss, and effectively balancing exploration and exploitation.
We learn from one structured example at a time and we do a single pass over the available examples.
We measure performance in terms of average cumulative loss on the online examples as well as on a held-out evaluation dataset.
The loss on the online examples measures how much the algorithm is penalized for unnecessary exploration.
We perform experiments on the three tasks described in detail in Appendix G: English
Part of Speech Tagging, English Dependency Parsing and Chinese Part of Speech Tagging.
4. C OMPARATIVE A LGORITHMS
We compare aginst three common reinforcement learning algorithms: Reinforce (Williams, 1992) with a baseline whose value is an exponentially weighted running average of rewards; Proximal Policy Optimization (PPO) (Schulman et al., 2017); and Advantage Actor-Critic (A2C) (Mnih et al., 2016).
For the structured prediction experiments, since the bandit feedback is simulated based on labeled data, we can also estimate an “upper bound” on performance by running a supervised learning algorithm that uses full information (thus forgoing issues of both exploration/exploitation and credit assignment).
We run supervised DAgger to obtain such an upper bound.
4. P OLICY A RCHITECTURE
In all cases, our policy is a recurrent neural network (Elman, 1990) that maintains a real-valued hidden state and combines: (a) its previous hidden state, (b) the features from the environment (described for each environment in the preceeding sections), and (c) an embedding of its previous action.
These form a new hidden state, from with a prediction is made.
Formally, v h is the hidden state representation at time step h , and f ( state h ) are the features coming from the environment for the state at time h , and a h is the action taken at time h .
The recursion is:
[Equation]
[Figure] [Figure] Here, A is a learned matrix, const is an initial (learned) state, emb is a (learned) action embedding function, and ReLU is a rectiﬁed linear unit applied element-wise.
Given the hidden state v h , an action must be selected.
This is done using a simple feedforward network operating on v h with either no hidden layers (in which case the output vector is o h
=
B v h ) or a single hidden layer (where o h = B 2 ReLU ( B 1 v h ) ).
In the case of R ESLOPE and DAgger, which expect cost estimates as the output of the policy, the output values o h are used as the predicted costs (and a h might be the argmin of these costs when operating greedily).
In the case of Reinforce, PPO and A2C, which expect action probabilities, these are computed as softmax ( − o h ) from which, for instance, an action a h is sampled.
Details on optimization, hyperparameters and “deep learning tricks” are reported in Appendix H.
E XPERIMENTAL R ESULTS
We study several questions empirically: 1.
How does R ESIDUAL L OSS P REDICTION compare to policy gradient methods on reinforcement learning and bandit structured prediction tasks?
What’s the effect of ablating various parts of the R ESLOPE approach, including multiple deviations?
Does R ESLOPE succeed in learning a good representation of the loss?
R EINFORCEMENT L EARNING AND B ANDIT S TRUCTURED P REDICTION R ESULTS
In our ﬁrst set of experiments, we compare R ESLOPE to the competing approaches on the four reinforcement learning tasks described above.
Figure 2 shows the results.
In Blackjack, Hex and
Grid, R ESLOPE outperforms all the competing approaches with lower loss earlier in the learning process (though for Hex and Grid they all ﬁnish at the same near-optimal policy).
For Cartpole,
R ESLOPE signiﬁcantly underperforms both Reinforce and PPO.
Furthermore, in both Blackjack and Grid, the bootstrap exploration signiﬁcantly improves upon Boltzmann exploration.
In general, both R ESLOPE and PPO perform quite well, though PPO often takes a bit longer to learn (likely because it is overly conservative).
[Figure]
In our second set of experiments, we compare the same algorithms plus the fully supervised DAgger algorithm on the three structured prediction problems; the results are in Figure 3.
Here, we can observe R ESLOPE signiﬁcantly outperforming all alternative algorithms (except, of course, DAgger) on training loss (also on heldout (development) loss; see Figure 8 in the appendix).
There is still quite a gap to fully supervised learning, but nonetheless R ESLOPE is able to reduce training error signiﬁcantly on all tasks: by over 25% on English POS, by about half on English dependency parsing, and by about 10% on Chinese POS tagging.
5. A BLATION OF R ESIDUAL L OSS P REDICTION
In our construction of R ESLOPE , there are several tunable parameters: which contextual bandit learner to use (IPS, DR, MTR), which exploration strategy (Uniform, Boltzmann, Bootstrap), and, for Bootstrap, whether to do greedy prediction and/or greedy update.
In Table 1 (in the Appendix), we show the results on all tasks for ablating these various parameters.
For the purpose of the ablation, we ﬁx the “baseline” system as: DR, Bootstrap, and with both greedy prediction and greedy updates, though this is not uniformly the optimal setting (and therefore these numbers may differ slightly from the preceding ﬁgures).
The primary take-aways from these results are: (1) MTR and DR are competitive, but IPS is much worse; (2) Bootstrap is much better than either other exploration method (especially uniform, not surprisingly); (3) Greedy prediction is a bit of a wash, with only small differences either way; (4) Greedy update is important.
In Appendix I, we consider the effect of single vs multiple deviations and observe that signiﬁcant importance of multiple deviations for all algorithms, with Reinforce and PPO behaving quite poorly with only signle deviations.
5. E VALUATING THE L EARNED L OSS R EPRESENTATION
In our ﬁnal set of experiments, we study R ESLOPE ’s performance under different—and especially non-additive—loss functions.
Our goal is to investigate R ESLOPE ’s ability to learn good representations for the episodic loss.
We consider the following different incremental loss functions for each time step: Hamming (0/1 loss at each position), Time-Sensitive (cost for an error at position h is equal to h ) and Distance-Sensitive (cost for predicting ˆ a instead of a is | ˆ a − a | ).
To combine these per-stop losses into a per-trajectory loss τ of length H , we compute the H -dimensional loss vector (cid:96) suffered by R ESLOPE along this trajectory.
To consider both additive and non-additive combinations, we consider Lp norms of this loss vector.
When the norm is L1, this is simple additive loss.
More generally, we consider (cid:96) ( τ ) =
p (cid:113)(cid:80) t = H t =
[Equation]
(cid:96) p ( t ) for any p > 0 .
We run six different experiments using different incremental and episodic loss functions.
For each incremental loss function (i.e. hamming, time sensitive, distance sensitive) we run two experiments: using the total hamming loss (additive) and an Lp norm of ﬁve (non-additive).
Results are presented in Figure 4.
We observe the following.
R ESLOPE can always learn the optimal representation for the incremental loss when the episodic loss function is additive.
This is the case for all the three incremental loss functions: hamming, time sensitive, and distance sensitive.
Learning is faster when the episodic loss function is additive.
While R ESLOPE is still able to learn a good representation even when using the L5 norm loss, this happens much later in comparison to the additive loss function (40k time steps for L5 norm vs 20k for total hamming loss).
Not surprisingly, performance degrades as the episodic loss function becomes non-additive.
This is most acute when using L-5 norm with the incremental hamming loss.
This is expected as in the distance and time sensitive loss functions,
R ESLOPE observes a smoother loss function and learns to distinguish between different time steps based on the implicit encoding of time and distance information in the observed loss.
R ESLOPE can still learn a good representation for smoother episodic loss functions.
This is shown empirically for time and distance sensitive loss functions.
R ELATED W ORK AND D ISCUSSION
R ESIDUAL L OSS P REDICTION builds most directly on the bandit learning to search frameworks
LOLS (Chang et al., 2015) and BLS (Sharaf & Daum´e, 2017).
The “bandit” version of LOLS was analyzed theoretically but not empirically in the original paper; Sharaf & Daum´e (2017) found that it failed to learn empirically.
They addressed this by requiring additional feedback from the user, which worked well empirically but did not enjoy any theoretical guarantees.
R ESLOPE achieves the best of both worlds: a strong regret guarantee, good empirical performance, and no need for additional feedback.
The key ingredient for making this work is using the residual loss structure together with strong base contextual bandit learning algorithms.
A number of recent algorithms have updated “classic” learning to search papers with deep learning underpinnings (Wiseman & Rush, 2016; Leblond et al., 2017).
These aim to incorporate sequencelevel global loss function to mitigate the mismatch between training and test time discrepancies, but only apply in the fully supervised setting.
Mixing of supervised learning and reinforcement signals has become more popular in structured prediction recently, generally to do a better job of tuning for a task-speciﬁc loss using either Reinforce (Ranzato et al., 2015) or Actor-Critic (Bahdanau et al., 2016).
The bandit variant of the structured prediction problem was studied by Sokolov et al.
(2016), who proposed a reinforce method for optimizing different structured prediction models under bandit feedback in a log-linear structured prediction model.
A standard technique for dealing with sparse and episodic reward signals is reward shaping (Ng et al., 1999): supplying additional rewards to a learning agent to guide its learning process, beyond those supplied by the underlying environment.
Typical reward shaping is hand-engineered; R ESLOPE essentially learns a good task-speciﬁc reward shaping automatically.
The most successful baseline approach we found is Proximal Policy Optimization (PPO, (Schulman et al., 2017)), a variant of Trust Region Policy Optimization (TRPO, (Schulman et al., 2015)) that is more practical.
Experimentally we have seen R ESLOPE to typically learn more quickly than PPO.
Theoretically both have useful guarantees of a rather incomparable nature.
Since R ESLOPE operates as a reduction to a contextual bandit oracle, this allows it to continually improve as better contextual bandit algorithms become available, for instance work of Syrgkanis et al.
(2016) and Agarwal et al.
Although R ESLOPE is quite effective, there are a number of shortcomings that need to be addressed in future work.
For example, the bootstrap sampling algorithm is prohibitive in terms of both memory and time efﬁciency.
One approach for tackling this would be using the amortized bootstrap approach by Nalisnick & Smyth (2017), which uses amortized inference in conjunction with implicit models to approximate the bootstrap distribution over model parameters.
A M ORE D ETAILS ON C ONTEXTUAL B ANDIT A LGORITHMS
We assume that contexts are chosen i.i.d from an unknown distribution D ( x ) , the actions are chosen from a ﬁnite action set A , and the distribution over loss D ( (cid:96) | a, x ) is ﬁxed over time, but is unknown.
In this context, the key challenge in contextual bandit learning is the exploration/exploitation problem.
Classic algorithms for the contextual bandit problem such as EXP4.P (Beygelzimer et al., 2011) can achieve a √ T regret bound; in particular:
[Equation]
) where K = | A | .
When the regret is provably sublinear in T , such algorithms are often called “no regret” because their average regret per time step goes to zero as T → ∞ .
The particular contextual bandit algorithms we will use in this paper perform a second level of reduction: they assume access to an oracle supervised learning algorithm that can optimize a costsensitive loss, and transform the contextual bandit problem to a cost-sensitive classiﬁcation problem.
Algorithms in this family typically vary along two axes:
How to explore?
I.e., faced with a new x how does the algorithm choose which action to take; 2.
How to update?
Given the observed loss (cid:96) t , how does the algorithm construct a supervised training example on which to train.
As a simple example, an algorithm might explore uniformly at random on 10% of the examples and return the best guess action on 90% of examples ( (cid:15) -greedy exploration).
A single round to such an algorithm consists of a tuple ( x , a, p ) , where p is the probability with which the algorithm took action a .
(In the current example, this would be 0 .
1 K for all actions except π ( x ) and 0 .
9 + 0 .
1 K for a = π ( x ) .)
If the update rule were “inverse propensity scaling” (IPS) (Horvitz & Thompson, 1952), the generated cost-sensitive learning example would have x as an input, and a cost vector c ∈ R K with zeros everywhere except in position a where it would take value (cid:96)p .
The justiﬁcation for this scaling is that in expectation over a ∼ p , the expected value of this cost vector is equal to the true costs for each action.
Neither of these choices is optimal (IPS has very high variance as p gets small); we discuss alternative exploration strategies and variance reduction strategies ( § 3.2).
B B ANDIT S TRUCTURED P REDICTION
Recently, it has become popular to solve structured prediction problems incrementally using some form of recurrent neural network (RNN) model.
When the output y contains multiple parts (e.g., words in a translation), the RNN can predict each word in sequence, conditioning each prediction on all previous decisions.
Although typically such models are trained to maximize cross-entropy with the gold standard output (in a fully supervised setting), there is mounting evidence that this has similar drawbacks to pre-RNN techniques, such as overﬁtting to gold standard preﬁxes (the model never learns what to do once it has made an error) and sensitivity to errors of different severity (due to error compounding).
By casting the structured prediction problem explicitly a sequential decision making problem (Daum´e & Marcu, 2005; Daum´e et al., 2009; Ross et al., 2011; Neu & Szepesv´ari, 2009) by, we can avoid these problems by applying imitation-learning style algorithms to their solution.
This “Learning to Search” framework (Figure 5) solves structured prediction problems by:
converting structured and control problems to search problems by deﬁning a search space of states S and an action set A ; 2.
deﬁning structured features over each state to capture the inter-dependency between output variables; 3.
constructing a reference policy π ref based on the supervised training data; 4.
learning a policy π learn that imitates or improves upon the reference policy.
In the bandit structured prediction setting, this maps nicely to the type of MDPs described at the beginning of this section.
The formal reduction, following (Daum´e & Marcu, 2005) is to ignore the ﬁrst action a 0 and to transition to an “initial state” s 1 by drawing an input x ∼ D X .
The search space of the structured prediction task then generates the remainder of the state/action space for this example.
The episode terminates when a state, s H that corresponds to a “ﬁnal output” is reached, at which point the structured prediction loss (cid:96) (ˆ y s H | x ) is computed on the output that corresponds to s H .
This then becomes the loss function L in the MDP.
Clearly, learning a good policy under this
MDP is equivalent to learning a structured prediction model with low expected loss.
C C OST - SENSTITIVE C LASSIFICATION
Many of the contextual bandit approaches we use in turn reduce the contextual bandit problem to a cost-sensitive classiﬁcation problem.
Cost-sensitive classiﬁcation problems are deﬁned by inputs x and cost vectors y ∈ R K , where y ( i ) is the cost of choosing class i on this example.
The goal in cost-sensitive classiﬁcation is to learn a classiﬁer f : x → [ K ]
such that E ( x , y ) ∼D (cid:2) y
( f ( x )) (cid:3) is small.
A standard strategy for solving cost-sensitive classiﬁction is via reduction to regression in a one-against-all framework (Beygelzimer et al., 2005).
Here, a regression function g ( x , i ) ∈ R is learned that predicts costs given input/class pairs.
A predicted class on an input x is chosen as argmin i g
( x , i ) .
This cost-sensitive one-against-all approach achieves low regret when the underly-
D P ROOF OF T HEOREM
In a now-classic lemma, (Kakade et al., 2003; Bagnell et al., 2004) show that the difference in total loss between two policies can be computed exactly as a sum of per-time-step advantages of one over the other:
Lemma 1 ((Bagnell et al., 2004; Kakade et al., 2003)) .
For all policies π and π (cid:48) :
[Equation]
Theorem 1.
Let π n be the n th learned policy and ¯ π be the average learned policy.
We wish to bound
J (¯ π ) − J ( π ∗ ) .
We proceed as follows, largely following the AggreVaTe analysis (Ross & Bagnell,
J ∗ ) =
J (¯ π ) − J ( π ref )+ J ( π ref ) − J ( π ∗ ) concern ourselves with bounding the ﬁrst difference
[Equation]
Fix an n , and consider the sum above for a ﬁxed deviation time step h dev :
[Equation]
where A ref ( s (cid:48) , π n ) =
Q π ref ( s (cid:48) , π n ) − Q π ref ( s (cid:48) , π ref ) is the policy (dis)advantage.
The term in the round parentheses ( . . . ) is exactly the expected value of the target of the contextual bandit cost; therefore:
[Equation]
If the CB algorithm has regret sublinear in N , the second term goes to zero as N → ∞ .
E P ROOF OF T HEOREM
Proof.
The proof follows a combination of the above analysis with the LOLS analysis.
Using the same notation as before, additionally let π out n be the mixture of π n with π ref for rollout.
First, we observe (LOLS Eq 6):
[Equation]
Then (LOLS Eq 7):
[Equation]
So far nothing has changed.
It will be convenient to deﬁne Q π n β ( s ) =
β min a Q π ref ( s, a ) + (1 − β ) min a Q π n ( s, a ) .
For each n ﬁx the deviation time step h dev n .
We plug these together ala LOLS and get:
[Equation]
[Equation]
[Figure]
[Equation]
CB
[Equation]

 

[Equation] [Equation]
[Equation]
The penultimate step follows because the inner-most expectation is exactly what the contextual bandit algorithm is estimating, and Q π n β ( s dev ) is exactly the expectation of the observed loss The ﬁnal step follows because the costs observed by the CB algorithm are exactly those in ( . . . ) , and so if the CB algorithm is no-regret, then it follows that R ESLOPE is also no-regret.
F D ETAILS ON R EINFORCEMENT L EARNING E NVIRONMENTS
Blackjack is a card game where the goal is to obtain cards that sum to as near as possible towithout going over.
Players play against a ﬁxed dealer who hits until they have at least 17 .
Face cards (Jack, Queen, King) have a point value of 10.
Aces can either count as 11 or 1, and a card is called “usable” at 11.
The reward for winning is +1 , drawing is 0 , and losing is − 1 .
The world is partially visible: the player can see one their own cards and one of the two initial dealer cards.
Hex is a classic two-player board game invented by Piet Hein and independently by John
Nash (Hayward & Van Rijswijck, 2006; Nash, 1952).
The board is an n × n rhombus of hexagonal cells.
Players alternately place a stone of their color on any empty cell.
To win, a player connects her two opposing sides with her stones.
We use n = 5 ; the world is fully visible to the agent, with each hexagon showing as unoccupied, occupied with white or occupied with black.
The reward is +1 for winning and − 1 for losing.
Cart Pole is a classic control problem variously referred to as the “cart-pole”, “inverted pendulum”, or “pole balancing” problem (Barto et al., 1983).
Is is an example of an inherently unstable dynamic system, in which the objective is to control translational forces that position a cart at the center of a ﬁnite width track while simultaneously balancing a pole hinged on the cart’s top.
In this task, a pole is attached by a joint to a cart which moves along a frictionless track (Figure 6c).
The system is controlled by applying a force of +1 or − 1 to the cart, thus, we operate in a discrete action space with only two actions.
The pendulum starts upright, and the goal is to prevent it from falling over.
The episode ends when the pole is more than 15 degrees from the vertical axis, or the cart moves more than 2 .
4 units from the center.
The state is represented by four values indicating the poles position, angle to the vertical axis, and the linear and angular velocities.
The total cumulative reward at the end of the episode is the total number of time steps the pole remained upright before the episode terminates.
Grid World consists of a simple 3 × 4 grid, with a +1 reward in the upper-right corner and −reward immediately below it; the cell at (1 , 1) is blocked (Figure 6d).
The agent starts at a random unoccupied square.
Each step costs 0 .
and the agent has a 10% chance of misstepping.
The agent only gets partial visibility of the world: it gets an indicator feature specifying which directions it can step.
The only reward observed is the complete sum of rewards over an episode.
Chinese POS
NT
NN NN
NN NN
AD 今 年 (this year) 全 球 (global) 手 机 (mobile) 市 场 (market) 规 模 (size) 将 (will) ...
English POS
NNP
NNP , CD NNS JJ , MD VB DT
NN IN DT
JJ
NN
Pierre Vinken , 61 years old , will join the board as a nonexecutive director ...
Parsing
Root
Flying planes can be dangerous
G S TRUCTURED P REDICTION D ATA S ETS
English POS Tagging we conduct POS tagging experiments over the 45 Penn Treebank (Marcus et al., 1993) tags.
We simulate a domain adaptation setting by training a reference policy on the
TweetNLP dataset (Owoputi et al., 2013) which achieves good accuracy in domain, but performs badly out of domain.
We simulate bandit episodic loss over the entire Penn Treebank Wall Street
Journal (sections 02 → 21 and 23 ), comprising 42k sentences and about one million words.
The measure of performance is the average Hamming loss.
We deﬁne the search space by sequentially selecting greedy part-of-speech tags for words in the sentence from left to right.
Chinese POS Tagging we conduct POS tagging experiments over the Chinese Penn Treebank (3.0) (Xia, 2000) tags.
We simulate a domain adaptation setting by training a reference policy on the Newswire domain from the Chinese Treebank Dataset (Xue et al., 2005) and simulate bandit episodic feedback from the spoken conversation domain.
We simulate bandit episodic loss over 40k sentences and about 300k words.
The measure of performance is the average Hamming loss.
We deﬁne the search space by sequentially selecting greedy part-of-speech tags for words in the sentence from left to right.
English Dependency Parsing For this task, we assign a grammatical head (i.e. parent) for each word in the sentence.
We train an arc-eager dependency parser (Nivre, 2003) which chooses among (at most) four
Treebank loss
In all structured prediction settings, the feature representation begins with pretrained (and nonupdated) embeddings.
For English, these are the 6gb Glove embeddings (Pennington et al., 2014); for Chinese, these are the FastText embeddings (Joulin et al., 2016).
We then run a bidirectional
LSTM (Hochreiter & Schmidhuber, 1997) over the input sentence.
The input features for labeling the n th word in POS tagging experiments are the biLSTM representations at position n .
The input features for dependency actions are a concatenation of the biLSTM features of the next word on the buffer and the two words on the top of the stack.
H O PTIMIZATION , H YPERPARAMETER S ELECTION AND “T RICKS ”
We optimize all parameters of the model using the Adam 5 optimizer (Kingma & Ba, 2014), with a tuned learning rate, a moving average rate for the mean of β 1 = 0 .
9 and for the variance of β 2 = 0 .
999 ; epsilon (for numerical stability) is ﬁxed at 1 e − 8 (these are the Dynet defaults).
The learning rate is tuned in the range { 0 .
For the structured prediction experiments, the following input features hyperparameters are tuned:
• Word embedding dimension ∈ { 50 , 100 , 200 , 300 } (for the Chinese embeddings, which come only in 300 dimensional versions, we took the top singular vectors to reduce the dimensionality).
• BiLSTM dimension ∈ { 50 , 150 , 300 } • Number of BiLSTM layers ∈ { 1 , 2 } •
Pretraining: DAgger or AggreVaTe initialization with probability of rolling in with the reference policy ∈ { 0 . 0 , 0 .
999 N , 0 .
99999 N , 1 . 0 } , where N is the number of examples • Policy RNN dimension ∈ { 50 , 150 , 300 } • Number of policy layers ∈ { 1 , 2 }
For each task, the network architecture that was optimal for supervised pretraining was ﬁxed and used for all bandit learning experiments 6 .
For the reinforcement learning experiments, we tuned:
• Policy RNN dimension ∈ { 20 , 50 , 100 } • Number of policy layers ∈ { 1 , 2 }
Some parameters we do not tune: the nonlinearities used, the size of the action embeddings (we use 10 in all cases), the input RNN form for the text experiments (we always use LSTM instead of RNN or GRU based on preliminary experiments).
We do not regularize our models (weight shrinkage only reduced performance in initial experiments) nor do we use dropout.
Pretraining of the structured prediction models ran for 20 passes over the data with early stopping based on held-out loss.
The state of the optimizer was reset once bandit learning began.
The variance across difference conﬁgurations was relatively small across RL tasks, so we chose a two layer policy with 20 dimensional vectors for all RL tasks.
Each algorithm also has a set of hyperparameters; we tune them as below:
• Reinforce: with baseline or without baseline • A2C: a multiplier on the relative importance of actor loss and critic loss ∈ { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 , 2 . 0 , 5 . 0 , 10 . 0 } • PPO: with baseline or without baseline; and epsilon parameter ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 4 , 0 . 8 } • R ESLOPE : update strategy (IPS, DR, MTR) and exploration strategy (uniform, Boltzmann or Bootstrap)
In each reinforcement/bandit experiment, we optimistically pick algorithm hyperparameters and learning rate based on ﬁnal evaluation criteria, noting that this likely provides unrealistically optimistic performance for all algorithms.
We perform 100 replicates of every experiment in the RL setting and 20 replicates in the structured prediction setting.
We additionally ablate various aspects of R ESLOPE in § 5.2.
We employ only two “tricks,” both of which are defaults in dynet: gradient clipping (using the default dynet settings) and smart parameter initialization (dynet uses Glorot initialization (Glorot &
Bengio, 2010)).
I E FFECT OF S INGLE VS M ULTIPLE D EVIATIONS
Next, we consider the single-deviation version of R ESLOPE (1) versus the multiple-deviation version (2).
To enable comparison with alternative algorithms, we also experiment with variants of
Reinforce, PPO and DAgger that are only allowed single deviations as well (also chosen uniformly at random).
The results are shown in Figure 9.
Not surprisingly, all algorithms suffer when only allowed single deviations.
PPO makes things worse over time (likely because its updates are very conservative, such that even in the original PPO paper the authors advocate multiple runs over the same data), as does Reinforce.
DAgger still learns, though more slowly, when only allowed a single
[Figure]
Reinforcement Learning
[Figure]
Cartpole
Grid
Hex
Zh-POS
En-Dep
En-POS
[Table]
loss
[Table]
−
[Table] [Table]
− 0 .1 .6 .7 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
− 1 .− 0 .−
[Table]
deviation.
R ESLOPE TODO write this when we have results.
Overall, this suggests that even though the samples generated with multiple deviations by R ESLOPE are no longer independent, the gain in number of samples more than makes up for this.
J S YNTHETIC DATA FOR E VALUATING THE L EARNED L OSS R EPRESENTATION
Experiments were conducted on a synthetic sequence labeling dataset.
Input sequences are random integers (between one and ten) of length 6 .
The ground truth label for the h th word is the corresponding input mod 4.
We generate 16k training sequences for this experiment.
We run R ESLOPE
[Figure]
[Figure]
[Figure]
with bootstrap sampling in multiple deviation mode.
We use the MTR cost estimator, and optimize the policies using ADAM with a learning rate of 0 . 01 .