Teacher Forcing Recovers Reward Functions for Text Generation
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Through the lens of inverse reinforcement learning, we propose a method to derivethe reward function from models trained with the teacher-forcing objective.
Themethod gives us a step-wise reward function for sequence generation and enablesreinforcement learning for text generation.
Compared to previous approaches, ourmethod is task-agnostic and does not require any human heuristics.
In addition,we develop a stable training strategy based on the policy gradient method, whichleads to further improvement on non-parallel datasets when combined with ourproposed reward.
The empirical results show that our method outperforms severalprevious works, which conﬁrms the effectiveness of our reward function.
The codeis publicly available 1 .
Introduction
The common training method for text generation tasks is teacher-forcing training on paralleldatasets [7].
Although this practice has been widely applied [7, 13, 56], there are mainly twodrawbacks.
1) Teacher-forcing training is data-hungry because parallel datasets are usually expensiveto obtain.
Larger models [3, 42] further intensify this problem.
Meanwhile, there are numerousunlabeled, non-parallel datasets available.
This poses an urge to efﬁciently exploit non-parallel data.2) Teacher forcing introduces a discrepancy between training and inference because the model learnsto predict the next word based on the partial ground-truth reference during training, whereas ininference the model predicts the next word based on its self-generated previous words.
This undesired discrepancy is known as exposure bias [45].
While many discussions on exposure bias takingplace [20, 64], abundant evidence [6, 28, 57] shows that the exposure bias cannot be overlooked.
To address the ﬁrst problem, a straightforward method is to generate pseudo-parallel sentencesfor data augmentation, such as self-training [2], sequence-level knowledge distillation [24], andback-translation [49].
However, the exposure bias remains in such cases.
To address the second problem, the model must be trained on the self-generated sentences, oftenbased on reinforcement learning (RL).
However, text generation does not have a naturally deﬁnedreward function for RL.
Researchers have proposed various heuristic scores as the reward, suchas BLEU [38] for translation and ROUGE [31] for summarization.
These reward functions aretask-speciﬁc, restricting the application of RL in different tasks.
Further, these rewards requireparallel data, failing to address the ﬁrst problem above; they are typically sparse (only non-zero at theend of a sentence), making RL training difﬁcult.
The goal of this paper is to address these two problems within one framework with a learned densereward function.
Our approach is a two-step strategy: we ﬁrst train a sequence-to-sequence (seq2seq) model on the parallel dataset and induce a reward function from the model.
Then, we apply RL onnon-parallel data based on our induced reward function.
Our method is task-agnostic and does not require handcrafted engineering or heuristics.
Further,our reward function assigns dense (step-wise) values, which makes RL training much easier thansparse rewards.
Additionally, the reward function derived from the seq2seq model does not directlyparticipate in the generation, which allows the model to explore based on its own prediction and thusalleviates the exposure bias.
We conduct experiments on dialogue and paraphrase generation tasks to show the generality of thismethod.
The empirical results suggest that our method leads to better performance than severalbaselines, including self-training and task-speciﬁc heuristic reward learning.
This conﬁrms theeffectiveness of our framework.
Approach
Our approach resorts to reinforcement learning to perform exploration on non-parallel data, whosefoundation is the Markov decision process (MDP).
In this section, we ﬁrst introduce the MDPformulation for text generation.
Then we describe our approach that induces the reward functionfrom a seq2seq model trained by teacher forcing.
Finally, we describe the policy gradient method for
RL training with our induced reward function.
2.1 Reinforcement Learning Formulation of Text Generation
Text Generation as Markov Decision Process (MDP).
We formulate the text generation processas an (undiscounted) MDP, which can be represented as a tuple ( S , A , T, r ) .
At every step, a decisiona 2 A is made based on its state s 2 S .
The transition dynamics T ( s 0 | s, a ) is the probability of the 54 next state being s 0 , given the current state s and the action a .
A function r : S ⇥ A !
R deﬁnes thereward based on a state and an action.
Typically, the decision making is assisted by a policy ⇡ , which is trained to maximize the expectedtotal reward, given by an action-value function
[Equation]
where H is the number of steps.
Theoretical results show that the optimal policy ⇡ ⇤ satisﬁes
the
Bellman optimality equation:
[Equation]
For text generation, the MDP state can be deﬁned as the partial generated sequence y <t :=( y 1 , . . .
, y t  1 ) , and the action as the next token y t in the vocabulary V .
The transition dynamic
T ( ·| s, a ) here is deterministic, since every state-action pair ( y <t , y t ) leads to a unique state y <
t
+1for the next step.
In previous RL-based text generation, there lacks a naturally deﬁned reward function r ( s, a ) .
Whileresearchers have applied various heuristics as the reward [1, 50], they suffer from a number ofshortcomings (e.g., sparsity and task-speciﬁcity) as mentioned in Section 1.
To address theseproblems, we propose to induce a reward function in a principled approach by inverse reinforcementlearning for a text generation task at hand.
Maximum Entropy Inverse Reinforcement Learning (IRL).
The goal of IRL is to learn areward function r ( s, a ) .
Especially, we wish the resulting action-value function q computed by
Eqn.
(1) satisﬁes q ( s, a )  q ( s, a 0 ) whenever a 0 6 = a for any sample ( s, a ) in a training set D :={ ( s ( i ) , a ( i ) ) } i .
In other words, the decisions in D are made greedily by argmax a q ( s, a ) given anystate s .
Unfortunately, Ng and Russell show that this is an ill-posed problem, since the desirablereward function r is not unique.
Therefore, we follow the assumption proposed previously [4, 44, 67]to resolve the ambiguities:
Assumption 1.
Given an action-value function q , the policy ⇡ takes the form that ⇡ q ( a | s ) :
=exp( q ( s, a )) /
P a 0 exp( q ( s, a 0 )) .
In traditional IRL [4, 44, 67], the reward learning is difﬁcult and this assumption does not directlyyield a reward function due to the stochastic state transition T ( s
| s, a ) .
However, our insight is thatthe transition is deterministic for text generation tasks, and thus we may utilize Assumption 1 toinduce the action-value function q , and then a reward function r , from some learned policy ⇡ .
2.2 Teacher Forcing Recovers IRL
One of our main contributions is that we show the seemingly complicated reward learning in
Section 2.1 can be recovered by the teacher-forcing training, the de facto common practice ofsupervised text generation.
Our discovery leads to a convenient approach that derives a step-wisereward function simply from general seq2seq models, without the need for task-speciﬁc heuristicsfor deﬁning the reward.
This makes RL more general for text generation, and our step-wise rewardlargely simpliﬁes RL training.
Maximum Likelihood Estimation (MLE) for IRL.
Following Assumption 1, we let the policy⇡ q !
( ·| s ) / exp( q !
( s, · )) , where q ! is a parameterized action-value function.
Under such policy, theprobability of each trajectory ⌧ :=
(( s 1 , a 1 ) , . .
, ( s H , a H )) in the dataset is given by the trajectorydistribution P ⇡ .
The likelihood of the dataset is then:
[Equation]
Teacher-Forcing Training.
As for text generation, the standard teacher-forcing seq2seq training isto minimize the loss:
[Equation]
) where the predicted probability of the next token being v is p !
( v | y <t ) =
exp( f !
( y <t ,v ))
P v 02 V exp( f !
( y <t ,v 0 )) for 96 the logit function f ! with parameters ! .
In seq2seq training, an additional input x may be added tothe conditional probabilities but is omitted here for simplicity.
The below theorem shows their equivalence up to an additional constant.
Theorem 1.
Suppose the policy q in Eqn.
(3) and the seq2seq model f in Eqn.
(4) have the sameparametrization !
, we have
[Equation]
Proof.
For the MLE of IRL under Assumption 1, the Ionescu–Tulcea theorem [23] asserts that thereexists a unique trajectory distribution P ⇡ µ satisfying
[Equation]
for any t  1 , given the initial state distribution µ , transition probability T , and policy ⇡ .
The likelihood can thus be factorized by the multiplication of µ , T , and ⇡ :
[Equation]
As mentioned, text generation has a deterministic transition, i.e., T ( s 0 | s, a )
= 1 for the next state s
= s
+ [ a ] .
Taking the µ terms out, we have
[Equation]
where the second term is a constant in terms of ! .
In Section 2.1, text generation is modeled as an
MDP with s t = y <t and a t = y t .
Therefore, the ﬁrst term of Eqn (6) is the same as Eqn (4) under theparametrization ⇡ q !
=
p ! , concluding the equivalence between MLE for IRL and the teacher-forcingtraining of a seq2seq model.
Inducing the Reward Function.
Our theorem shows that seq2seq training with teacher forcingactually learns an IRL model.
Thus, we may derive the reward function as
[Equation]
where the ﬁrst equality is due to the Bellman optimality condition (2); the second equality is due to theparametrization of q !
=
f !
and the deterministic transition T ( s 0 | s, a ) =
for s 0 as the concatenationof the preﬁx s and token a .
2.3 Periodically Synchronized Behavior Policy in Policy Gradient
In text generation, a neural network can be viewed as a policy ⇡ that predicts the word distributiongiven the state of a decoding step.
The reward induced from Section 2.2 can be used to improvethe policy through RL training.
In addition, we propose a variant of the off-policy policy gradientmethod [9] by using the periodically synchronized behavior policy.
Our RL training algorithm ﬁrst adopts the off-policy variant of the REINFORCE [60], a policygradient method, as the backbone of our algorithm.
Speciﬁcally, we ﬁrst parameterize the modelpolicy ⇡ ' with ' .
Given a behavior policy ⇡ b , the gradient of the expected reward with regard to theparameter ' can be obtained by the off-policy policy gradient theorem [9]
[Equation]
where ⇢ t :=
⇡ ' ( a t | s t ) / ⇡ b ( a t | s t ) is the importance factor, ˆ q r ( s t , a t ) :
=
P | ⌧ | t =1 r ( s t , a t ) is theestimated value function by accumulating the rewards.
The off-policy REINFORCE [9] is more exploratory compared to the on-policy ( ⇡ b = ⇡ ' ) onebecause the samples from the behavior policy ⇡ b are able to cover all possible trajectories, whereasthe model policy ⇡ ' may only generate ﬁxed trajectories when it becomes more optimal.
However, the ﬁxed behavior policy ⇡ b has the exploitation problem and does not capture the currentstate of the model policy ⇡ ' .
This leads to less informative training because trajectories sampledfrom the behavior policy are unlikely to occur under the current model policy ⇡ ' .
To balance theexploration and exploitation, we aim to make the behavior policy close to the model policy but stayexploratory for a long time.
We propose a periodical updating schedule of the behavior policy by keeping the behavior policy at aproper distance from the current model policy.
Particularly, the behavior policy keeps unchanged fora certain period until the model policy is sufﬁciently updated.
In other words, the behavior policysynchronizes the weights with the model policy after k steps of gradient updates for some positiveinteger k .
Overall, our ﬁnal algorithm is described in Algorithm 1.
In practice, all loops in the algorithm areable to execute in parallel to speed up the training.
Further, we distribute the behavior policy andthe model policy to different processes and communicate the weights through the shared memorymechanism to reduce the waiting time of the sampling.
Details are presented in Appendix B.
Experiments
We conduct experiments on two different tasks, namely, dialogue generation and paraphrase generation.
In this section, we show the details of our experiments.
beginfor i do if i then
policy update
Sample a source sentence
2 D Construct the initial state
Sample a trajectory t do
| then r else
. Accumulating rewards
[Equation] [Equation]
. Importance weight
[Equation] [Equation]
. Gradient ascent
return ⇡ '
3.1 Settings
Our approach involves training a reward function and a policy network by teacher forcing, whichrequires (at least a small amount) of parallel data.
In this paper, we mainly focus on semi-supervisedlearning, where we assume a small parallel corpus and a large non-parallel corpus are available.
Nevertheless, our approach may be extended to other scenarios, such as generative adversarialnetworks (GANs) [18, 65], which may be addressed in the future work.
For each task, we ﬁrst train a model on the parallel data by Eqn.
We initialize the parameters withthe pre-trained T5-Base [43] model, and ﬁne-tune all parameters by teacher forcing in a supervisedmanner.
Then, we apply our proposed approach and further train the model by reinforcement learningon the non-parallel data.
We compare our approach with the following competing methods.
Self-Training.
We use the supervised model to generate the corresponding pseudo-target sentencesgiven the non-parallel datasets.
This generated pseudo-parallel dataset is then applied for training onthe same model.
This is commonly used as a semi-supervised approach for text generation [22, 66].
R-Regression.
We replicate the reward regression (R-Regression) method [63] by following thesettings of [63] and train an additional token-level reward function through regression with BLEUscores.
Similarly, we then apply the same REINFORCE [60] method to further train the supervisedmodel.
Compared to our method, the reward function of R-Regression directly predicts the metricof interest, whereas our method does not have the access to any deﬁned metrics.
Note that the onlydifference between this method and ours is the reward function.
The details of our experiments are included in Appendix A.
3.2 Datasets and Metrics
Dialogue Generation.
For the dialogue generation task, we use DailyDialogue [30] as the paralleldataset and collect only source sentences in OpenSubtitle [55] as the non-parallel dataset.
The
DailyDialogue dataset is constructed from English dialogues crawled from the Internet,
whereas the
OpenSubtitle dataset is constructed from movie subtitles based on IMDB identiﬁers.
For each piece
[Table] [Table]
of the dialogue in both datasets, there are only two speakers and they speak in an in-turn manner.
Foreach turn, we view it as a context-response paradigm and our model learns to generate the responsegiven the context.
We only consider the single-turn dialogue in this experiment.
However, there are duplicated sentences across training and test sets in both two datasets whichwould result in an unfair comparison [59].
To address this problem, we turn to using the cleanedversion [59].
In this version, DailyDialogue contains 60K/6.5K/7K samples for training/validation/testand OpenSubtitle contains 1M/12K/12K samples.
Additionally, for the cleaned OpenSubtitle datasetwe only use the source sentences of the training set to serve as a non-parallel dataset.
Note thatparallel sentences in the OpenSubtitle dataset are noisy and often result in lower performance [59].
We use the BLEU score [38] as the major evaluation metric, which is widely used in the dialoguegeneration tasks [59].
BLEU- n evaluates the geometric average of i -gram precision scores where iis from 1 up to n .
Following the previous work [59], we lowercase all sentences and tokenize themwith the NLTK [33] library.
Paraphrase Generation.
For the paraphrase generation task, the dataset is constructed from the
Quora Question Pair classiﬁcation dataset [41] following the common practice [10, 32, 34].
Thisclassiﬁcation dataset contains question pairs labeled as either paraphrase or non-paraphrase.
Wecollect all paraphrase pairs as the parallel dataset and non-paraphrase sentences as the non-paralleldataset.
Following the previous work [10, 32, 34], we split the parallel dataset into 124k/4k/20k astraining/validation/test; the resulting non-parallel dataset contains 510k sentences.
We use the iBLEU [53] score as the main evaluation metrics.
The calculation of iBLEU involves apenalty of the Self-BLEU (the BLEU score between the generation and the source corpus) to avoiddirectly copying the source sentences.
Speciﬁcally, iBLEU
3.3 Main Results
Our main results are shown in Table 1.
Results of Dialogue Generation.
For the dialogue generation task shown in Table 1a.
The selftraining method only provides a very slight improvement in BLEU2 and BLEU4 and stays roughlyclose to the supervised model.
The reward regression (R-Regression) model shows a moderate improvement over the supervised model.
However, it is still obviously lower than the performance ofour model.
Speciﬁcally, our method signiﬁcantly improves the supervised model by +2.06 (+23.0%)in BLEU2 and +0.61 (+16.5%) in BLEU4.
It also outperforms the R-Regression by 0.68 (+6.6%) in
BLEU2 and 0.12 (+2.9%) in BLEU4.
Note that throughout the process, our model does not havethe access to the evaluation metric (i.e., BLEU score) of its generation, whereas the R-Regressiondirectly optimizes towards higher BLEU score predictions.
Even with this obstacle, our proposedreward achieves higher ﬁnal performance, suggesting the generality of our method.
We notice that RL training with either reward functions shows superior performance compared to self-training, which indicates theimportance of explorations on the dialogue task.
This is within ourexpectation because tasks like dialogue usually require the modelto capture the multi-modality of responses.
RL training naturallypossesses the advantage of exploring different possible hypothesesof the response given the context and thus generally tend to have
For comparison with RL training, we also show the fully-supervisedresult by using the ground truth references of OpenSubtitle startedwith the same model.
Interestingly, due to the low quality of the
OpenSubtitle dataset, the resulting model performs even worse compared to the one trained on the parallel DailyDialogue only.
Thisemphasizes the need for semi-supervised approaches when the quality of the parallel dataset is not desirable.
Results of Paraphrase Generation.
For the paraphrase generation task in Table 1b, our approach consistently achieves the bestperformance in iBLEU among all baselines.
We report Self-BLEU(SBLEU)
scores to directly display the similarity between the modelgeneration and the source sentence.
As shown, several previousmethods [10, 24, 40, 63] cannot maintain a good balance between
BLEU and SBLEU, some even achieve SBLEU scores over 60, indicating a severe problem of the overlapping between the sourcesentences and model generation.
On the contrary, our method showsa total advantage by having the highest BLEU4 and a lower SBLEU4 at the same time, resulting in amuch more desired iBLEU score.
This further implies that our reward function implicitly controls thesimilarity between the source sentence and generation without any human heuristics injected, whichagain conﬁrms the generality of our method.
3.4 Analysis
Step-Wise Reward.
In Figure 1, we show that our derived reward function solves the sparsityproblem as the rewards are mostly distributed around 0.5.
However, The ground-truth BLEU scoreslargely concentrate on 0, providing little information for the training.
For the regression with theground-truth BLEU scores (R-regression), although the sparsity problem is mitigated, R-Regression
[Table]
-Regression [63] 30.
Ours 31. still assigns near 0 rewards extensively.
This results in a more sparse distribution with a higher peakcompared to our method.
The density of our reward is beneﬁcial for training.
In Table 2, we show the results when we reducethe step-wise (dense) reward to the sentence-level (sparse) one by only using the summation ofrewards at the end of the sentence.
As shown, when the reward is sparse, the ﬁnal performance woulduniformly drop to a lower level.
This veriﬁes that the step-wise reward is essential to RL training intext generation.
The Effect of the Synchronizing Period.
The synchronizingperiod k we introduced in Section 2.3 plays an important role inthe RL training.
In Figure 2 we show that if k = 1 (on-policy),then the training will be unstable and drastically drop into adegeneration model, where the generation will be uninformative.
We provide examples of the generation in Appendix C.
The training is more stable when k = 1000 .
It quickly goes upto a more optimal point but slowly decreases with more updatesteps.
This is due to the lack of exploration as we mentionedin Section 2.3.
When k is inﬁnitely large, the behavior policyis then ﬁxed as in the conventional off-policy case throughoutthe training.
In this case, the training will be inefﬁcient andstop improving after a certain step within each period.
As aresult, we choose k = 5000 in all experiments to balance theexploitation and exploration.
Data Efﬁciency.
Our method also demonstrates the abilityto efﬁciently utilize the data.
Speciﬁcally, we sample differentsubsets of various sizes from the non-parallel OpenSubtitletraining set and compare the performance of our method andself-training.
This is shown in Figure 3.
The trend showsthat our method achieves a higher performance even whenthere is only 0.1% of the training set available.
Additionally,the performance of our method increases to an even higherlevel when more data is available, whereas self-training can noteffectively utilize the data and grows slowly.
Related Work
Semi-Supervised Learning for Text Generation.
In text generation, the most popular ways toutilize both the parallel and non-parallel data are by self-training [22, 24, 66] or back-translation [49].
Speciﬁcally, they are ﬁrst trained on the parallel sentences and then generate the pseudo-parallelpairs for the non-parallel sentences.
These approaches are widely applied to different paradigms.
Forexample, Gu et al.
propose to train the non-autoregressive models [11, 15, 21] on the pseudo-parallelpairs.
We mainly compare with self-training as we do not train the reversed model.
In this work,we show that such a practice (self-training) cannot efﬁciently exploit the data because of the lack ofexploration.
Additionally, since they are trained on the given sentence, the exposure bias remains.
Text Generation beyond Teacher Forcing.
Many researchers try to alleviate the exposure bias bytraining the model on self-generated sentences.
These methods usually involve generative adversarialnetworks (GAN) [14] training or RL.
For example, Yu et al.
and Guo et al.
propose to use GAN-styletraining to generate text similar to the training set.
Shi et al.
further formulate the adversarial trainingusing the IRL interpretation.
The GAN-style methods are different from ours in mainly two ways.
First, GAN-style training requires the parallel corpora and thus can not be applied in the case of semisupervised learning to utilize non-parallel datasets.
Second, GAN-style training involves optimizinga counterpart component during the training, making it unstable (e.g. mode collapse [14]).
In anotherline of work, Bahdanau et al.
and
Shen et al.
heuristically deﬁne BLEU as the reward and conduct RLtraining with parallel datasets.
These methods are task-speciﬁc and suffer from the sparse reward aswe mentioned in Section 3.4.
More importantly, these approaches require the ground-truth referencein parallel data to calculate the reward.
measured
Dialogue set.
In an effort to remove the reliance on the ground-truth references for calculating the reward, the most straightforward wayis learning to predict the reward function by regression on theparallel dataset [63].
As we discussed in Section 1, this methodis still task-speciﬁc because it requires human heuristics of thetask to deﬁne the proper reward function.
Additionally, thereward regression also suffers from the sparsity of the metricin Section 3.4 and hence reduces the quality of the generated
Searching is also a popular way to replace teacher forcing.
The
Learning to Search (L2S) framework [5, 8] enables the modelto search for a better score during learning and is widely appliedin text generation.
For example, Wiseman and Rush proposeto optimize the beam-search results through training.
Li et al.develop an unsupervised learning approach for text generation based on local search.
In addition to the L2S framework,
Leblond et al.
leverage Monte Carlo tree search [26, 52] tounify the training and inference in text generation.
These methods are different from ours since they need either heuristicallydeﬁned score functions or parallel data, limiting their methodsto certain tasks or to the supervised paradigm.
However, giventhe success of these methods, we consider the search-basedapproach an interesting future extension of our work.
Learning from Demonstration.
Our work is also related to learning from demonstration in general.
Typically, these methods aim to return a good policy given the demonstration containing state-actionpairs.
The easiest approach is behavior cloning [39] which greedily imitates the demonstration.
Similar to the exposure bias, behavior cloning also faces the compounding errors problem [47],which gives rise to many solutions [47, 48].
Du and Ji empirically show that these methods arealso helpful in text generation.
Recently, Pang and He frame text generation tasks as an ofﬂinereinforcement learning problem given the demonstrations from the experts.
Compared to our method,these approaches rely on parallel sentence pairs and cannot effectively make use of non-paralleldatasets.
Conclusion
Summary.
We show that models trained with the teacher-forcing objective can be viewed as rewardfunctions.
The derivation of the reward function does not rely on human heuristics for certain tasks.
Additionally, the reward function assigns step-wise scores and makes the RL training easier.
Ourconvenient approach leads to a training algorithm in a semi-supervised manner.
The empirical resultsshow that the performance of our approach is strong compared to several other methods.
Speciﬁcally,it outperforms self-training and reward regression as two popular semi-supervised baselines.
Wefurther analyze our reward function and show the beneﬁts of our approach.
Limitations and future works.
In Figure 3, we show that our method greatly boosts the dataefﬁciency when the non-parallel data is insufﬁcient.
However, we notice that if the dataset growslarger, although we still maintain the edge compared to other methods, the help of the explorationdoes not effectively scale with the dataset.
We leave the solution to this as a direction for future work.
We also notice that Assumption 1 has a deep connection with entropy regularized RL [17, 19, 46].
Ourapproach can be easily extended to the such cases in the future.
Another interesting direction would beusing the reward as an interface between humans and the model to control the generation.
Speciﬁcally,the current seq2seq models treat data as the ground truth.
But the data may be contaminated withundesired or harmful information.
We hope that our approach provides a way for human to applyadditional rules to the reward function to avoid the model generating harmful information.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] See Section 2-4.
b)
Did
you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any
potential negative societal
impacts of your work?
[Yes] See
Section 5.
(d) Have you read the ethics review guidelines and
ensured
that your paper conforms to
them?
[Yes]
If you are including
theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section2.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 2.2.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix.(c)
Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 3.
(b) Did you mention the license of the assets?
[Yes] See Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We make our code public.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Appendix.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No] It is discussed in the papers of the datasets.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Additional Appendix: Theoretical Results
In this section
[Equation]
we will provide an additional theorem that links an approximate q -value function with an approximate reward.
Should the paper be accepted, we would merge this theorem into our appendix.
Theorem 2.
Let r ∗ be an underlying true reward function and q ∗ be the corresponding optimal value function.
Given an approximate value function q
[Equation]
we denote by r the reward function derived from
Eqn.
[Equation]
Then
[Equation]
we must have ∥ r − r ∗ ∥ ∞ bounded by O ( ∥ q − q ∗ ∥ ∞ ) .
Here, ∥ · ∥ ∞ essentially takes the maximum absolute value over all s ∈ S and
a ∈ A
[Equation]
Proof.
For any s ∈ S and a ∈ A
[Equation]
we have
[Equation]
Here, Eqn.
(9) is from the Bellman equation; Eqn.
(10) follows the triangle inequality; and Eqn.
(11) generalizes certain s and a to all possible s ′ ∈ S , a ′ ∈ A .
Eqn.
(12) discusses two possible cases: whether max a ′ q ( s ′ , a ′ ) ≥ max a ′ q ∗ ( s ′ , a ′ ) or not.
Eqn.
(13) is because − max a ′ q ( s ′ , a ′ ) ≤ − q ( s ′ , a ′′ ) for any a ′′ ∈ A .
Eqn.
(14) merges all the maximum operation, and Eqn.
(15) is the definition of the infinity norm.
Since the last equation does not depend on s, a , we conclude ∥ r − r ∗ ∥ ∞ is bounded by O ( ∥ q − q ∗ ∥ ∞ ) .