Teacher Forcing Recovers Reward Functions for Text Generation
Abstract
Reinforcement learning (RL) has been widely used in text generation to alleviate the exposure bias issue or to utilize non-parallel datasets.
The reward function plays an important role in making RL training successful.
However, previous reward functions are typically task-speciﬁc and sparse, restricting the use of RL.
In our work, we propose a task-agnostic approach that derives a step-wise reward function directly from a model trained with teacher forcing.
We additionally propose a simple modiﬁcation to stabilize the RL training on non-parallel datasets with our induced reward function.
Empirical results show that our method outperforms selftraining and reward regression methods on several text generation tasks, conﬁrming the effectiveness of our reward function.
Introduction
Teacher forcing [7] is the common training method for text generation models.
Although this practice has been widely applied [7, 13, 58], there are two main issues: 1) Teacher-forcing training is data-hungry because parallel datasets are usually expensive to obtain.
On the other hand, there are numerous unlabeled, non-parallel datasets available.
This poses an urge to efﬁciently exploit non-parallel data.
Teacher forcing introduces a discrepancy between training and inference because the model learns to predict the next word based on the partial groundtruth reference during training, whereas in inference the model predicts the next word based on its self-generated previous words.
This undesired discrepancy is known as exposure bias [44, 6, 28, 59].
To address the ﬁrst problem, a straightforward method is to generate pseudo-parallel sentences for data augmentation, such as self-training [2], sequence-level knowledge distillation [23] and back-translation [49].
However, the exposure bias remains in such cases.
To address the second problem, the model should be trained on self-generated sentences.
Common solutions are often based on reinforcement learning (RL).
In text generation, however, there does not exist a naturally deﬁned reward function for RL.
Researchers have proposed various heuristic scores as the reward, such as BLEU [38] for translation and ROUGE [31] for summarization.
These reward functions are task-speciﬁc and not generalizable to other tasks.
Further, these rewards require parallel data, failing to address the ﬁrst problem above; they are typically sparse (only non-zero at the end of a sentence), making RL training difﬁcult.
The goal of this paper is to address these two problems in one framework with a learned, dense reward function.
Our approach has two steps: we ﬁrst train a sequence-to-sequence (seq2seq) model on the parallel dataset and induce a reward function from the model.
Then, we apply RL on non-parallel data based on our induced reward function.
Our method is task-agnostic and does not require handcrafted engineering or heuristics.
Further, our reward function provides dense (step-wise) training signals, which makes RL training much easier than sparse rewards.
Additionally, the reward function derived from the seq2seq model does not directly participate in the generation, which allows the model to explore based on its own prediction and thus alleviates the exposure bias.
We conduct experiments on dialogue generation and paraphrase generation.
The empirical results suggest that our method leads to better performance compared with several baselines, including self-training and task-speciﬁc heuristic reward learning, on both tasks.
This conﬁrms the effectiveness and generality of our framework.
Approach
Our approach trains the seq2seq model on non-parallel data with reinforcement learning, whose foundation is the Markov decision process (MDP).
In this section, we ﬁrst introduce the MDP formulation for text generation.
Then we describe our method to derive the reward function from a seq2seq model trained by teacher forcing.
Finally, we describe the policy gradient method used in RL training with our induced reward function.
2.1 Reinforcement Learning Formulation of Text Generation
Text Generation as a Markov Decision Process (MDP).
We formulate the text generation process as an (undiscounted) MDP, which can be represented as a tuple ( S , A , T, r ) .
At every step, a decision a 2 A is made based on its state s 2 S .
The transition dynamic T ( s 0 | s, a ) is the probability of the next state being s 0 , given the current state s and the action a .
A function r : S ⇥ A !
R deﬁnes the reward based on a state and an action.
Typically, the decision making is assisted by a policy ⇡ , which is a predicted distribution over actions and is trained to maximize the expected total reward, also known as an action value function: [Equation]
[Equation]
where H is the number of steps.
Theoretical results show that the optimal policy ⇡ ⇤ satisﬁes the
Bellman optimality equation:
[Equation]
For text generation, the MDP state can be deﬁned as the partial generated sequence y <t :=
( y 1 , · · · , y t  1 ) , and the action as the next token y t in the vocabulary V .
The transition dynamic T ( ·| s, a ) here is deterministic, since every state–action pair ( y <t , y t ) leads to a unique state y <
t
+ for the next step.
In previous RL-based text generation, there lacks a naturally deﬁned reward function r ( s, a ) .
While researchers have applied various heuristics as the reward [1, 50], they suffer from several shortcomings (e.g., sparsity and task speciﬁcity) as mentioned in Section 1.
To address these problems, we propose to induce a reward function for text generation tasks in a principled approach by inverse reinforcement learning.
Inverse Reinforcement Learning (IRL).
The goal of IRL is to learn a reward function r ( s, a ) .
Especially, we wish the resulting action value function q computed by Eqn.
(1) could satisfy q ( s, a )  q ( s, a 0 ) for every a 0 2 A and every ( s, a ) pair in the training set D .
In other words, the decisions in
D are made greedily by argmax a q ( s, a ) given any state s .
Unfortunately, Ng and Russell [36] show that this is an ill-posed problem since the desirable reward function r is not unique.
Therefore, we follow a common assumption [4, 43, 69] to resolve the ambiguity:
Assumption 1.
Given an action value function q , the policy ⇡ takes the form of ⇡ q ( a | s ) :
= exp( q ( s, a )) / P a 0 exp( q ( s, a 0 )) .
In traditional IRL [4, 43, 69], reward learning is difﬁcult and this assumption does not directly yield a reward function due to the stochastic state transition T ( s {+
+} | s, a ) .
However, our insight is that the transition is deterministic for text generation tasks, and thus we may utilize Assumption 1 to induce an action value function q , and then a reward function r , from some learned policy ⇡ , as explained in the next part.
2.2 Teacher Forcing Recovers IRL
One of our main contributions is that we show the seemingly complicated reward learning in
Section 2.1 can be recovered by teacher forcing, the de facto common practice of supervised text generation.
Our discovery leads to a convenient approach that derives a step-wise reward function simply from general seq2seq models, without the need for task-speciﬁc heuristics.
This makes RL more general for text generation, and our step-wise reward largely simpliﬁes RL training.
Maximum Likelihood Estimation (MLE) for IRL.
Following Assumption 1, we let the policy ⇡ q ! ( ·| s ) / exp( q ! ( s, · )) , where q ! is a parameterized action value function.
Under such a policy, the probability of each trajectory ⌧ :=
(( s 1 , a 1 ) , . .
, ( s | ⌧ | , a | ⌧ | )) in the dataset is given by the trajectory distribution P ⇡ q ! .
The likelihood of the dataset is given by
[Equation]
Teacher-Forcing Training.
For text generation, the standard teacher-forcing seq2seq training is to minimize the loss:
[Equation]
where the predicted probability of the next token being v is p ! ( v | y <t ) =
[Equation]
for the logit function f ! with parameters ! .
In seq2seq training, an additional input x may be added to the conditional probabilities but is omitted here for simplicity.
The below theorem shows their equivalence up to an additional constant.
Theorem 1.
Suppose the value function q in Eqn.
(3) and the seq2seq model f in Eqn.
(4) have the same parametrization ! , we have
[Equation]
Proof.
For the MLE of IRL under Assumption [Equation] the Ionescu–Tulcea theorem [22] asserts that there exists a unique trajectory distribution P ⇡ µ satisfying
[Equation]
for any t  1 , given the initial state distribution µ , transition probability T , and policy ⇡ .
The likelihood can thus be factorized by the multiplication of µ , T , and ⇡ : [Equation] As mentioned
[Equation]
text generation has a deterministic transition, i [Equation] e., T ( s 0 | s, a ) =
for the next state s {+
+}
= s
+ [ a ] .
Taking the µ terms out, we have
[Equation]
where the second term is a constant in terms of ! .
In Section 2.1, text generation is modeled as an MDP with s t = y <t and a t = y t .
Therefore, the ﬁrst term of Eqn.
(6) is the same as Eqn.
(4) under the parametrization ⇡ q ! = p ! , concluding the equivalence between MLE for IRL and the teacher-forcing training of a seq2seq model.
Inducing the Reward Function.
Theorem 1 shows that seq2seq training with teacher forcing actually learns an IRL model.
Thus, we may derive a reward function assuming the action value function is well trained:
[Equation]
where the ﬁrst equality is due to the Bellman optimality condition (2); the second equality is due to the parametrization of q ! = f ! and the deterministic transition T ( s 0 | s, a ) = {+
+}
for s 0 being the concatenation of the preﬁx s and token a .
Remark 1 . It is easy to notice that f ! ( s, · ) may be arbitrarily shifted by a constant c s without changing ⇡ ! . This also shifts the derived reward r ( s, a ) by c s  c
s
+[ a ] .
However, it does not affect the optimal policy.
We will prove this in Theorem 3 after introducing policy gradient methods.
Our use of the Bellman optimality condition is different from classic RL, where the reward is welldeﬁned and the action value function is thus learned [55].
Instead, we induce the underlying reward assuming the action value function is known (given by Assumption 1).
The following diagram shows the whole process of our derivation.
[Equation]
In real-world applications, the learned action value function might be imperfect; in this case, we may bound the error of our induced reward with the following theorem.
Theorem 2.
Let r ⇤ be an underlying true reward function and q ⇤ be the corresponding optimal value function.
Given an approximate value function q , we denote by r the reward function derived from Eqn.
Then, we must have k r  r ⇤ k 1 bounded by O ( k q  q ⇤ k 1 ) .
Here, k · k 1 takes the maximum absolute value over all s 2 S and a 2 A .
Proof.
See Appendix A.
2.3 Periodically Synchronized Behavior Policy in Policy Gradient
In text generation, a neural network can be viewed as a policy ⇡ that predicts the word distribution given the state of a decoding step.
The reward induced from Section 2.2 can be used to improve the policy through RL.
To stabilize training, we propose a variant of off-policy policy gradient methods [10] with a periodically synchronized behavior policy.
Our RL training adopts the off-policy REINFORCE [63] as the backbone of our algorithm.
Let ⇡ ' be the model policy (i.e., the model’s prediction) to be optimized, and ⇡ b be the behavior policy (i.e., the sampling distribution during training).
Through importance sampling, the gradient of the expected total reward with respect to ' can be obtained by the off-policy policy gradient theorem [10]
[Equation]
where ⇢ t := ⇡ ' ( a t | s t ) / ⇡ b ( a t | s t ) is the importance weight, and ˆ q r ( s t , a t ) :=
P i  t r
( s i , a i ) is the total reward of the trajectory.
In practice, off-policy REINFORCE ( ⇡ ' 6 = ⇡ b ) is more exploratory than the on-policy one ( ⇡ ' = ⇡ b ), since the model policy ⇡ ' would become more concentrated during optimization and does not explore much, whereas ⇡ b is typically chosen to cover more trajectories.
However, Degris et al.
[10] adopt a ﬁxed behavior policy ⇡ b , which does not perform exploitation according to the current model policy.
The lack of exploitation might lead to less informative training.
To balance exploration and exploitation, we would like the behavior policy to be close to the model policy but stay exploratory at the same time.
We thus propose a periodically updating schedule, where the behavior policy is frozen for a long period to encourage exploration but keeps track of the current model policy to enhance exploitation.
Particularly, we synchronize the behavior policy with the model policy for every k gradient updates of the latter (e.g., k = 5000 ).
Our remedy is a simple method overcoming the instability of REINFORCE.
It shares a common ground with a number of policy gradient methods like the proximal policy optimization (PPO) [48], especially in that both methods involve multiple updates with a ﬁxed behavior policy.
As the main contribution of this paper is reward induction, we resort to this simple ﬁx and leave the mathematical connection as an interesting future direction.
Algorithm 1 summarizes our approach.
Our implementation is able to execute the loops in parallel, which speeds up the training process.
Our periodically synchronized behavior policy further enables us to parallelize sampling and model updates to reduce the awaiting time.
2.4 Application to Semi-Supervised Learning
Our approach naturally aligns with the paradigm of semi-supervised learning, as it involves training a seq2seq model to induce the reward function, which requires (at least a small volume of) parallel data
D p .
Additionally, we assume there is a non-parallel dataset D u containing input sentences only for
RL training with the induced reward.
Our semi-supervised approach consists of two stages.
We ﬁrst train a seq2seq model f ! on the parallel dataset D p to induce the reward function r by Eqn.
The procedure is described in Section 2.2.
The reward function then facilitates RL training on the non-parallel dataset D u , which is shown in
Algorithm 1.
Algorithm 1: Our Algorithm
Input: A non-parallel dataset D u , learned logit (value) function f ! , policy ⇡ ' with the initial parameter ' , total update steps U , and synchronizing period k
Output: A policy ⇡ ' parameterized by ' beginfor i 1 ...
U do if i ⌘ 0 (mod k ) then ⇡ b ⇡ ' ; . Behavior policy update
Sample a source sentence x 2 D u Construct the initial state s ( x , [BOS] ) ; . [BOS] is the beginning token
Sample a trajectory ⌧ from the behavior policy ⇡ b ˆ q rh + 0 and g for t | ⌧ | ... 1 do if t = | ⌧ | then r t f ! ( s t , a t ) ; .
Termination step, no s else r t f ! ( s t , a t )  max a 0 f ! ( s t +1 , a 0 ) ; .
By Eqn.
(7) ˆ
[Equation]
+ ˆ [Equation] . Accumulating rewards [Equation]
. Importance weight
[Equation]
[Equation]
. Gradient ascent
As mentioned in Remark 1, the reward r ( s, a ) can be arbitrarily shifted by c s  c s +[ a ] .
We show that this shift does not affect the optimal policy.
Theorem 3.
Suppose r 0 ( s, a ) =
r ( s, a ) + c s  c s +[ a ] .
Then the learned policies under r 0 ( s, a ) and r ( s, a ) are the same.
Proof.
By Eqn.
(1), function q returns the expected total reward.
In Algorithm 1, we sample it by ˆ q r 0 t ( s t , a t ) :
= r 0 ( s t , a t ) + r 0 ( s t +1 , a t +1 ) + · · · + r 0 ( s | ⌧ | , a | ⌧ | )
[Equation]
⌧ |
[Equation] The last line suggests the constant plays a role as the baseline in policy gradient, which is shown to be irrelevant to the optimal policy [55]. [Table] [Table]
[Table]
Experiments 3.1 Datasets and Metrics
Dialogue Generation.
We adopt two widely used datasets, DailyDialog [30] and OpenSubtitles [57], for the dialogue experiment.
The DailyDialog dataset is constructed from English dialogues crawled from the Internet, whereas the OpenSubtitles dataset is constructed from movie subtitles based on IMDB identiﬁers.
A dialogue session is split into single-turn context–response pairs in our experiment.
For semi-supervised learning, we use the smaller dataset, DailyDialog, as the parallel corpus D p , and the larger dataset, OpenSubtitles, as the non-parallel corpus D u (i.e., we only retain the context sentence in the OpenSubtitles dataset).
This follows the common setup for semi-supervised learning, where the unlabeled dataset is larger than the labeled one.
It should be emphasized that a recent study [62] shows more than 20% of test samples are identical to some training samples in both DailyDialog and OpenSubtitles.
This results in meaningless comparison and inﬂated performance of previous methods, e.g., a BLEU4 of 11.01 in AdaLabel [60] and 14.61 in DialogBERT [16].
Therefore, we use the deduplicated datasets in [62], containing 60K/6.5K/7K samples for training/validation/test in DailyDialog and 1M non-parallel samples in
OpenSubtitles.
Although our scores will be lower than previous inﬂated ones, we follow the correct setting for research.
We use BLEU scores [38] as main evaluation metrics, which are widely used in dialogue generation [62].
In particular, BLEU- n evaluates the geometric average of i -gram precision scores for i
= 1 , · · · , n .
Following previous work [62], we lowercase all sentences and tokenize them with the
NLTK library [33].
Paraphrase Generation.
We follow previous studies [11, 32, 34] and use the Quora Question Pair dataset 2 for the paraphrasing experiment.
The Quora dataset is originally designed for paraphrase classiﬁcation, containing both paraphrase and non-paraphrase pairs.
The paraphrase pairs naturally form a parallel dataset for the generation purpose; following the common practice [34], we split it into 124K/4K/20K samples for training/validation/test.
The non-paraphrase pairs, containing 510K sentences, are discarded in previous work, but we are able to utilize them in a semi-supervised manner.
We use the standard iBLEU score [54] as the main evaluation metric.
It involves a penalty of SelfBLEU (SBLEU) between the generated and input sentences, as the paraphrasing task requires using different lexicons.
Speciﬁcally, it is calculated by iBLEU = (1  ↵ )
BLEU  ↵ SBLEU, where ↵ is typically set to 0 .
For clarity, we also report BLEU and S-BLEU scores in our experiment.
3.2 Settings and Competing Methods
For each task, we ﬁrst ﬁne-tune a T5-Base model [42] on the parallel data by Eqn.
Then we apply our proposed method to induce the reward and further train the model by Algorithm 1 on the non-parallel data.
We compare our approach with the following semi-supervised methods.
Self-Training.
We apply the supervised model to the non-parallel dataset and generate pseudo-target sentences, which are used to continue training the model.
This is a commonly used semi-supervised approach in text generation literature [21, 68].
R-Regression.
Wu et al.
[66] propose a reward regression (R-Regression) approach, where the reward is deﬁned as the BLEU score.
Since their reward is the same as the evaluation metric, such a method may achieve higher BLEU scores without actually improving the generation quality.
By contrast, our reward is induced in a principled way and is agnostic to evaluation metrics.
In our experiment, we replicate the R-regression method, which constitutes a controlled comparison to our approach, as the only difference is the reward function.
Appendix B provides implementation details and hyperparameters of our approach.
3.3 Main Results
Results of Dialogue Generation.
Table 1a shows the results of the dialogue generation task.
We notice that our ﬁne-tuned T5-Base model [42] has already outperformed dedicated methods,
AdaLabel [60] and DialogBERT [16].
This is consistent with the ﬁndings of [62, 61] in that the alleged “state-of-the-art” dialogue systems do not outperform standard pretrained language models on deduplicated datasets, highlighting the importance of working with the correct setting.
We then apply semi-supervised learning (Self-Training, R-Regression, and our approach) with the non-parallel OpenSubtitles dataset.
We achieve higher performance than T5-Base trained only on parallel DailyDialog.
Interestingly, the fully supervised model—trained on both parallel DailyDialog and parallel OpenSubtitles—does not achieve high performance, even lower than the one trained with
DailyDialog only.
It is noticed that the OpenSubtitles dataset is noisy [8], which likely causes the performance degradation.
This signiﬁes the need of semi-supervised learning.
Among semi-supervised approaches, RL-based methods (R-Regression and ours) are generally better than Self-Training.
This is within our expectation because Self-Training learns from its own generation and may be overconﬁdent, whereas RL approaches are able to explore different parts of the data space, being a more effective way of semi-supervised learning.
Moreover, our approach outperforms RL with R-Regression, where the reward is the only difference.
The controlled experiment conﬁrms that the reward induced from models trained with teacher forcing is effective for RL training.
It is also worth noting that R-Regression uses the evaluation metric as the reward, and thus may deliberately improve the metric rather than text quality.
By contrast, our reward is induced in a principled manner and is agnostic to evaluation metrics, and our approach still achieves higher performance even with such a disadvantage.
[Table]
In general, our approach achieves the best performance in both metrics.
In particular, it signiﬁcantly improves DailyDialog-trained T5-Base by +2.06 (+23.0%) in BLEU2 and +0.61 (+16.5%) in BLEU4.
It also outperforms the second-best method, R-Regression, by 0.68 (+6.6%) in BLEU2 and 0.
(+2.9%) in BLEU4, verifying the effectiveness of our approach.
Results of Paraphrase Generation.
The results of paraphrase generation are shown in Table 1b.
As seen, directly copying the input already achieves a high BLEU score against the reference.
iBLEU addresses this by penalizing the SelfBLEU score (against input) and is considered the main metric.
We consider another semi-supervised baseline LTSL [11].
It performs retrieval-based paraphrase expansion and meta optimization, thus being task speciﬁc.
We see that LTSL has an extremely high Self-BLEU, suggesting the generated paraphrase largely resembles the input.
It achieves a lower iBLEU score than other semi-supervised approaches.
We also see that RL approaches generally achieve lower SelfBLEU than Self-Training.
This is because Self-Training learns from its own predictions, which overlap the input more than groundtruth paraphrases do (Self-BLEU of groundtruth: 29.87); as a result, Self-BLEU increases to 48.02 from 44.77 of T5Base.
By contrast, RL learns by exploring different possible paraphrases and is able to retain low Self-BLEU.
Overall, our approach achieves the highest BLEU and a reasonably low Self-BLEU, yielding the best iBLEU among all competing methods.
The results are consistent with Table 1a, showing the generality of our approach.
3.4 Analyses
Step-Wise Reward.
In Figure 1, we show the distributions of different reward functions.
As seen, the BLEU score is mostly concentrated at 0, providing little information for training.
RRegression consequently suffers from a similar problem, as it is trained by the groundtruth BLEU scores.
The distribution of our induced reward, on the other hand, has the lowest peak and is the most wide-spreading one.
We conduct another analysis to show the importance of step-wise rewards for RL training.
We compare our approach with a sparse reward function that defers all rewards to the end of a sentence.
In other words, the last step’s reward is the sum of our step-wise rewards, whereas all previous steps have a reward of 0.
This constitutes a rigorous analysis, as the total reward and thus the training objective are the same in both cases.
Results in Table 2 show that our step-wise reward outperforms the sparse reward in all cases.
This suggests our approach serves as a meaningful credit assignment of the total reward, which is beneﬁcial for RL training.
The Effect of the Synchronizing Period.
We analyze the effect of the synchronizing period k introduced in Section 2.3.
In Figure 2, we see that the training is unstable if k = 1 (on-policy), in which case the model generates uninformative and meaningless sentences (illustrated in Appendix D).
When k = 1000 , the performance increases quickly at the beginning, but it starts to decrease with further training.
We hypothesize that this is due to the lack of exploration (Section 2.3).
When k is inﬁnitely large (the behavior policy is ﬁxed), the performance grows slowly and stops improving after a certain number of steps.
Based on this analysis, we choose k = 5000 to balance exploitation and exploration.
Although the experiment is conducted only on DailyDialog due to the limit of time and resources, we directly apply the setting to other experiments, showing the robustness of our approach.
Data Efﬁciency.
In Figure 3, we analyze data efﬁciency by sampling different numbers of data points from the non-parallel corpus.
As shown, our method consistently outperforms self-training, even with only 0.1% (the leftmost points) of the training set.
Additionally, the performance of our method quickly increases with more data, whereas self-training grows slowly.
This is expected because RL training explores different parts of the sentence space and learns from their rewards, whereas self-training only learns from the single generated sentence by the model itself given an input.
We also investigate how performance changes according to the size of the parallel dataset, which reﬂects the quality of the learned policy.
Results are shown in Figure 4, Appendix C.
Related Work
Scores are measured on the DailyDialog test set.
Semi-Supervised Learning for Text Generation.
In text generation, popular ways to utilize both parallel and non-parallel data include self-training [21, 68] and backtranslation [49].
Both methods ﬁrst train a model on the parallel data and then generate pseudo-parallel pairs for the non-parallel sentences.
The difference is that self-training generates pseudoparallel pairs from source to target, whereas back-translation generates from target to source.
We mainly consider selftraining as a baseline because it does not require an additional model in the reversed direction, making the comparisons fairer.
Our implementation of self-training is also similar to sequencelevel knowledge distillation [23, 15, 20], except that the latter augments the parallel data instead of the non-parallel ones.
In
Figure 3, we show that self-training cannot efﬁciently utilize the data because of the lack of exploration.
Additionally, the exposure bias issue remains because they are trained with the teacher-forcing objective.
Text Generation beyond Teacher Forcing.
Teacher forcing is known to have the exposure bias issue.
A line of work uses the generative adversarial network (GAN) [14] to alleviate the issue.
For example, Yu et al.
[67] and Guo et al.
[18] propose to use
GAN-style training to generate text similar to the training set in an on-the-ﬂy manner.
This practice reduces the discrepancy between training and inference because GAN sends its own generation as inputs rather than using groundtruth sentences during training.
Shi et al.
[51] further formulate the adversarial training using the IRL interpretation.
These GAN-style methods are different from ours in two main ways.
First, GAN-style training requires parallel corpora and thus cannot be directly applied to semi-supervised learning on non-parallel datasets.
Second, GAN-style training involves the optimization of an adversarial objective, making the training unstable, e.g., suffering from mode collapse [14].
Another paradigm to alleviate the exposure bias is RL.
For instance, Sokolov et al.
[53] and Kreutzer et al.
[26] leverage the bandit-structured prediction framework for text generation with BLEU as the heuristically deﬁned reward.
Bahdanau et al.
[1] and Shen et al.
[50] utilize different variants of policy gradient for RL training.
However, these methods are task-speciﬁc and suffer from the problem of sparse rewards, as mentioned in Section 3.4.
More importantly, these approaches require parallel data to calculate the reward and cannot utilize non-parallel data either.
To address this, Wu et al.
[66] propose to learn a reward regression model on the parallel dataset and perform RL on the non-parallel data with the learned reward.
As mentioned, such a method is still task-speciﬁc because it requires the human heuristics of the task to deﬁne the proper reward function.
Additionally, it suffers from the reward-sparsity problem, as seen in Figure 1 and Table 2.
Search is also a popular way to replace teacher forcing.
The Learning to Search (L2S) framework [5, 9] enables the model to search for a better score during learning and is widely applied to text generation.
For example, Wiseman and Rush [64] propose to optimize the beam search results through training.
Li et al.
[29] develop an unsupervised learning approach to text generation based on local search.
In addition to the L2S framework, Leblond et al.
[27] leverage the Monte Carlo tree search [25, 52] to select better tokens in a step from the sampled generation.
These methods are different from ours since they need either heuristically deﬁned scoring functions or parallel data, limiting their methods to certain tasks or to the supervised paradigm.
However, given the success of these methods, we consider the search-based approach an interesting future extension of our work.
Imitation Learning.
The intuition behind our work is also related to imitation learning methods in general.
Typically, these methods aim to obtain a good policy given a dataset containing state–action pairs.
The easiest approach is behavior cloning [39], which greedily imitates the demonstration.
Similar to the exposure bias, behavior cloning also faces the problem of compounding errors [46].
SMILe [46] and DAgger [47] mitigate the problem by querying an expert.
In text generation, Du and Ji [12] empirically verify that imitation learning methods are helpful.
Recently, Pang and He [37] frame the text generation task as an ofﬂine reinforcement learning problem, which learns from a dataset containing tuples of state, action, and reward.
Compared with our method, these approaches rely on parallel sentence pairs and cannot effectively make use of non-parallel datasets.
Conclusion
Summary.
In this paper, we show that a reward function can be derived from a model trained with teacher forcing.
The derivation does not rely on human heuristics for certain tasks.
Additionally, the derived reward function assigns step-wise scores and makes the RL training easier.
Our approach leads to a training algorithm in a semi-supervised manner and utilizes both parallel and non-parallel data.
We conduct experiments on the dialogue and paraphrase generation tasks.
The empirical results show that the performance of our approach is better compared with the baselines: self-training and reward regression.
We further analyze our reward function and show the beneﬁts of our approach.
Limitation and Future Work.
First, the scale of the experiments in this paper is restricted by computational resources.
It is interesting to see if our approach could obtain better performance with large models [3, 41] and larger datasets.
We also notice that Assumption 1 has a deep connection with entropy-regularized RL [17, 19, 45].
Our approach can be easily extended to such cases in the future.
Another interesting direction would be using the reward as an interface between humans and the model to control the generation.
Speciﬁcally, the current seq2seq models treat data as the ground truth, but the data may be contaminated with undesired or harmful information.
We hope that our approach provides a way for humans to apply additional rules to the reward function to avoid the model generating harmful information. Appendices
The full paper, including appendices, is available at https://arxiv.org/abs/2210.08708 .
Acknowledgments
We thank all reviewers for their valuable comments.
We also thank Guoqing Luo for early discussions.
The research is supported in part by the Natural Sciences and Engineering Research Council of
Canada (NSERC) under grant No.
RGPIN2020-04465, the Amii Fellow Program, the Canada CIFAR
AI Chair Program, a UAHJIC project, a donation from DeepMind, and the Digital Research Alliance of Canada (alliancecan.ca).