Teacher Forcing Recovers Reward Functions for Text Generation
Abstract
Reinforcement learning (RL) has been widely used in text generation to alleviate the exposure bias issue or to utilize non-parallel datasets.
The reward function plays an important role in making RL training successful.
However, previous reward functions are typically task-specific and sparse, restricting the use of RL.
In our work, we propose a task-agnostic approach that derives a step-wise reward function directly from a model trained with teacher forcing.
We additionally propose a simple modification to stabilize the RL training on non-parallel datasets with our induced reward function.
Empirical results show that our method outperforms selftraining and reward regression methods on several text generation tasks, confirming the effectiveness of our reward function.
Introduction
Teacher forcing [7] is the common training method for text generation models.
Although this practice has been widely applied [7, 13, 58], there are two main issues: 1) Teacher-forcing training is data-hungry because parallel datasets are usually expensive to obtain.
On the other hand, there are numerous unlabeled, non-parallel datasets available.
This poses an urge to efficiently exploit non-parallel data.
Teacher forcing introduces a discrepancy between training and inference because the model learns to predict the next word based on the partial groundtruth reference during training, whereas in inference the model predicts the next word based on its self-generated previous words.
This undesired discrepancy is known as exposure bias [44, 6, 28, 59]
To address the first problem a straightforward method is to generate pseudo-parallel sentences for data augmentation, such as self-training [2], sequence-level knowledge distillation [23] and back-translation [49].
However, the exposure bias remains in such cases.
To address the second problem, the model should be trained on self-generated sentences.
Common solutions are often based on reinforcement learning (RL).
In text generation, however, there does not exist a naturally defined reward function for RL.
Researchers have proposed various heuristic scores as the reward, such as BLEU [38] for translation and ROUGE [31] for summarization.
These reward functions are task-specific and not generalizable to other tasks.
Further, these rewards require parallel data, failing to address the first problem above; they are typically sparse (only non-zero at the end of a sentence), making RL training difficult.
The goal of this paper is to address these two problems in one framework with a learned, dense reward function.
Our approach has two steps: we first train a sequence-to-sequence (seq2seq) model on the parallel dataset and induce a reward function from the model.
Then, we apply RL on non-parallel data based on our induced reward function.
Our method is task-agnostic and does not require handcrafted engineering or heuristics.
Further, our reward function provides dense (step-wise) training signals, which makes RL training much easier than sparse rewards.
Additionally, the reward function derived from the seq2seq model does not directly participate in the generation, which allows the model to explore based on its own prediction and thus alleviates the exposure bias.
We conduct experiments on dialogue generation and paraphrase generation.
The empirical results suggest that our method leads to better performance compared with several baselines, including self-training and task-specific heuristic reward learning, on both tasks.
This confirms the effectiveness and generality of our framework.
Approach
Our approach trains the seq2seq model on non-parallel data with reinforcement learning, whose foundation is the Markov decision process (MDP).
In this section, we first introduce the MDP formulation for text generation.
Then we describe our method to derive the reward function from a seq2seq model trained by teacher forcing.
Finally, we describe the policy gradient method for RL training with our induced reward function.
2.1 Reinforcement Learning Formulation of Text Generation
Text Generation as a Markov Decision Process (MDP).
We formulate the text generation process as an (undiscounted) MDP, which can be represented as a tuple ( S , A , T, r ) .
At every step, a decision a ∈ A is made based on its state s ∈ S .
The transition dynamic T ( s ′ | s, a ) is the probability of the next state being s ′ , given the current state s and the action a .
A function r : S ×
A → R defines the reward based on a state and an action.
Typically, the decision making is assisted by a policy π , which is a predicted distribution over actions and is trained to maximize the expected total reward, also known as an action value function:
[Equation]
where H is the number of steps.
Theoretical results show that the optimal policy π ∗ satisfies the
Bellman optimality equation:
[Equation]
For text generation, the MDP state can be defined as the partial generated sequence y <t :=
( y 1 , · · · , y t − 1 ) , and the action as the next token y t in the vocabulary V .
The transition dynamic
T ( ·| s, a ) here is deterministic, since every state–action pair ( y <t , y t ) leads to a unique state y <
t
+ for the next step.
In previous RL-based text generation, there lacks a naturally defined reward function r ( s, a ) .
While researchers have applied various heuristics as the reward [1, 50], they suffer from several shortcomings (e.g., sparsity and task specificity) as mentioned in Section 1.
To address these problems, we propose to induce a reward function for text generation tasks in a principled approach by inverse reinforcement learning.
Inverse Reinforcement Learning (IRL).
The goal of IRL is to learn a reward function r ( s, a ) .
Especially, we wish the resulting action value function q computed by Eqn.
(1) could satisfy q ( s, a ) ≥ q ( s, a ′ ) for every a ′ ∈ A and every ( s, a ) pair in the training set D .
In other words, the decisions in
D are made greedily by argmax a q ( s, a ) given any state s .
Unfortunately, Ng and Russell [36] show that this is an ill-posed problem since the desirable reward function r is not unique.
Therefore, we follow a common assumption [4, 43, 69] to resolve the ambiguity:
Assumption 1.
Given an action value function q , the policy π takes the form of π q ( a | s ) :
=
exp( q ( s, a )) / (cid:80) a ′ exp( q ( s, a ′ )) .
In traditional IRL [4, 43, 69], reward learning is difficult and this assumption does not directly yield a reward function due to the stochastic state transition T ( s ′ | s, a ) .
However, our insight is that the transition is deterministic for text generation tasks, and thus we may utilize Assumption 1 to induce an action value function q , and then a reward function r , from some learned policy π , as explained in the next part.
2.2 Teacher Forcing Recovers IRL
One of our main contributions is that we show the seemingly complicated reward learning in
Section 2.1 can be recovered by teacher forcing, the de facto common practice of supervised text generation.
Our discovery leads to a convenient approach that derives a step-wise reward function simply from general seq2seq models, without the need for task-specific heuristics.
This makes RL more general for text generation, and our step-wise reward largely simplifies RL training.
Maximum Likelihood Estimation (MLE) for IRL.
Following Assumption 1, we let the policy π q ω ( ·| s ) ∝ exp( q ω ( s, · )) , where q ω is a parameterized action value function.
Under such a policy, the probability of each trajectory τ :=
(( s 1 , a 1 ) , . .
, ( s | τ | , a | τ | )) in the dataset is given by the trajectory distribution P π qω .
The likelihood of the dataset is given by
[Equation]
Teacher-Forcing Training.
For text generation, the standard teacher-forcing seq2seq training is to minimize the loss:
[Equation]
where the predicted probability of the next token being v is p ω ( v | y <t )
=
[Equation]
for the logit function f ω with parameters ω .
In seq2seq training, an additional input x may be added to the conditional probabilities but is omitted here for simplicity.
The below theorem shows their equivalence up to an additional constant.
Theorem 1.
Suppose the value function q in Eqn.
(3) and the seq2seq model f in Eqn.
(4) have the same parametrization ω , we have
[Equation]
Proof.
For the MLE of IRL under Assumption 1, the Ionescu–Tulcea theorem [22] asserts that there exists a unique trajectory distribution P πµ satisfying
[Equation]
for any t ≥ 1 , given the initial state distribution µ , transition probability T , and policy π .
The likelihood can thus be factorized by the multiplication of µ , T , and π :
[Equation]
As mentioned, text generation has a deterministic transition, i.e., T ( s ′ | s, a ) =
for the next state s ′
=
s + [ a ] .
Taking the µ terms out, we have
[Equation]
where the second term is a constant in terms of ω .
In Section 2.1, text generation is modeled as an MDP with s t = y <t and a t =
y t .
Therefore, the first term of Eqn.
(6) is the same as Eqn.
(4) under the parametrization π q ω = p ω , concluding the equivalence between MLE for IRL and the teacher-forcing training of a seq2seq model.
Inducing the Reward Function.
Theorem 1 shows that seq2seq training with teacher forcing actually learns an IRL model.
Thus, we may derive a reward function assuming the action value function is well trained:
[Equation]
where the first equality is due to the Bellman optimality condition (2); the second equality is due to the parametrization of q ω = f ω and the deterministic transition T ( s ′ | s, a ) = 1 for s ′ being the concatenation of the prefix s and token a .
Our use of the Bellman optimality condition is different from classic RL, where the reward is welldefined and the action value function is thus learned [55].
Instead, we induce the underlying reward assuming the action value function is known (given by Assumption 1).
The following diagram shows the whole process of our derivation.
[Equation]
In real-world applications, the learned action value function might be imperfect, and in this case, we may bound the error of our induced reward with the following theorem.
Theorem 2.
Let r ∗ be an underlying true reward function and q ∗ be the corresponding optimal value function.
Given an approximate value function q , we denote by r the reward function derived from Eqn.
Then, we must have ∥ r − r ∗ ∥ ∞ bounded by O ( ∥ q − q ∗ ∥ ∞ ) .
Here, ∥ · ∥ ∞ takes the maximum absolute value over all s ∈ S and a ∈ A .
Proof.
See Appendix A.
2.3 Periodically Synchronized Behavior Policy in Policy Gradient
In text generation, a neural network can be viewed as a policy π that predicts the word distribution given the state of a decoding step.
The reward induced from Section 2.2 can be used to improve the policy through RL.
To stabilize training, we propose a variant of off-policy policy gradient methods [10] with a periodically synchronized behavior policy.
Our RL training adopts the off-policy REINFORCE [63] as the backbone of our algorithm.
Let
π
φ be the model policy (i.e., the model’s prediction) to be optimized, and π b be the behavior policy (i.e., the sampling distribution during training).
Through importance sampling, the gradient of the expected total reward with respect to φ can be obtained by the off-policy policy gradient theorem [10]
[Equation]
where ρ t := π φ ( a t | s t )
/π
b ( a t | s t ) is the importance weight, and ˆ q r ( s t , a t ) :
= (cid:80)
i ≥ t r ( s i , a i ) is the total reward of the trajectory.
In practice, off-policy REINFORCE ( π φ ̸ = π b ) is more exploratory than the on-policy one ( π φ = π b ), since the model policy π φ would become more concentrated during optimization and does not explore much, whereas π b is typically chosen to cover more trajectories.
However, Degris et al.
[10] adopt a fixed behavior policy π b , which does not perform exploitation according to the current model policy.
The lack of exploitation might lead to less informative training.
To balance exploration and exploitation, we would like the behavior policy to be close to the model policy but stay exploratory at the same time.
We thus propose a periodically updating schedule, where the behavior policy is frozen for a long period to encourage exploration but keeps track of the current model policy to enhance exploitation.
Particularly, we synchronize the behavior policy with the model policy for every k gradient updates of the latter (e.g., k = 5000 ).
Our remedy is a simple method overcoming the instability of REINFORCE.
It shares a common ground with a number of policy gradient methods like the proximal policy optimization (PPO) [48], especially in that both methods involve multiple updates with a fixed behavior policy.
As the main contribution of this paper is reward induction, we resort to this simple fix and leave the mathematical connection as an interesting future direction.
Algorithm 1 summarizes our approach.
In the implementation, we are able to execute the loops in parallel, which speeds up the training process.
Our periodically synchronized behavior policy further enables us to parallelize sampling and model updates to reduce the awaiting time.
2.4 Application to Semi-Supervised Learning
Our approach naturally aligns with the paradigm of semi-supervised learning, as it involves training a seq2seq model to induce the reward function, which requires (at least a small volume of) parallel data
D p .
Additionally, we assume there is a non-parallel dataset D u containing input sentences only for
RL training with the induced reward.
Our semi-supervised approach consists of two stages.
We first train a seq2seq model f ω on the parallel dataset D p to induce the reward function r by Eqn.
The procedure is described in Section 2.2.
The reward function then facilitates RL training on the non-parallel dataset D u , which is shown in
Algorithm 1.
Algorithm 1: Our Algorithm
Input: A non-parallel dataset D u , learned logit (value) function f ω , policy π φ with the initial parameter φ , total update steps U , and synchronizing period k
Output: A policy π φ parameterized by φ beginfor i ← 1 ...
U do if i ≡ 0 (mod k ) then π b ← π φ ; ▷ Behavior policy update
Sample a source sentence x ∈ D u
Construct the initial state s ← ( x , [BOS] ) ; ▷ [BOS] is the beginning token
Sample a trajectory τ from the behavior policy π b ˆ q rh +1 ← 0 and g ←for t ← | τ | ... 1 do if t = | τ | then r t ← f ω ( s t , a t ) ; ▷ Termination step,
no s t + else r t ← f ω ( s t , a t ) − max a ′ f ω ( s t +1 , a ′ ) ;
▷ By Eqn.
(7) ˆ q rt ← r t + ˆ q rt +1 ; ▷ Accumulating rewards ρ t ← π φ ( a t | s t ) /π
b ( a t | s t ) ; ▷ Importance weight g
Experiments 3.1 Datasets and Metrics
Dialogue Generation.
We adopt two widely used datasets, DailyDialog [30] and OpenSubtitles [57], for the dialogue experiment.
The DailyDialog dataset is constructed from English dialogues crawled from the Internet, whereas the OpenSubtitles dataset is constructed from movie subtitles based on IMDB identifiers.
A dialogue session is split into single-turn context–response pairs in our experiment.
For semi-supervised learning, we use the smaller dataset, DailyDialog, as the parallel corpus D p , and the larger dataset, OpenSubtitles, as the non-parallel corpus D u (i.e., we only retain the context sentence in the OpenSubtitles dataset).
This follows the common setup for semi-supervised learning, where the unlabeled dataset is larger than the labeled one.
It should be emphasized that a recent study [62] shows more than 20% of test samples are identical to some training samples in both DailyDialog and OpenSubtitles.
This results in meaningless comparison and inflated performance of previous methods, e.g., a BLEU4 of 11.01 in AdaLabel [60] and 14.61 in DialogBERT [16].
Therefore, we use the deduplicated datasets in [62], containing
[Table] [Table]
[Table] [Table]
[Table] [Table]
60K/6.5K/7K samples for training/validation/test in DailyDialog and 1M non-parallel samples in
OpenSubtitles.
Although our scores will be lower than previous inflated ones, we follow the correct setting for research.
We use BLEU scores [38] as main evaluation metrics, which are widely used in dialogue generation [62].
In particular, BLEU- n evaluates the geometric average of i -gram precision scores for i
= 1 , · · · , n .
Following previous work [62], we lowercase all sentences and tokenize them with the
NLTK library [33].
Paraphrase Generation.
We follow previous studies [11, 32, 34] and use the Quora Question Pair dataset 2 for the paraphrasing experiment.
The Quora dataset is originally designed for paraphrase classification, containing both paraphrase and non-paraphrase pairs.
The paraphrase pairs naturally form a parallel dataset for the generation purpose; following the common practice [34], we split it into 124K/4K/20K samples for training/validation/test.
The non-paraphrase pairs, containing 510K sentences, are discarded in previous work, but we are able to utilize them in a semi-supervised manner.
We use the standard iBLEU score [54] as the main evaluation metric.
It involves a penalty of SelfBLEU (SBLEU) between the generated and input sentences, as the paraphrasing task requires using different lexicons.
Specifically, we have iBLEU = α BLEU − (1 − α )
SBLEU, where α is typically set to 0 .
For clarity, we also report BLEU and Self-BLEU scores in our experiment.
3.2 Settings and Competing Methods
For each task, we first fine-tune a T5-Base model [42] on the parallel data by Eqn.
Then we apply our proposed method to induce the reward and further train the model by Algorithm 1 on the non-parallel data.
We compare our approach with the following semi-supervised methods.
Self-Training.
We apply the supervised model to the non-parallel dataset and generate pseudo-target sentences, which are used to continue training the model.
This is a commonly used semi-supervised approach in text generation literature [21, 68].
R-Regression.
Wu et al.
[66] propose a reward regression (R-Regression) approach, where the reward is defined as the BLEU score.
Since their reward is the same as the evaluation metric, such a method may achieve higher BLEU scores without actually improving the generation quality.
By contrast, our reward is induced in a principled way and is agnostic to evaluation metrics.
In our experiment, we replicate the R-regression method, which constitutes a controlled comparison to our approach, as the only difference is the reward function.
Appendix B provides implementation details and hyperparameters of our approach.
3.3 Main Results
Results of Dialogue Generation.
Table 1a shows the results of the dialogue generation task.
We notice that our fine-tuned T5-Base model [42] has already outperformed dedicated methods,
AdaLabel [60] and DialogBERT [16].
This is consistent with the findings of [62, 61] in that the alleged “state-of-the-art” dialogue systems do not outperform standard pretrained language models on deduplicated datasets, highlighting the importance of working with the correct setting.
We then apply semi-supervised learning (Self-Training, R-Regression, and our approach) with the non-parallel OpenSubtitles dataset.
We achieve higher performance than T5-Base trained only on parallel DailyDialog.
Interestingly, the fully supervised model—trained on both parallel DailyDialog and parallel OpenSubtitles—does not achieve high performance, even lower than the one trained with
DailyDialog only.
It is noticed that the OpenSubtitles dataset is noisy [8], which likely causes the performance degradation.
This signifies the need of semi-supervised learning.
Among semi-supervised approaches, RL-based methods (R-Regression and ours) are generally better than Self-Training.
This is within our expectation because Self-Training learns from its own generation and may be overconfident, whereas RL approaches are able to explore different parts of the data space, being a more effective way of semi-supervised learning.
Moreover, our approach outperforms RL with R-Regression, where the reward is the only difference.
The controlled experiment confirms that the reward induced from models trained with teacher forcing is effective for RL training.
It is also worth noting that R-Regression uses the evaluation metric as the reward, and thus may deliberately improve the metric rather than text quality.
By contrast, our reward is induced in a principled manner and is agnostic to evaluation metrics, and our approach still achieves higher performance even with such a disadvantage.
In general, our approach achieves the best performance in both metrics.
In particular, it significantly improves DailyDialog-trained T5-Base by +2.06 (+23.0%) in BLEU2 and +0.61 (+16.5%) in BLEU4.
It also outperforms the second-best method, R-Regression, by 0.68 (+6.6%) in BLEU2 and 0.
(+2.9%) in BLEU4, verifying the effectiveness of our approach.
Results of Paraphrase Generation.
The results of paraphrase generation are shown in Table 1b.
As seen, directly copying the input already achieves a high BLEU score against the reference.
iBLEU addresses this by penalizing the Self-BLEU score (against input) and is considered the main metric.
We consider another semi-supervised baseline LTSL [11].
It performs retrieval-based paraphrase expansion and meta optimization, thus being task specific.
We see that LTSL has an extremely high
Self-BLEU, suggesting the generated paraphrase largely resembles the input.
It achieves a lower iBLEU score than other semi-supervised approaches.
We also see that RL approaches generally achieve lower Self-BLEU than Self-Training.
This is because Self-Training learns from its own predictions, which overlap the input more than groundtruth paraphrases do (Self-BLEU of groundtruth: 29.87); as a result, Self-BLEU increases to 48.02 from
[Table]
44.77 of T5-Base.
By contrast, RL learns by exploring different possible paraphrases and is able to retain low Self-BLEU.
Overall, our approach achieves the highest BLEU and a reasonably low Self-BLEU, yielding the best iBLEU among all competing methods.
The results are consistent with Table 1a, showing the generality of our approach
3.4 Analyses
Step-Wise Reward.
In Figure 1, we show the distributions of different reward functions.
As seen, the BLEU score is mostly concentrated at 0, providing little information for training.
RRegression consequently suffers from a similar problem, as it is trained by the groundtruth BLEU scores.
The distribution of our induced reward, on the other hand, has the lowest peak and is the most wide-spreading one.
We conduct another analysis to show the importance of stepwise rewards for RL training.
We compare our approach with a sparse reward function that defers all rewards to the end of a sentence.
In other words, the last step’s reward is the sum of our step-wise rewards, whereas all previous steps have a reward of 0.
This constitutes a rigorous analysis, as the total reward and thus the training objective are the same in both cases.
Results in Table 2 show that our step-wise reward outperforms the sparse reward in all cases.
This suggests our approach serves as a meaningful credit assignment of the total reward, which is beneficial for RL training.
The Effect of the Synchronizing Period.
We analyze the effect of the synchronizing period k introduced in Section 2.3.
In Figure 2, we see that the training is unstable if k = 1 (on-policy), in which case the model generates uninformative and meaningless sentences (illustrated in Appendix C).
When k = 1000 , the performance increases quickly at the beginning, but it starts to decrease with further training.
We hypothesize that this is due to the lack of exploration (Section 2.3).
When k is infinitely large (the behavior policy is fixed), the performance grows slowly and stops improving after a certain number of steps.
Based on this analysis, we choose k = 5000 to balance exploitation and exploration.
Although the experiment is conducted only on DailyDialog due to the limit of time and resources, we directly apply the setting to other experiments, showing the robustness of our approach.
Data Efficiency.
In Figure 3, we analyze data efficiency by sampling different numbers of data points from the non-parallel corpus.
As shown, our method consistently outperforms self-training, even with only 0.1% (the leftmost points) of the training set.
Additionally, the performance of our method quickly increases with more data, whereas self-training grows slowly.
This is expected because RL training explores different parts of the sentence space and learns from their rewards, whereas self-training only learns from the single generated sentence by the model itself given an input.
We also investigate how performance changes according to the size of the parallel dataset, which reflects the quality of the learned policy.
Results are shown in Figure 4, Appendix D.
Related Work
Semi-Supervised Learning for Text Generation.
In text generation, popular ways to utilize both parallel and non-parallel data include self-training [21, 68] and backtranslation [49].
Both methods first train a model on the parallel data and then generate pseudo-parallel pairs for the non-parallel sentences.
The difference is that self-training generates pseudoparallel pairs from source to target, whereas back-translation generates from target to source.
We mainly consider selftraining as a baseline because it does not require an additional model in the reversed direction, making the comparisons fairer.
Our implementation of self-training is also similar to sequencelevel knowledge distillation [23, 15, 20], except that the latter augments the parallel data instead of the non-parallel ones.
In
Figure 3, we show that self-training cannot efficiently utilize the data because of the lack of exploration.
Additionally, the exposure bias issue remains because they are trained with the teacher-forcing objective.
Text Generation beyond Teacher Forcing.
Teacher forcing is known to have the exposure bias issue.
A line of work uses the generative adversarial network (GAN) [14] to alleviate the issue.
For example, Yu et al.
[67] and Guo et al.
[18] propose to use GAN-style training to generate text similar to the training set in an on-the-fly manner.
This practice reduces the discrepancy between training and inference because the inputs of the model are its own generation rather than the groudtruth sentences.
Shi et al.
[51] further formulate the adversarial training using the IRL interpretation.
These GAN-style methods are different from ours in two main ways.
First, GAN-style training requires parallel corpora and thus cannot be directly applied to semi-supervised learning on non-parallel datasets.
Second, GAN-style training involves the optimization of an adversarial objective, making the training unstable, e.g., suffering from mode collapse [14].
Another paradigm to alleviate the exposure bias is RL.
For instance, Sokolov et al.
[53] and Kreutzer et al.
[26] leverage the bandit-structured prediction framework for text generation with BLEU as the heuristically defined reward.
Bahdanau et al.
[1] and Shen et al.
[50] utilize different variants of policy gradient for RL training.
However, these methods are task-specific and suffer from the sparse reward as mentioned in Section 3.4.
More importantly, these approaches require parallel data to calculate the reward and cannot utilize non-parallel data either.
To address this, Wu et al.
[66] propose to learn a reward regression model on the parallel dataset and perform RL on the non-parallel data with the learned reward.
As mentioned, such a method is still task-specific because it requires the human heuristics of the task to define the proper reward function.
Additionally, it also suffers from the reward-sparsity problem, as seen in Figure 1 and Table 2.
Search is also a popular way to replace teacher forcing.
The Learning to Search (L2S) framework [5, 9] enables the model to search for a better score during learning and is widely applied to text generation.
For example, Wiseman and Rush [64] propose to optimize the beam search results through training.
Li et al.
[29] develop an unsupervised learning approach to text generation based on local search.
In addition to the L2S framework, Leblond et al.
[27] leverage the Monte Carlo tree search [25, 52] to select better tokens in a step from the sampled generation.
These methods are different from ours since they need either heuristically defined scoring functions or parallel data, limiting their methods to certain tasks or to the supervised paradigm.
However, given the success of these methods, we consider the search-based approach an interesting future extension of our work.
Imitation Learning.
Our work is also related to imitation learning methods in general.
Typically, these methods aim to obtain a good policy given a dataset containing state–action pairs.
The easiest approach is behavior cloning [39], which greedily imitates the demonstration.
Similar to the exposure bias, behavior cloning also faces the problem of compounding errors [46].
SMILe [46] and DAgger [47] mitigate the problem by querying an expert.
In text generation, Du and Ji [12] empirically verify that imitation learning methods are helpful.
Recently, Pang and He [37] frame the text generation task as an offline reinforcement learning problem, which learns from a dataset containing tuples of state, action, and reward.
Compared with our method, these approaches rely on parallel sentence pairs and cannot effectively make use of non-parallel datasets.
Conclusion
Summary.
In this paper, we show that a reward function can be derived from a model trained with teacher forcing.
The derivation does not rely on human heuristics for certain tasks.
Additionally, the derived reward function assigns step-wise scores and makes the RL training easier.
Our approach leads to a training algorithm in a semi-supervised manner.
We conduct experiments on the dialogue and paraphrase generation tasks.
The empirical results show that the performance of our approach is better compared with the baselines: self-training and reward regression.
We further analyze our reward function and show the benefits of our approach.
Limitation and Future Work.
First, the scale of the experiments in this paper is restricted by computational resources.
It is interesting to see if our approach could obtain better performance with large models [3, 41] and larger datasets.
We also notice that Assumption 1 has a deep connection with entropy-regularized RL [17, 19, 45].
Our approach can be easily extended to such cases in the future.
Another interesting direction would be using the reward as an interface between humans and the model to control the generation.
Specifically, the current seq2seq models treat data as the ground truth.
But the data may be contaminated with undesired or harmful information.
We hope that our approach provides a way for humans to apply additional rules to the reward function to avoid the model generating harmful information.
Acknowledgments
We thank all reviewers for their valuable comments.
We also thank Guoqing Luo for early discussions.
The research is supported in part by the Natural Sciences and Engineering Research Council of
Canada (NSERC) under grant No.
RGPIN2020-04465, the Amii Fellow Program, the Canada CIFAR
AI Chair Program, a UAHJIC project, a donation from DeepMind, and the Digital Research Alliance of Canada (alliancecan.ca)
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Sections 2–4.
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 2.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section 2.2.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We included our code repository as a URL.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 3.
(b) Did you mention the license of the assets?
[Yes] See Appendix B and our code repository.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We included our code repository as a URL.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] It is discussed in the papers of the datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] It is discussed in the papers of the datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Proof of Theorem
Theorem 2.
Let r ∗ be an underlying true reward function and q ∗ be the corresponding optimal value function.
Given an approximate value function q , we denote by r the reward function derived from Eqn.
Then, we must have ∥ r − r ∗ ∥ ∞ bounded by O ( ∥ q − q ∗ ∥ ∞ ) .
Here, ∥ · ∥ ∞ takes the maximum absolute value over all s ∈ S and a ∈ A .
Proof.
For any s ∈ S and a ∈ A [Equation] we have
[Equation]
Here, Eqn.
(9) is from the Bellman equation; Eqn.
(10) follows the triangle inequality; and Eqn.
(11) generalizes certain s and a to all possible s ′ ∈ S , a ′ ∈ A .
Eqn.
(12) discusses two possible [Table] whether max a ′ q ( s ′ , a ′ ) ≥ max a ′ q ∗ ( s ′ , a ′ ) or not.
Eqn.
(13) is because − max a ′ q ( s ′ , a ′ ) ≤ − q ( s ′ , a ′′ ) for any a ′′ ∈ A .
Eqn.
(14) merges all the maximum operation, and Eqn.
(15) is the definition of the infinity norm.
Since the last equation does not depend on s and a , we conclude ∥ r − r ∗ ∥ ∞ is bounded by O ( ∥ q − q ∗ ∥ ∞ ) .
B Experiments Details
For all experiments, we initialize the model with T5-Base [42] provided by HuggingFace [65].
We use the label smoothing [56] with a coefficient of 0 . 1 .
We use the Adam [24] optimizer with ( β 1 , β 2 ) =
Each batch contains around 32K tokens.
For all conventional seq2seq training, the learning rate is scheduled according to the original Transformer [58] with the warm-up steps set as 4000 .
For all RL training, we drop the warm-up phase and set the maximum learning rate to 1 e − 5 .
We set the synchronizing period k to 5000 .
The reward of our method is scaled down by 100 times.
We apply the reward clipping trick [35] to bound the reward within [ − 1 , 1] to stabilize the training.
For inference, we follow previous work and use greedy decoding in the dialogue generation task and use beam search with a beam size of 5 in the paraphrase generation task.
All the experiments are done on either 4 × NVIDIA A100 or 4 × NVIDIA V100.
C Case Study
We demonstrate several cases from the generation of different models.
These cases come from the
DailyDialog validation set.
Examples of Generated Dialogue Responses.
In the first case of Table 3, we show a phenomenon that previous methods tend to generate short and meaningless responses.
On the other hand, our method usually generates more informative sentences and makes the conversation more natural and human-like.
We also find that previous methods tend to generate sentences with inconsistent or even conflicting semantics.
In the second case in Table 3, for example, both Self-Training and R-Regression reply “I got engaged” but the next sentences are illogical.
This implies that previous methods may generate low-quality sentences even if they have seemingly decent BLEU scores.
By contrast, our model generates a more proper response.
On-Policy Degeneration.
In Section 2.3, we mention that if k = 1 (on-policy), the generation will become deterministic and uninformative.
We show such cases in Table 4.
The responses are generated by the first save (1000 updates) of the model in the experiment.
For both cases, the model replies “I’ll have to think about it” at the end of the sentences.
In fact, most of the generated responses end with this phrase, which is redundant and meaningless.
This phenomenon is likely to be a result of over-deterministic and insufficient exploration of the on-policy [Table] update.
If the behavior policy becomes more deterministic of a certain phrase, it will have a smaller chance to explore other hypotheses.
Hence, it will enhance the preferred responses and become even more deterministic.
On the contrary, our periodically synchronized behavior policy keeps to be exploratory and does not have the degeneration problem as shown in Table 1 and Figure 2.
D Additional Results
We analyze the effect of the sizes of parallel data in Figure 4.
Our approach consistently outperforms competing methods in all settings.
The results show that a high-quality f ω indeed leads to better performance, but our model is still robust when f ω is trained with limited data.
Notably, our method drops by 6.8% when having 10% of the parallel data, whereas R-Regression drops by 10.6%.
This show that our reward induction approach utilizes the parallel data more effectively.