I S R EINFORCEMENT L EARNING (N OT ) FOR N ATURAL L ANGUAGE P ROCESSING : B ENCHMARKS , B ASELINES , AND B UILDING B LOCKS FOR N ATURAL L ANGUAGE P OLICY O PTIMIZATION
A BSTRACT
We tackle the problem of aligning pre-trained large language models (LMs) with human preferences.
If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework.
However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of opensource libraries and benchmarks customized for LM alignment.
Thus, a question rises in the research community: is RL a practical paradigm for NLP?
To help answer this, we first introduce an open-source modular library, RL4LMs 1 ,for optimizing language generators with RL.
The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the
HuggingFace library (Wolf et al., 2020) with an arbitrary reward function.
Next, we present the GRUE (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference.
GRUE is the first leaderboard-style evaluation of RL algorithms for
NLP tasks.
Finally, we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language Policy Optimization) that learns to effectively reduce the combinatorial action space in language generation.
We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al., 2017)), based on both automatic and human evaluations.
I NTRODUCTION
The ultimate aim of language technology is to interact with humans.
However, most language models are trained without direct signals of human preference, with supervised target strings serving as (a sometimes crude) proxy.
One option to incorporate user feedback is via human-in-the-loop, i.e., a user would be expected to provide feedback for each sample online as the model trains, but this degree of dense supervision is often prohibitive and inefficient.
Automated metrics offer a promising compromise: models of human preference like pairwise learned preference models (Ouyang et al., 2022), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) have significantly improved correlation with human judgment compared to earlier metrics (BLEU, METEOR, etc.), and are cheap to evaluate.
But — these functions are usually not per-token differentiable: like humans, metrics
[Figure]
can only offer quality estimates for full generations.
Reinforcement Learning (RL) offers a natural path forward for optimizing non-differentiable, scalar objectives for LM-based generation when it is cast as a sequential decision-making problem.
However, Goodhart’s Law 3 looms: particularly in the case of imperfect metrics that use neural networks, it is easy to find nonsense samples that achieve high-quality estimates.
Recent works have shown promising results in aligning LMs to human preferences via RL by constraining preference-based rewards to incorporate notions of fluency (Wu et al., 2021a; Ouyang et al., 2022) but progress in this line of work is heavily hindered by a lack of open-source benchmarks and algorithmic implementations—resulting in perception that RL is a challenging paradigm for NLP (Choshen et al., 2020; Kreutzer et al., 2021).
To facilitate research in building RL algorithms to better align LMs, we release a library, a benchmark, and an algorithm.
First, we release the RL4LMs library , which enables generative HuggingFace models (e.g., GPT-2 or T5) to be trained using a variety of existing RL methods like PPO/A2C/etc.
Next, we apply models trained using RL4LMs to the new GRUE (General Reinforced-language
Understanding Evaluation) benchmark: GRUE is a collection of 7 contemporary NLP tasks (see
Table1 for details); in contrast to other benchmarks, instead of supervised training, we pair each task with reward function(s).
GRUE challenges models to optimize these reward functions while remaining fluent language generators.
We train language models via RL—both with and without task specific supervised pre-training—to optimize rewards.
Finally, beyond existing RL methods, we introduce a novel on-policy RL algorithm called NLPO (Natural Language Policy Optimization) , that dynamically learns task-specific constraints over the distribution of language at a token level.
Experiments on GRUE and human evaluations show that NLPO better balances learning preference rewards while maintaining language fluency compared to alternatives, including PPO (Figure 1).
We find that using RL to learn from scalar reward feedback can be more: (1) data efficient than using additional expert demonstrations via supervised learning (though a combination of both is best)—a learned reward function enables greater performance when used as a signal for an RL method than a supervised method trained with 5 times more data, and (2) parameter efficient—enabling amillion parameter model trained with a combination of supervision and NLPO to outperform abillion supervised model.
We hope that the benchmarks, baselines, and building blocks we release serve to drive forward research in aligning LMs to human preferences.
R ELATED W ORK
Imitation learning for NLP.
Algorithms such as Schedule Sampling (SS) (Bengio et al., 2015),
Parallel SS (Duckworth et al., 2019), SS for Transformers (Mihaylova & Martins, 2019), Diffential SS (Goyal et al., 2017), LOLS (Lampouras & Vlachos, 2016; Chang et al., 2015), TextGAIL (Wu et al., 2021b), and SEARNN (Leblond et al., 2017), have been inspired by DAGGER (Ross et al., 2011) and SEARN (Daumé et al., 2009).
However, these algorithms are known to suffer from exposure bias in generation (Chiang & Chen, 2021; Arora et al., 2022) and the cliff MDP problem (Huszár, 2015;
Agarwal et al., 2019; Swamy et al., 2021).
RL for Large Action Spaces.
MIXER (Ranzato et al., 2016) combined ideas from schedule sampling and REINFORCE (Williams, 1992).
Bahdanau et al.
(2016) proposed an actor-critic algorithm to address the variance/large action space problems when using REINFORCE for language generation; follow-up works such as KG-A2C (Ammanabrolu & Hausknecht, 2020), TrufLL (Martin et al., 2022),
AE-DQN (Zahavy et al., 2018), and GALAD (Ammanabrolu et al., 2022) addressed similar issues by attempting to eliminate and reduce the action space during exploration.
RL for NLP.
RL, often in the form of bandit learning, has been used to improve models in machine translation (Wu et al., 2016; Nguyen et al., 2017; Kiegeland & Kreutzer, 2021), summarization (Stiennon et al., 2020; Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017; Jaques et al., 2020), image captioning (Rennie et al., 2017), question generation (Pang & He, 2021), text-games (Narasimhan et al., 2015; Hausknecht et al., 2020), and more (Ranzato et al., 2016; Snell et al., 2022).
Lu et al.
(2022) adapt reward-conditioned transformers (Chen et al., 2021) for several language generation tasks.
RL has been the focus of efforts to align LMs with human preferences (Stiennon et al., 2020;
Wu et al., 2021a; Nakano et al., 2021; Ziegler et al., 2019), e.g., Ouyang et al.
(2022) fine-tuned large language model with PPO Schulman et al.
(2017) to align with models of human preference, but their non-public dataset doesn’t enable comparison.
Though RL has been successful in some of the use cases described above, it has simultaneously been critiqued for being significantly less stable than supervised LM training (Choshen et al., 2020).
As a result, there is relatively little consensus if RL is a worthwhile consideration for training LMs compared to, say, collecting additional supervised data.
RL4LM S : A L IBRARY FOR T RAINING LM S WITH RL
We introduce RL4LMs, an open-source library with building blocks for fine-tuning and evaluating
RL algorithms on LM-based generation.
The library is built on HuggingFace (Wolf et al., 2020) and stable-baselines-3 (Raffin et al., 2021), combining important components from their interfaces.
RL4LMs can be used to train any decoder only or encoder-decoder transformer models from HuggingFace with any on-policy RL algorithm from stable-baselines-3.
Furthermore, we provide reliable implementations of popular on-policy RL algorithms that are tailored for LM fine-tuning such as
PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015a), A2C (Mnih et al., 2016), and our own
NLPO (§4).
The library is modular, which enables users to plug-in customized environments, reward functions, metrics, and algorithms.
In the initial release, we provide support for 6 different NLP tasks, 16 evaluation metrics and rewards, and 4 RL algorithms.
3. E NVIRONMENTS : G ENERATION AS A T OKEN - LEVEL MDP
Each environment is an NLP task: we are given a supervised dataset D = { ( x i , y i ) }
Ni =1 of N examples, where x ∈ X is an language input and y ∈ Y is the target string.
Generation can be viewed as a Markov Decision Process (MDP) ⟨S , A , R , P, γ, T ⟩ using a finite vocabulary V .
Each episode in the MDP begins by sampling a datapoint ( x , y ) from our dataset and ends when the current time step t exceeds the horizon T or an end of sentence (EOS) token is generated.
The input x = ( x 0 , · · · , x m ) is a task-specific prompt that is used as our initial state s
= ( x 0 , · · · , x m ) , where s 0 ∈ S and S is the state space with x m ∈ V .
An action in the environment a t ∈ A consists of a token from our vocabulary V .
The transition function P : S ×
A → ∆( S ) deterministically appends an action a t to the end of the state s t − 1 = ( x 0 , · · · , x m , a 0 , · · · , a t − 1 ) .
This continues until the end of the horizon t ≤ T and we obtain a state s T = ( x 0 , · · · , x m , a 0 , · · · , a T ) .
At the end of an episode a reward R : S × A × Y
→ R 1 that depends on the ( s T , y )
(e.g., an automated metric like PARENT
Dhingra et al.
(2019)) is emitted.
RL4LMs provides an OpenAI gym (Brockman et al., 2016) style
API for an RL environment that simulates this LM-Based MDP formulation.
This abstraction allows for new tasks to be added quickly with compatibility across all implemented algorithms.
3. R EWARD F UNCTIONS AND E VALUATION M ETRICS
Because RL4LMs provides a generic interface for per-token or per-sequence generation rewards, it is possible to quickly apply a wide array of RL algorithms to a similarly diverse range of textual metrics-as-rewards.
Specifically, we provide interfaces to 1) n-gram overlap metrics metrics such as
ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), SacreBLEU (Post, 2018), METEOR (Banerjee & Lavie, 2005); (2) model-based semantic metrics such as BertScore (Zhang et al., 2019) and
BLEURT (Sellam et al., 2020) which generally provide higher correlation with human judgment; 3) task-specific metrics such as CIDER (Vedantam et al., 2015), SPICE (Anderson et al., 2016) (for captioning/commonsense generation), PARENT (Dhingra et al., 2019) (for data-to-text) and
SummaCZS (Laban et al., 2022) (for factuality of summarization); 4) diversity/fluency/naturalness metrics such as perplexity,
Mean Segmented Type Token Ratio (MSSTR) (Johnson, 1944), Shannon entropy over unigrams and bigrams (Shannon, 1948), the ratio of distinct n-grams over the total number of n-grams (Distinct-1, Distinct-2) and count of n-grams that appear only once in the entire generated text (Li et al., 2015); 5) task-specific, model-based human preference metrics such as classifiers trained on human preference data collected in the methodology of Ouyang et al.
3. O N - POLICY A CTOR - CRITIC A LGORITHMS
RL4LMs supports fine-tuning and training LMs from scratch via on-policy actor-critic algorithms on language environments.
Formally, this class of algorithms allows us to train a parameterized control policy defined as π θ : S → ∆( A ) , a function that attempts to select an action in a given state so as to maximize long term discounted rewards over a trajectory E π [ (cid:80)
Tt =
γ t R ( s t , a t )] .
Our benchmark experiments focus on fine-tuning a pre-trained LM denoted as π 0 from which we initial our agent’s policy π θ = π 0 .
Similarly, the value network V ϕ used to estimate the value function is also initialized from π 0 except for the final layer which is randomly initialized to output a single scalar value.
As with other deep RL actor-critic algorithms, we define our value and Q-value functions as
V πt =
E a t ∼ π
[ (cid:80) Tτ =
t γR ( s τ , a τ , y )] , Q πt ( s t , a t ) =
R ( s t , a t , y ) + γ E s t +1 ∼ P
[ V πt +
( s t +1 )] leading to a definition of our advantage function as A πt ( s , a ) =
Q πt ( s , a ) − V πt .
To increase training stability, advantage is appoximated using Generalized Advantage Estimation (Schulman et al., 2015b).
Given an input-output pair ( x , y ) and generation predictions from our agent; because the environment rewards are sequence-level and sparse, following Wu et al.
(2021a)
we regularize the reward function using a token-level KL penalty for all on-policy algorithms, to prevent the model from deviating too far from the initialized LM π 0 .
Formally, the regularized reward function is:
[Equation]
(1) where ˆ R is the regularized KL reward, y is gold-truth predictions, KL ( π θ ( a t
| s t )
||
π 0 ( a t | s t ))
= (log π 0 ( a t
| s t ) − log π θ ( a t | s t )) and the KL coefficient β is dynamically adapted (Ziegler et al., 2019).
Further details on actor-critic methods can be found in Appendix A.
NLPO: N ATURAL L ANGUAGE P OLICY O PTIMIZATION
Language generation action spaces are orders of magnitude larger than what most discrete action space RL algorithms are designed for (Ranzato et al., 2016; Ammanabrolu, 2021), e.g., GPT-2/3 and
T5 have a vocabulary size of 50K and 32K respectively.
We hypothesize that the size of the action space is a core cause of instability when training LMs with existing RL methods.
To address this issue, we introduce NLPO (Natural Language Policy Optimization), which is inspired by work on action elimination/invalid-action masking (Zahavy et al., 2018; Huang & Ontañón, 2020; Ammanabrolu & Hausknecht, 2020).
NLPO, a parameterized-masked extension of PPO, learns to mask out less relevant tokens in-context as it trains.
NLPO accomplishes this via top-
p sampling, which restricts tokens to the smallest possible set whose cumulative probability is greater than the probability parameter p
(Holtzman et al., 2018).
Specifically, NLPO maintains a masking policy π ψ : the masking policy is a copy of the current policy ( π θ ), but is updated only every µ steps.
A parameterized-invalid-mask is created from π ψ by first selecting the top- p tokens from the vocabulary, 4 and then applying an invalid-mask to the remaining tokens—i.e. setting their probabilities to zero when sampling actions from π θ during training; this periodic updating policy π ψ is inspired by off-policy Q-learning algorithms (Andrychowicz et al., 2017), providing the policy π θ with an additional constraint that balances between the benefits of containing more task relevant information than the KL penalty derived from π 0 and the risk of reward hacking.
We provide pseudocode in Algorithm 1 (green portions highlight the differences with PPO).
D repeat
Sample mini-batch D m
= { (
Collect trajectories T m
= { τ i } by running policy π ψ n in for batch D m in env.
▷ Eq.
Compute Preference and KL penalty rewards ˆ R t ▷ Eq.
Compute the advantage estimate ˆ A t ▷ Sec. 3.
Update the policy by maximizing the PPO-Clip objective: π θ m +1 = argmax θ |D m | T (cid:88) τ ∈D m
T (cid:88) τ
= min (cid:16) r t ( θ )
A π θm , clip ( r t ( θ ) , 1 − ϵ, 1 + ϵ )
A π θm ) (cid:17) where r t ( θ ) =
π θ ( a t | s t )
π θm ( a t | s t ) .
Update the value function:
V ϕ m +1 = argmin ϕ
|D m | T (cid:88) τ ∈D m
T (cid:88) t = (cid:16) V ϕ
[Equation]
s t
−
R t (cid:17)
[Equation]
n +
[Equation]
m +
GRUE (G ENERAL R EINFORCED - LANGUAGE U NDERSTANDING E VAL )
GRUE is a collection of 7 generative NLP tasks.
To combat reward hacking for any single metric, each task is evaluated at test time according to a task-specific mix of metrics, detailed in Table 1.
The metrics span two categories.
Task preference metrics capture how well the models produce generations that satisfy the desiderata of the specific generation task, e.g., for Commongen, if the generations contain all the required words, or for IMDB, how positive the generated completions are.
Naturalness metrics capture fluency, readability, etc. and provide perspective on factors beyond semantics.
At training time, there are no special restrictions: models are free to use the supervised data, compute metrics on intermediate generations, etc.
Train/val/test splits follow the original works.
All results are averaged over multiple seeds, with exact counts being found in Appendix B.
Experimental Setup.
We use RL4LMs to test a large range of algorithms on the GRUE benchmark.
Specifically: We compare 3 algorithms for direct fine-tuning — Supervised, PPO, 5 and NLPO.
In 4 π ψ could be trained with alternate sampling techniques like top- k or beam search (or even hard-coded via rules by domain experts), though we find top- p sampling to be most effective in practice.
We consider PPO representative of the present state-of-the-art — in particular, we do not consider the popular REINFORCE (Willianms, 1988; Williams, 1992), as recent works have shown PPO to be strictly superior to REINFORCE in multiple domains (Schulman et al., 2017)
[Table]
addition, we consider a hybrid approach of supervised learning and our RL methods by applying
PPO and NLPO on checkpoints that have been fine-tuned in a supervised fashion—we call these
Supervised+PPO, Supervised+NLPO.
As an additional baseline, we additionally run zero-shot evaluations where we design prompts which aim to elicit task-specific generations, but with no training data or parameter updates.
For each task, to isolate the effect of training method, we select a single pre-trained LM backbone.
For IMDB text continuation we use GPT-2 (117m parameters), and for the rest of the tasks we use
T5-base (220m parameters).
For our RL models (PPO, NLPO, Supervised+PPO, Supervised+NLPO), for a thorough investigation of how reward-hacking might interplay with GRUE, we run a separate set of experiments optimizing multiple task rewards for each task independently, e.g., for Commongen which has 6 task rewards (CIDER, ROUGE-2, ROUGE-L, BLEU-3, BLEU-4, METEOR)
we rundifferent experiments optimizing each metric independently and report all possible metrics seen in
Table 1 regardless of which individual metric was being optimized for.
Human Participant Study.
We gather human judgments for five of the tasks in GRUE.
In doing so, our goals are 1) to validate that the automated metrics we selected for GRUE correlate with human judgments with respect to relative ranking between models; and 2) to provide additional empirical comparisons regarding NLPO vs. PPO, ablations to study the effects of the KL naturalness penalty, etc.
We specifically consider IMDB, Commongen, ToTTo, DailyDialog, and CNN Daily Mail.
For each individual sample in a task, we ask 3 unique human raters to provide Likert judgments of 1) quality, i.e., for the specific task, how correct/appropriate is the generation, given the context, and 2) fluency, i.e., how well-written is the generation.
We used Amazon Mechanical Turk, and paid crowdworkers a minimum of $15/hr.
More details, including qualification information, interface screenshots, instructions, etc. are given in the corresponding Appendicies.
5. R ESULTS ON GRUE: W HICH A LGORITHM S HOULD BE U SED TO L EARN P REFERENCES ?
Figures 2(a)
2(b) present the results on GRUE, split into task metrics and naturalness metrics, and
Tables 2, 3 highlight key results via ablation studies.
Full results are available in Appendix B. For text continuation and summarization, with non-trivial zero-shot performance, RL tends to perform better than supervised training, but for tasks like Commongen and ToTTo, which have very low zero-shot performance, supervised training performs best—with both approaches outperforming zero-shot.
However, using RL+Supervised learning in conjunction works best; NLPO+supervised and
PPO+supervised usually always outperforms NLPO/PPO (or supervised in isolation) across both task metrics and naturalness metrics.
Supervised warm-starting is particularly effective for Commongen and ToTTo, which our results suggest are more prone to reward hacking.
The one exception to this trend is DailyDialog where the RL models outperform warm-started Supervised+RL models likely due to the low performance of the Supervised models.
We note that Supervised+NLPO using a
[Figure]
DailyDialog 0.32 0.35 0.39 0.43 0.
[Figure] [Figure]
Commongen
IMDB
NarrativeQA
WMT
[Figure]
CNN
DailyDialog 0.19 0.22 0.25 0.
Supervised PPO Supervised+PPO NLPO
Supervised+NLPO 0.95 0.
(b) Automated Naturalness Metrics
[Figure]
CNN
[Figure]
CNN
[Figure] [Figure]
[Figure] [Figure]
[Figure]
human participant studies for the 5 tasks suitable for human studies.
Test results are averaged over all the respective metrics seen in Table 1.
[Table]
[Figure]
✗ ✓ ✗ ✓ ✗ ✓
[Table]
Easilyrewardhackable?
✓ ✓ ✗ ✗ ✗ ✗ ✗ RL > Sup(auto)?
✓ ✗ ✗ ✗ ✗ ✗ ✓ RL > Sup(human)?
✓ ✗ ✗ ✗ - - ✓ Sup+RL > Sup(auto)?
✓ ✓ ✓ ✓ ✓ ✓ ✗ Sup
[Table]
[Table] [Table]
✓ ✗ ✓ ✓ - -
[Table]
✓ ✓ ✓ ✓ ✓ ✓
[Table]
✓ ✓ ✓ ✓ - - ✓
This table summarizes the results found in the ablations and
Fig. 2 and provides an overview of the questions we ask in Section 5: which tasks require warm starts or are easily reward hackable; when to use RL over Supervised, when to use both; and when to use NLPO over PPO.
All the of statistical analysis as discussed
[Table]
T5-base (220m parameter) LM currently outperforms all the models on the ToTTo leaderboard, many of which have ≥ 3b parameter supervised models—suggesting that RL is parameter efficient as well.
In these cases, it is critical that the initial policy already contain (some) signal for the task due to it being used as a KL constraint and masking constraint in NLPO.
If the mask contains no initial priors about task specific language, it will be eliminating the wrong actions—a better initial policy leads to better RL performance downstream.
Human agreement with automated metrics.
As human judgments can be noisy, we run additional statistical analysis such as measuring inter-annotator agreement, via Krippendorf’s alpha score, and using a one-way ANOVA followed by a post-hoc Tukey HSD test to measure if differences in means of average scores between pairs of models are significant.
We find that trends in our human evaluations generally match those seen in the automated metrics for both task and naturalness metrics (see Figures 2(c), 2(d) which summarize Appendix Tables 10,15,21,26, 35—Supervised+NLPO >
Supervised ≥ Supervised+PPO > NLPO ≥ PPO > Zero-shot—with the exception of Supervised outperforming Supervised+PPO on 2 out of 5 tasks when automated metrics would indicate that
Supervised+PPO outperforms Supervised on all of the tasks.
We draw two conclusions from this: (1) if the generated text is above a certain threshold of naturalness, the automated metrics usually correlate with human judgements; (2) usually but not always as seen in the relative performance of
Supervised and Supervised+PPO, potentially indicating reward hacking behaviors undetected by automated metrics but caught by human preference feedback.
5. P REFERENCE R EWARD L EARNING , S ELECTION , AND H ACKING
While the GRUE benchmark’s metric for each task is an average over several measures, the RL models we trained optimized only a single metric independently.
Thus, we can empirically investigate which metric for which GRUE produces the best results.
We observe that many possible single metric rewards provide task performance gains over supervised methods (results shown in Fig. 3(a), 2(c) are averaged across these reward functions) with the condition that the text is also coherent and natural.
Which constraints best prevent reward hacking?
The reward function in Equation 1 balances a task-specific reward with a KL constraint — models are penalized from straying too far from a base
LM in their pursuit of high reward (Table 3 and Appendix Table 5) clearly show that if KL constraints are removed entirely, models reward hack).
But which model works best as a base regularizing LM?
When the initial policy (i.e., the raw, pretrained model) has low performance on the task, the KL penalty pushes the policy towards nonsense, e.g. on Commongen and ToTTo the trained policy learns to simply repeat portions of the input (as seen in Tables B.4.5, B.6.4).
This behavior is mitigated if the base regularizing LM is the supervised model—the reward encourages the policy to balance the task-specific reward and a more reasonable regularization term.
Deriving KL penalties from warm-started initial policies is critical for performance on such tasks.
PPO vs. NLPO.
Figure 2 shows that NLPO generally outperforms PPO and supervised, especially when applied after supervised training.
We hypothesize that the primary reason for NLPO’s improved performance and stability is because the masking policy provides an additional constraint for the current policy.
This constraint is not based on the initial untuned policy like the KL penalty but of the policy from µ iterations ago and likely contains more task-relevant information learned during
RL training.
Table 3 (and Appendix Table 8) shows how performance increases up to a point and then decreases as p in top-
p sampling is increased for the masking policy, relaxing the constraint by eliminating less tokens at each step, implying that there is a balance to be found in how much the model should be constrained during RL training.
Human Preference Reward Learning.
To this point, our experiments have largely focused on optimizing evaluation metrics that correlate with human judgments, e.g., METEOR.
Here: we additionally test how well preferences can be learned from direct human feedback.
For this, we focus on Commongen — a GRUE dataset well-suited for displaying differences due to human preferences.
First, we randomly select prompts from the Commongen train dataset and sample a single completion from both the Supervised and Supervised+NLPO models.
We then present the prompt and the two completion candidates to 3 unique crowdworkers and ask them to select which one they prefer with respect to commonsense/fluency for 417 unique pairs (Krippendorf α = . 28 ).
We use this data to train a reward model, T5-11B Raffel et al.
(2020), on the balanced binary classification task of predicting which of the pair was preferred by a majority of 3 annotators, conditioned on the prompt and completion.
The resulting model achieved 69.5 test ROC AUC suggesting it indeed captures average human preferences.
Additional details on this process are found in Appendix B.4.4.
We train
Supervised+RL with a METEOR-only reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model.
Finally, we rerun the same pairwise preference collection procedure—this time sampling from Commongen test—with human participants to compare the generations from a preference optimized RL policy to the previously best Supervised+NLPO policy.
Comparing the METEOR-only to the preference model, the generations produced by the human feedback model are preferred in 682 cases, compared to the METEOR-only model which is preferred in 587 cases ( p < 0 .
the models are equally preferred).
This implies that this pipeline of collecting preferences, training a reward, and further tuning the policy improves alignment to human preferences.
5. D ATA B UDGET : I MPROVE YOUR R EWARD OR G ATHER M ORE D EMONSTRATION ?
Given a fixed data collection budget, is it more efficient to gather feedback to improve a learned reward function or to gather more expert demonstrations?
We use the IMDB text continuation task as a case study.
In the IMDB task, a model is given a partial movie review as a prompt, and is asked to continue it as positively as possible (even if the prompt was negative).
The original dataset consists of movie reviews and sentiment labels of positive, negative, or neutral.
A DistilBERT (Sanh et al., 2019) classifier is trained on these labels and used to provide sentiment scores on how positive a given piece of text is, which serves as the task reward.
The trade-off is between gathering more: 1) sentiment labels (improving the reward); or 2) positive sentiment reviews (improving supervised training).
We train a classifier on varying amounts of training data and evaluate on the held out test dataset— finding as expected that more training data improves test accuracy and so results in a higher quality reward.
We then use each of these rewards of varying quality during RL training, and evaluate using the same metric as GRUE (i.e., a classifier trained with the entire training set).
As seen in Table 3, we find that improving the reward quality improves LM performance as well.
Further, we trained a supervised model with at least as many samples used to train each of these reward classifiers.
We find that a learned reward function enables greater performance when used as a signal for an RL method than a supervised method trained with 5 times more data.
This implies that improving reward models can be more data efficient than collection expert demonstrations for a task—and that’s not accounting for the fact that assigning sentiment labels is likely a simpler task than writing full demonstrations.
Further details on this ablation are found in Appendix Table 7.
5. P RACTICAL
Generation as a token-level MDP, not a bandit environment.
Most recent works that tune LMs using RL do so by calculating a reward for all the tokens in the sentence (Wu et al., 2021a;
Ouyang et al., 2022; Lu et al., 2022).
This setting is equivalent to a bandit feedback environment where the action space is the space of all possible generations for the task (Sutton & Barto, 2018).
This type of environment can be simulated within our RL formulation by setting the discount factor γ = 1 .
Table(and Appendix Table 6) shows that this causes instability in training with respect to naturalness in both PPO and NLPO for IMDB.
Our standard setting is γ = 0 .
95 when calculating discounted rewards-to-go in the token-level MDP formulation, which reduces the magnitude of the reward that is applied to tokens selected at the beginning.
The sentiment scores are approximately the same between both settings but the naturalness of language in the bandit setting is significantly less—indicating that discounting rewards with γ < 1 via a token-level MDP formulation is at least sometimes more effective for language generation.
Dropout and Sampling.
We found two other implementation details to be critical for stability of
RL training.
The first is dropout, which in its standard form was found to cause instability in policy gradient methods in continuous control settings by Hausknecht & Wagener (2022).
We find a similar effect when using dropout when RL training LMs as well, with training loss often diverging for dropout > 0 in training.
The second important detail, particularly affecting the machine translation task, is sampling methods.
We find that using the same sampling methods during exploration and inference is critical to translating training performance to test performance–else the model exhibits high train rewards but low test metrics.
C ONCLUSIONS
We’re hopeful that the GRUE benchmark and the RL4LMs library can push progress in aligning language models to human preferences via RL methods by providing the community with a standard means of comparing methods.
Furthermore, we’re optimistic that, as the stability and consistency of training improves, our methods provide a path towards iterative improvement of language technologies, with deployment, user feedback collection, and re-optimization enabling better user experiences when interacting with generative models.
A CKNOWLEDGEMENTS
We’d like to acknowledge the support of DARPA MCS program through NIWC Pacific (N66001-192-4031), Google Cloud Compute and the ReViz team at the Allen Institute for AI.
KB is supported by NSF under grant No. 2127309 to the Computing Research Association for the CIFellows Project
C ONTENTS 1 Introduction
2 Related Work 3 RL4LMs:
A Library for Training LMs with RL 3.
Environments: Generation as a Token-level MDP . . . . . . . . . . . . . . . . . .
Reward Functions and Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . .
On-policy Actor-critic Algorithms . .
4 NLPO: Natural Language Policy Optimization 5 GRUE (General Reinforced-language
Understanding Eval) 5.
Results on GRUE:
Which Algorithm Should be Used to Learn Preferences? . . . .
Preference Reward Learning, Selection, and Hacking . .
Data Budget: Improve your Reward or Gather More Demonstration? . . . . . . . .
Practical Considerations: Which Implementation Details Matter Most? . . . . . . .
6 Conclusions
Acknowledgements
A On-policy Algorithm Implementation Details
A.1 PPO Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 NLPO Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B Experimental Details
B.1 Crowdworking Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 GRUE Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 IMDB . .
B.3.
Setup . .
B.3.
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .
B.3.
Human Participant Study . .
B.3.
Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.4 CommonGen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.4.
Setup . .
B.4.
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .
B.4.
Human Participant Study . .
B.4.
Human Preference Learning Experiments . .
B.4.
Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.5 CNN Daily Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.5.
Setup . .
B.5.
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .
B.5.
Human Participant Study . .
B.5.
Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.6 ToTTo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.6.
Setup . .
B.6.
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .
B.6.
Human Participant Study . .
B.6.
Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.7 Narrative QA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.7.
Setup . .
B.7.
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .
B.7.
Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B
[Table]
8 Machine Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.8.
Setup . .
B.8.
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .
B.8.
Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B [Table] 9 Daily Dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.9.
Setup . .
B [Table]
Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . .
B [Table]
Human Participant Study . .
B [Table]
Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A O N - POLICY A LGORITHM I MPLEMENTATION D ETAILS A. PPO D ETAILS
Given discussion and equations in Section 3.3, we further note that we follow (Ziegler et al., 2019) and dynamically adapt the KL coefficient β during training where,
[Equation]
where KL target is user-specified KL divergence between initial model h and current policy π and K β is rate of update which we generally set to 0 .
2 in our experiments.
To increase stability during training, we further use Generalized Advantage Estimation (GAE) (Schulman et al., 2015b) and define the advantage estimator ˆ A ( s n , a n ) based on the Temporal Difference residual as:
[Equation]
where λ provides the trade-off between bias and variance.
A. NLPO D ETAILS
NLPO learns to mask irrelevant language by maintaining a masking policy π ψ : the masking policy is a copy of the current policy ( π θ ), but is updated only every µ steps.
Given Z ( π θ ) =
(cid:80) a ∈V π θ 0 ( a | s ) the normalization value of the sum of probabilities of all action
a ∈ A given a particular State s ∈ S , let the parameterized top- p vocabulary V pπ θ ⊂
V be the subset of the vocab, consisting of the top- p highest probability vocabulary tokens with respect to π θ .
Formally, let Z p be the normalization value for the parameterized top- p vocabulary, can be defined as the subset of tokens that maximizes Z p ( π θ )
= (cid:80) a ∈V kπθ π θ ( a | s ) .
Then optimizing a policy according to the parameterized top- p vocabulary can be defined as:
[Equation]
B E XPERIMENTAL D ETAILS B. C ROWDWORKING D ETAILS
Qualification round We ran a qualification round using the IMDB task.
We opened the qualification around to users from { AU, CA, NZ, GB, US } with 5K prior approved HITs and a minimum acceptance rate of 97% on their previous HITs.
We gathered judgments over 600 generations from 3 annotators per generation.
One of the authors of this paper also completed 17 random HITs to serve as a proxy for “ground truth."
After gathering these annotations, we selected workers who: 1) didn’t significantly disagree with other annotators on the same instance more than 20% of the time; 2) who completed at least 5 HITs; 3) who didn’t disagree with the author annotator on the 17 HITs by more than 1 point; and 4) (likely) spent a reasonable amount of time reading the instructions/examples provided.
In the end, 56 annotators were qualified.
Additional per-task details are provided in the per-task sections of the Appendix.
Compensation details As per Amazon Mechanical Turk policy, annotators were compensated on a per-HIT basis.
In addition, we used a timing script to estimate hourly wages to ensure our target of $15/hr was met.
In cases where this minimum hourly rate was not met, we manually assigned bonuses.
B. GRUE E XPERIMENT S ETUP
We benchmark 5 training algorithms on 6 tasks (see Table 1) using either an encoder model (eg.
GPT-2) or encoder-decoder model (eg. T5).
We train policies using PPO, NLPO with variations of whether supervised pre-training is applied before RL fine-tuning and compare against supervised policy.
The choice of LM is based on the type of task.
For IMDB text continuation, we use GPTand T5 for rest of the tasks.
We use two separate LM models as actor and critics networks (i.e. no shared layers) in which the critic network has an additional linear layer mapping last token’s hidden representation to a scalar value.
We use AdamW optimizer Loshchilov & Hutter (2017) with fixed learning rate and no scheduling.
Commongen
IMDB
NarrativeQA
WMT
ToTTo
CNN
DailyDialog 0.32 0.35 0.39 0.43 0.
[Figure]
Metrics
Commongen
IMDB
NarrativeQA
WMT
ToTTo
CNN
DailyDialog 0.19 0.22 0.25 0.
Supervised PPO Supervised+PPO
NLPO Supervised+NLPO 0.95 0.
[Figure]
[Table] [Table]
S ETUP
We consider IMDB dataset for the task of generating text with positive sentiment.
The dataset consists of 25k training, 5k validation and 5k test examples of movie review text with sentiment labels of positive and negative.
The input to the model is a partial movie review text (upto 64 tokens) that needs to be completed (generating 48 tokens) by the model with a positive sentiment while retaining fluency.
For RL methods, we use a sentiment classifier Sanh et al.
(2019) that is trained on pairs of text and labels as a reward model which provides sentiment scores indicating how positive a given piece of text is.
For supervised Seq2Seq baselines, we consider only the examples with positive labels.
We chose GPT-2 as LM for this task as it is more suited for text continuation than encoder-decoder
LMs (eg. T5).
We use top-k sampling with K = 50 as the decoding method and for fair comparison, we keep this setting for all methods.
For PPO and NLPO models, we train for 64 k steps in total and update policy and value networks every 1280 steps with a mini-batch size of 64 and epochs of 5 per update.
We apply adaptive KL controllers with different target KLs of 0 .
05 , 0 . 1 , inf with an initial KL co-efficient of β = 0 .
Table 4 provides an in-depth summary of all hyperparameters and other implementation details.
B.3. R ESULTS AND D ISCUSSION
Target KL ablation Fig 4 shows learning curves for PPO and NLPO in terms of episodic training reward, corpus level sentiment scores and perplexity scores on validation set averaged for 5 random seeds.
It is seen that higher target KL of 0 .
1 is desired to achieve higher rewards but results in drifting
[Figure] [Figure]
[Table]
away from pre-trained LM and loses fluency.
Therefore, a lower target KL (0.02 or 0.05) is required to keep the LM closer to original LM.
This is also seen in Table 5 where we presented a comparative analysis of final performance of all models.
Training data size ablation We vary the amount of data used to train the reward classifier and the supervised baseline model to understand whether it is more efficient to gather data to improve reward model or to gather expert demonstrations for supervised learning.
As observed in Table 7, improving the quality of reward function increases the performance on the overall task better than training with more data for supervised training, indicating that improving reward models is efficient than collect expert demonstrations for supervised training from a data efficiency perspective.
Discount factor ablation To understand the effect of discounted vs undiscounted (bandit) environments, we report sentiment and perplexity scores for different values of discount factor ( 0 . 5 , 0 .
95 and 1 . 0 ) in Table 6 and observe that using a bandit environment (discount factor of 1 . 0 ) results in performance loss in the case of NLPO and reward hacking in the case of PPO, indicating that discounted setting (with 0 .
95 ) is desired.
NLPO params Table.
8 shows ablation on different hyperparameters in NLPO algorithm
[Table] [Table]
[Figure]
[Table] [Table]
B.3. H UMAN P ARTICIPANT S TUDY
Figure 5 shows the IMDB instructions, example, and interface used both for the qualification round, and then later, for the human evaluation experiments.
Tables 9, 10 show averaged results, annotator agreement
[Table]
and the results of statistical significance tests to determine which models output better generations when rated by humans.
[Table]
NLPO without KL
Zero Shot 0.
Supervised
Human 0.
Supervised
Zero Shot 0.
Human
Zero Shot -0.
0.9 Supervised+PPO
Supervised+NLPO 0.
Supervised+PPO
NLPO with KL 0.
0.9 Supervised+PPO
NLPO without KL -0.
Supervised+PPO
PPO without KL -0.
Supervised+PPO
PPO with KL 0.
Supervised+PPO
Zero Shot 0.
Supervised+PPO
Supervised 0.
Supervised+PPO
Human 0.
Supervised+NLPO
NLPO with KL 0.
Supervised+NLPO
NLPO without KL -0.
Supervised+NLPO
PPO without KL -0.
Supervised+NLPO
PPO with KL -0.
Supervised+NLPO
Zero Shot -0.
Supervised+NLPO
Supervised 0.
Supervised+NLPO
Human 0.
B.3. Q UALITATIVE R ESULTS
We show sample generations from each of the algorithms for three randomly picked prompts below.
[Table]
PPO
[Table]
public.
At the
[Table]
, the woman is killed and the wife is found naked and with a red head tattooed on her back.
This is part of a special feature called"On the
Road, Hidden" which was PPO -no- KL : police and other local governments.
Written by G. Scott Fitzgerald, who also stars in this production, "A Very Merry Christmas" is a wonderful family movie with a wonderful soundtrack and wonderful characters.
It brings home the true beauty of small-
[Table]
: mother while the brother goes under.
After all, his sister, who was apparently kidnapped but who turns out to be a man, is no longer in her place.
If a film that was made in the 1950s
NLPO -no- KL : friends of the wife, and also the friend of the wife of the woman that is missing, and eventually he begins to find that she has gone to Germany and he has made a very good friend.
There is a good scene between John and Supervised : amily tree.
The widower, now widowed, returns, the daughter, daughter is not pleased when his wife is mistaken for one of her sons, and the patriarch, then a wealthy man, falls victim to a jealous husband.
The
Sample 2 Prompt : A very credible and unsettling movie portraying the effects of war on the general public, Shame gives us a chilling understanding
oh Ingmar Bergman’s beliefs.
The absence of color and sound (as in a soundtrack) helps to give the film a more realistic feeling.
There is no soft or hard lighting or a variety of camera angles to give the story a charming or dramatic effect.
Everything is very simply portrayed as it would be in reality.
At times the storyline was confusing, but perhaps in order to give a sense of the characters’ frame of mind how they too do now know what is happening or why.
The simplicity of the
: camera work is at once apparent and fascinating in the manner in which it affects the way things are portrayed.
In other films these two have been referred to from time to time in a form and manner more suited to the screen.
The film shows References : film made it somewhat boring, but I could understand how the director is trying not to glamorize the story and points.
He depicted the harsh and unsettling events and effects of war.
In the beginning the characters were introduced and shown as content with the lighter settings relationships is so engrossing and the great and unique story presented by each pair of brothers and sisters creates a truly gripping and enjoyable experience.
To put it simply, I strongly recommend to everyone who plays the role to buy this DVD!!!
: movie helps it to flow along in a believable way.
Rating 4 out of 5 Great Movie!
I highly recommend these movies! I highly recommend these movies!
[Table]
out of 5 Great Movie!
I highly recommend these movies!
[Table]
10 of" novel makes it one of the first books I have read by a British actor.
It is a fun way for someone who likes their books, like me, to sit through a novel.
If you like to read novels, check it out.
: setting makes the final product very enjoyable to watch.
Highly recommended!!
~
Michael
[Table]
, SPSA Film Festival
[Table]
, SPSA, WA (December 17,
[Table] [Table]
Great photography, good acting, Supervised : "Dakota Clan" and his sister, the two become friends with Ted Masters who soon finds the chance to fly back to their home land and go back to save his brother’s life and his family.
The
Sample
[Table]
: Shakespearean language is wild and rambling, saturated in multiple meanings, word play and metaphor.
To be understood it must be wrangled and tamed by an actor with the strength and knowledge to do so.
When an actor fails, the words pour forth in a torrent of incomprehensible words, but when he succeeds, the English language springs to life with an immediacy and vibrancy that takes your breath away.
Al Pacino is one such actor, and here displays an incredible level of clarity and control that, were there any justice,
Zero - Shot : might still be seen in the world today.
This movie, like "Vacation", is an effort at telling the tale of two families from different races.
One is African but never grew up the other.
One of them, a man References : would sweep every award in the offering.
He meets the challenge of presenting
Shylock head on, and delivers an extraordinarily subtle and nuanced performance.
It would be a crime if we never got the opportunity to see what he does with King Lear.<br /><br /
[Table]
PPO
[Table]
instant-classic.
This movie captures the spirit of childhood and has everyone in the world watching it.
It’s a love story
and it’s really heartbreaking.
And at the same time, it’s a classic.
It
[Table]
B. C OMMON G EN B.4. S ETUP
CommonGen (Lin et al., 2020) deals with task of generating coherent sentences describing an input set of concepts (eg. "
a man is throwing a frisbee").
For training RL methods, we consider 3 traditional lexical rewards namely Rouge-1, Rouge-avg (which is an average of Rouge-1,
and
L) and meteor.
Additionally, we also train with task-specific rewards such as CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016) and SPiDer (Liu et al., 2017) which is a just a linear combination of both with equal weights.
We chose T5-base as the base LM since it is well-suited for structure to text tasks.
We additionally note that concept set inputs are prefixed with "generate a sentence with:" to encourage exploration.
During our initial experiments when fine-tuning directly on LM, we observed that policy learns to repeat the prompted concepts in order to maximize rewards resulting in a well-known problem of reward hacking .
To mitigate this, we add a penalty score of − 1 to final task reward if the n-grams of prompt text overlaps with generated text.
In contrast, when initialized with a supervised policy, this problem is not seen and hence penalty score is not applied.
We use beam search as the decoding method during evaluation whereas for rollouts, we use top k sampling to favor exploration over exploitation.
Table 11 provides an in-depth summary of setting of hyperparameter values along with other implementation details.
[Table]
B.4. R ESULTS AND D ISCUSSION
Tables 13, 12 presents our benchmarking results with 6 reward functions along with supervised baseline performances on dev and test sets respectively.
Our main finding is that warm-started initial policies are crucial for learning to generate coherent sentences with common sense.
Without warmstart, policies suffer from reward hacking despite application of repetition penalty and task-specific metrics such as CIDer etc.
Further, we find that RL fine-tuned models obtain very high concept coverage which is also seen in Table B.4.5.
Supervised models often tend to miss few concepts in its generation compared to RL methods
[Figure]
SPider
[Figure]
P ub li
[Table]
h e d a s a c on f e r e n ce p a p e r a t
I C L
R
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure]
Figure 6 shows the commongen instructions, examples, and interface used for the human evaluation experiments.
Different from the other human evaluations, we didn’t provide any prompt because knowing the set of words to be used isn’t required for rating either of the axes.
Tables 14, 15 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans.
B.4. H UMAN P REFERENCE L EARNING E XPERIMENTS
First, we randomly select prompts from the Commongen train dataset and sample a single completion from both the Supervised and Supervised+NLPO models.
Next, we filter to prompts where both models at least attempted to use all input concepts.
This filtration step was conducted because if a model fails to use all concepts, it may generate a more natural/fluent sentence, but, a priori , it shouldn’t be preferred by crowdworkers; instead of training crowdworkers to prefer sentences with all concepts, we perform this filter.
Figure 7 shows the task presented to the crowdworkers.
We then present the prompt and the two completion candidates to 3 unique crowdworkers and ask them to select which one they prefer with respect to commonsense/fluency; We gathered 3 annotations on 417 pairs (Krippendorf α = . 28 ), and split into 60/20/20 train/val/test split.
We then trained a reward model, T5-11B Raffel et al.
(2020), on the balanced binary classification task of predicting which of the pair was preferred by a majority of 3 annotators, conditioned on the prompt and completion.
The resulting model achieved 69.5 test ROC AUC suggesting it indeed captures average human preferences.
The model is then used as a reward function.
We train Supervised+RL with a METEORonly reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model.
We design the reward function based on the preference model as r = meteor + pref/ (1 + | miss | ) where miss is a set of concepts not covered in the generated text, in an attempt to mimic the data collection process that humans are instructed to follow.
This reward function accounts for both the task of using all concepts and also human’s preferences for how a sentence should look within the constraints stipulated by the task.
Finally, we rerun the same pairwise preference collection procedure—this time sampling from Commongen test—with human participants to compare the generations from a preference optimized RL policy to the previously best Supervised+NLPO policy.
Comparing the METEOR-only to the preference model head-to-head, the generations produced by the human feedback model are preferred in 682 cases, compared to the METEOR-only model which is preferred in 587 cases ( p < 0 .
the models are equally preferred).
B.4. Q UALITATIVE A NALYSIS
This section shows sample generations from different algorithms for three randomly picked prompts.
Sample 1 Prompt : generate a sentence with: apron cut hat kitchen sausage Zero - Shot : generate a sentence with: apron cut hat kitchen sausage.
PPO : sausage in the kitchen on an apron.
NLPO : sausage cut hat cut hat cut hat cut apron cut hat Supervised : A man is cutting sausage on an apron in a kitchen.
Supervised + PPO :
A man cutting sausage from
[Table]
[Table]
B. CNN D AILY M AIL B.5. S ETUP
As a representative of the summarization task
[Figure]
we consider CNN/DM dataset consisting of long news articles and their highlights written by news authors.
The dataset consists of 287k training, 13k validation and 11k test examples.
We trained RL methods using 3 different automated metrics, namely Rouge-1, Rouge-avg and Meteor.
We chose T5 as our base LM as it is pre-trained in a unified text-to-text framework and relishes Zero-Shot capabilities.
For decoding, we use multinomial sampling with a temperature of 0 .
7 for all the models
[Figure]
B.5. R ESULTS AND D ISCUSSION
Table 17 presents benchmarking results on test set reporting a wide range of metrics: lexical, semantic, factual correctness and diversity metrics.
As baselines, we report lead-3 which selects first three sentences as the summary, Zero-Shot and a supervised model.
PPO and NLPO models are on par with supervised performance on several metrics including Rouge-2, Rouge-L, and Bleu.
On fine-tuning on top of supervised model, performance improves consistently on all metrics indicating that RL fine-tuning is beneficial.
Another interesting finding is that, RL fine-tuned models are factually consistent as measured by SummaCZS metric.
For ablations on PPO params, NLPO params, we refer to Tables 18,19.
[Table]
ub
[Table]
a
[Table]
L
[Table]
[Figure] [Figure]
[Table] [Table]
[Figure]
Figure 8 shows the summarization instructions and interface used for the human evaluation experiments.
Participants weren’t required to read the entire article, but to encourage some reading, a minimum time on the window of 15s was enforced via hiding the sliders.
Tables 20, 21 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans.
B.5. Q UALITATIVE A NALYSIS
We show sample generations from each of the algorithms for three randomly picked prompts below.
[Table]
like Raheem Sterling, Gareth Bale, Kevin de Bruyne and Ross Barkley.
City boss Manuel Pellegrini has been hampered over the past year by UEFA restricting them to a net transfer spend of 49million in each window and keeping the clubâ˘A´Zs overall wage bill to its current level of 205million-a-year.
UEFAâ˘A´Zs settlement with City published in May stated those penalties would remain in place until the end of the 2015/16 season but the clubâ˘A´Zs latest financial figures showed drastically-reduced losses of 23million which they feel proves they are now compliant with FPP regulations.
Manuel Pellegrini is hoping that the financial restrictions imposed by UEFA for a breach of FFP rules will be lifted at the end of this season .
Manchester City have been limited to a net spend of 49 million in the last two transfer windows - they spent 25m bringing Wilfried Bony in from
Swansea in January .
Ahead of Monday nightâ˘A´Zs trip to Crystal Palace, Pellegrini was certainly talking like a man excited at the prospect of signing ’crack’ players this summer.
â˘AŸI think that next season we donâ˘A´Zt have any restrictions so we will be in the same position that all the other English clubs have,â˘A´Z said Pellegrini.
â˘AŸItâ˘A´Zs important.
You have so many strong teams here in England and in Champions League, you can not allow them to keep the advantage every year; having less players to put in your squad or spending less money.
We spend money, of course we always spend money, but they spent more.â˘A´Z Manchester United, Barcelona, Liverpool and Arsenal have all paid more in transfer fees in the past 12 months than City who were traditionally Europeâ˘A´Zs biggest spenders after the club was taken over by Abu Dhabi owners in 2008.
Uefa also ordered City to play with a reduced squad from 25 players to 21 in the Champions League this season and while that restriction has now ended, any time reduction in the penalties on spending and wages is more controversial.
Arsenal have paid more in transfer fees than
City in the last 12 months, including 30m on Alexis Sanchez .
The document published last May by UEFAâ˘A´Zs Club Financial Control Body investigative chamber explicitly said
Cityâ˘A´Zs financial penalties would run for two seasons at least and there has been no official deviation from that decision.
The published statement said at the time: â˘AŸManchester City agrees to significantly limit spending in the transfer market for the seasons 2014/15 and 2015/16.
It means City will have to argue their case with Uefa that as they have been financially compliant over the past year, they deserve to be free of restrictions moving forward.
They have successfully argued their case with UEFA before.
Last summer they persuaded the governing body to allow them to bypass the normal quota of eight homegrown players as their Champions
League squad had been reduced.
Eliaquim Mangala joined the champions from Porto for 32m last summer .
The reigning Premier League champions have only paid money for Fernando,
Willie Caballero, Eliaquim Mangala and Wilfried Bony in the last two transfer windows and that was part-paid by the sales of Javi Garcia and Jack Rodwell.
Pellegrini admitted they werenâ˘A´Zt in a position to compete with neighbours United for established world stars like Angel di Maria because of the FFP penalties.
It has cost City on the pitch.
Even if they win their remaining eight Premier League fixtures their points tally will be lower than last season and they crashed out once again at the last-16 stage of the
Champions League.
Pellegrini and director of football Txiki Begiristain both accept the squad needs replenishing and they want to be in the market for the top available players.
Manchester City have been heavily linked with a move for former Chelsea midfielder Kevin de Bruyne .
The club also want to be involved in any race for Liverpool contract rebel
Raheem Sterling .
They have been strongly linked with former Chelsea winger Kevin de Bruyne who has been one of the best players in the Bundesliga with Wolfsburg.
And if
Liverpool do allow Sterling to leave, City want to be involved in the chase for one of Englandâ˘A´Zs most outstanding talents to help fit in with quotas for homegrown players.
â˘AŸIf you are the manager or a player of a big tem, you are always open to criticism because everyone expects a high level of performance,â˘A´Z he said.
â˘AŸThe major issue with our team always seems to be the money but this season I think we were the team that spent less than any other team.
That is the weird thing.
â˘AŸFor this club trying to be a big team in so few years, maybe we are paying the cost for that.â˘A´Z Since the transfer penalties were introduced, City have spent 91million on players in the last two windows (
Net spend 68million).
That compares to Manchester United
[Table]
5million, Liverpool 113 million,
[Table]
5million and
[Table]
6million.
Over the same time period Barcelona spent
[Table]
on
[Table]
and
[Table]
though they also
[Table]
the
[Table]
at the end of the current season in order to bid for stars such as Raheem Sterling.
Manuel Pellegrini is hoping that the financial restrictions will be lifted at the end of the 2015/16 season
Supervised + PPO :
Manchester City have been restricted to a net spend of 49million in each window and kept their overall wage bill at 205m. Manuel Pellegrini believes these restrictions will be lifted at the end of the season.
Arsenal have spent more in transfer fees than City in the last 12 months, including 30m on Alexis Sanchez.
Supervised + NLPO :
Manchester City are confident UEFAâ˘A´Zs punishment for a breach of financial fairplay regulations will be lifted this summer.
The club have been restricted to a net transfer spend of 49million over the past two window.
They aim to bid for stars like Raheem Sterling, Gareth Bale and Kevin de Bruyne.
City lost 23m on transfers in the last 12 months due to financial restrictions imposed by UEFA for breach
Sample 2 Prompt : (CNN)Larry Johnson remembers the fear and feeling of helplessness from being on the SkyWest Airlines flight that made an emergency landing in Buffalo, New York.
"I thought we were done," he said Thursday, one day later.
"There was no doubt in my mind that we weren’t going to land."
Johnson was flying with his brother, his girlfriend and his 8month-old son when he says a flight attendant came over the speaker asking for someone who was medically trained to help with a sick passenger.
Minutes later, Johnson says, the attendant announced there was a pressurization problem and told passengers to prepare for the emergency landing.
"You’re going to feel dizzy and woozy and lightheaded, and you’ re not going to be able to breathe," Johnson recalled the flight attendant saying.
The Federal Aviation Administration on Wednesday initially reported a pressurization problem with SkyWest Flight 5622, and said it would investigate.
It later issued a statement that did not reference any pressurization issues.
"It was like being trapped and you couldn’t do anything.
You have no control over it.
Your body just felt pressured," Johnson said.
Johnson said his girlfriend was seeing spots and that "when I was blinking, it was really bright.
I could barely see."
According to Marissa Snow, spokeswoman for SkyWest, three passengers reported a loss of consciousness while on the flight.
Fifteen passengers were examined by medical personnel at the Buffalo airport, but no one was transported to the hospital, she said.
The spokeswoman said that maintenance personnel found no indication of a pressurization problem with the aircraft, an Embraer E170, and that the airline continues to investigate the cause.
An official with the National Transportation Safety
Board told CNN that the agency is in communication with the FAA and SkyWest to gather information on the incident to better understand what took place.
Mary Cunningham was the nurse on board who attended to the first sick passenger.
"There was a woman, very lethargic, very nauseous, turning gray, just not looking good," Cunningham said.
After administering oxygen to the woman, Cunningham, a nurse at Yale-New Haven Hospital in Connecticut, returned to her seat.
Soon after, Cunningham said, she was called up again by the flight attendant after another women near the first started to lose consciousness.
"It was something that was affecting oxygen in that area," which was near the middle of the plane, she said.
Cunningham said she had to return to her seat after feeling lightheaded.
There were 84 passengers on board the plane, including nine crew members.
Flight 5622 was originally scheduled to fly from Chicago to Hartford, Connecticut.
The plane descended 28,000 feet in three minutes.
"It would feel like a roller coaster -- when you’re coming over the top and you’re going down," CNN aviation analyst Mary Schiavo said, describing how such a descent would feel.
"You know that these pilots knew they were in a very grave and very serious situation."
Johnson, flying to visit his parents with their only grandchild, was glad that roller coaster ride ended safely
[Table] [Table]
was a shaky ride
[Table]
It wasn’t anything I would like to experience again,"
[Table]
report a
[Table]
.But officials say
[Table]
no evidence of a pressurization problem
[Table]
PPO : SkyWest flight 5312 made emergency landing in Buffalo, new york.
passengers were told they were not going to be able to breathe, passenger recalls attendant.
the faa initially reported a pressurization problem with the flight 5622.
NLPO : SkyWest flight 5622 made emergency landing in buffalo, new york.
Larry Johnson says flight attendant announced there was a pressurization problem.
three passengers reported loss of consciousness while on flight; no one transported to hospital.
Supervised : The FAA says it is investigating the cause of the emergency landing.
SkyWest
Flight 5622 made an emergency landing in Buffalo, New York.
The flight was carrying a family of three people.
The flight attendant told the passengers to prepare for the landing.
Supervised
[Table]
PPO
[Table]
"I thought we were done," the father of one passenger says.
The flight attendant announced there was a pressurization problem.
The flight made an emergency landing in Buffalo, New York.
Three passengers reported a loss of consciousness while on the flight.
Supervised
[Table]
NLPO : SkyWest Airlines flight made an emergency landing in Buffalo, New York, on Thursday.
"There was no doubt in my mind that we weren’t going to land," passenger says.
"You’re going to feel dizzy and woozy and lightheaded...
you’re not going to be able to breathe," he says."
[Table]
in their
[Table]
featured not one, but two
[Table]
One
[Table]
Celtic didn’t get for
[Table]
0 in which
[Table]
handball should have also led to a sending off.
The other the spot kick they did, followed by a
[Table]
for
[Table]
Gordon. ’
I’ve not seen it yet, but going by the reaction of the Celtic players we got away with a penalty and a sending off and that was probably the turning point in the game,’ acknowledged Caley manager John Hughes after.
Inverness’s Josh Meekings appears to get away with a handball on the line in their win over Celtic .
Caley boss John Hughes says the break, which could have meant a penalty and red card, was a turning point . ’
I’ve not spoken to Josh.
I haven’t seen it - but going by the media it was definitely a hand ball.
We look at the referee behind the line and all that
and I know Ronny will feel aggrieved - because I certainly would. ’
But it’s part and parcel of football and you need a wee bit of luck to beat Celtic. ’
This was their biggest game of the season because they will go on and win the league and if they had beaten us today there was a good chance they would have gone on and won the Scottish Cup.
But when Marley Watkins was clipped by Craig Gordon and they were down to 10 men that was advantage Inverness.
We weren’t going to give Celtic the ball back, they had to come and get it and we had to be patient. ’
When big Edward put us into the lead we thought it was going to be our day on the back of things that had happened. ’
Celtic equalised with another free kick but it’s typical of Inverness that we don’t do anything easy. ’
We do it the hard way and we came up with the winner through David Raven.’
Hughes hauled Raven, his Scouse defender, from his backside as extra-time beckoned.
Offended by the sight of one of his players resting he had a message to impart.
Caley players celebrate after upsetting Celtic in a Scottish Cup semi-final 3-2 thriller .
Celtic, depleted by games and absentees, were virtually on their knees after a relentless programme of midweek games.
In last season’s League Cup Final Inverness had been passive and unambitious prior to losing on penalties.
This was no time to repeat the mistake. ’
I tried to emphasise to the players they would never have a better time to go on and beat Celtic, down to 10 men in the semi final of a cup.
We needed to go for it,’ Hughes said. ’
Before Raven scored at the back post I was looking to change it.
I was going to bring on another winger, Aaron Doran, and put him in the full-back position over on the right, but more advanced so he could take their left back on.
Thankfully I didn’t do that and David Raven came up with the goal.
Virgil Van Dijk (centre) fired Celtic into an early lead with a superb free-kick in the 18th minute . ’
I didn’t realise this is the first time the club have been in the final of the Scottish Cup and that’s a remarkable achievement given it was only formed 20 years ago. ’
It is a great story isn’t it?
It’s an absolutely fantastic story.
It is 20 odd years since the amalgamation.
We are a small provincial club up there in the Highlands. ’
We have lost a real inspirational skipper in Richie Foran right from the start of the season.
He has never played.
We have had to adjust to that. ’
We had to sell Billy McKay, our top goalscorer, at Christmas.
We have had to go again and adjust.
I am a very humble guy and I am grateful and thankful that injuries have never caught up with us.’
There is remarkable irony in the fact
Falkirk will be the opponents for the final.
A former Bairns captain, he was manager of the club in 2009 when they lost to Rangers at Hampden.
Former Falkirk captain and manager
John Hughes will take on his former club in the final . ’
I had a lot of great times at Falkirk.
So much so that it is possibly my favourite time in my playing career.
I am still friendly with an awful lot of the characters who were in that dressing room.
Neil Oliver is a good friend of mine from my Falkirk days.
He comes along and sits on the bench and gives me a wee hand out. ’
That is the spirit that we had at that club.
I have met some great guys.
Brian Rice, my ex-coach, Davie Weir, Ian McCall, the list is endless.
I was just talking the other day about that day at Falkirk.
There are times even now when I see. ’
I have a real special, special feel for the place.
I am not kidded Falkirk is a right good side, trust me on that.
No side goes to Tynecastle and beats Hearts without being a good side.
No side beats Hibs the way they were playing without being a good side.
You might say we are favourites I say it is even stevens and I am just looking forward to when the days comes.’
David Raven (second right) celebrates with his
Inverness team-mates following his extra-time winner .
An Inverness and Falkirk final would not have been the choice of sponsors William Hill, you suspect.
Vast swathes of
Hampden’s west stand were empty, a 12.15 kick-off for the benefit of broadcasters proving unhelpful .
There will be no ticket rows this time.
Not over a final unlikely to raise a crowd much higher than 35,000 in a stadium of 52,000.
The contract with Hampden plc is signed, with no question of moving it.
If the bitter amalgamation of 21 years ago produced a fine football club it also disenfranchised many of those who followed the old ones. ’
I hope this encourages the next generation to follow Inverness,’ added Hughes. ’
In the peer groups it is still Celtic and Rangers they follow
but I hope the next ones will want to support their local club. ’
If we can add another few hundred or thousand to the support then we are playing our part.
Shot : inverness beat Celtic
0 in their opening game of the season.
Josh Meekings was the subject of a handball after the break.
the break could have meant a penalty and a red card for Craig Gordon.
References : Josh Meekings handball should have been a penalty and a red card .If
the penalty was awarded and converted Celtic would have gone to 2-0 .Caley
manager John Hughes admitted his side were fortunate in win .Virgil
van Dijk scored Celtic’s opener with a superb free-kick .Celtic
keeper Craig Gordon was sent off early in the second half .Greg
Tansey, Edward Ofere and finally Daven Raven scored for Caley .
PPO :
Josh Meekings handball could have meant sending off for Celtic in extra-time win.
John
Hughes says the break could have meant an penalty and red card.
Caley boss also admits
[Table]
B. T O TT O B.6. S ETUP
ToTTo (Parikh et al., 2020) is a controlled table-to-text generation task in which the goal is to produce one-sentence description of highlighted table cells.
For training RL methods, we considerdifferent reward functions: BLEU, SacreBLEU, METEOR, PARENT and a combination of Meteor and PARENT.
We chose T5 as our base LM here too, as they are more suitable for structure to text tasks.
For decoding, we use beam search during inference and for generating rollouts, we use top k sampling.
Other implementation details are captured in Table
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
R ESULTS AND D ISCUSSION
, 23 presents our benchmarking results with 5 reward functions along with supervised baseline performances on dev and test sets respectively.
Similar to other tasks, our main finding is that warm-started initial policies are crucial for learning to generate descriptions from highlighted cells.
Without warm-start, policies suffer from reward hacking and resulting in sub-optimal solutions despite application of task-specific metrics such as PARENT etc.
We find that Supervised+NLPO method outperforms all models on ToTTo leaderboard in terms of PARENT metric.
[Table]
[Figure]
P ub li
[Table]
e d a s a c on f e
[Table]
e n ce p a p e r a t
I C L
R
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Without supervised pre-training, both PPO and NLPO results in sub-optimal solutions, with NLPO better than PPO.
With supervised pre-training, PPO and NLPO achieve better scores across all metrics showing RL fine-tuning is beneficial.
Most importantly, RL fine-tuned models produce more factually correct text as seen in higher PARENT scores.
Another observation, fine-tuning with a task-specific metric PARENT is better than training just on task-agnostic lexical metrics
[Table] [Table]
B.6.
[Figure]
Figure 9 shows the ToTTo instructions, example, and interface used for the human evaluation experiments.
We made small modifications to the original code release’s HTML renderer to make the tables display in our HITs.
Tables 25, 26 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans.
B.6. Q UALITATIVE A NALYSIS
We show sample generations from each of the algorithms for three randomly picked prompts below
[Table]
B. N ARRATIVE QA B.7. S ETUP
NarrativeQA (Koˇcisk`y et al., 2018) deals with task of generating answers to questions about a given story.
For training RL methods, we consider 2 traditional lexical rewards namely Rouge Combined and Rouge-L-Max.
We chose T5-base as the base LM since it has been shown to do well at question answering in prior work (Khashabi et al., 2020).
We note that the supervised models we use are trained on the UnifiedQA dataset, which contains other QA datasets
[Table]
and is shown by Khashabi et al.
(2020) to outperform supervised fine-tuning only on NarrativeQA.
Hyperparams for our models can be found in Table
[Table] [Table]
ub
[Table]
e
[Table]
a
[Table]
a c
[Table]
e n ce p a
[Table]
e
[Table]
a
[Table]
I C L
[Table]
B.7. R ESULTS AND D ISCUSSION
Table 28 presents our benchmarking results with 2 reward functions along with supervised baseline performances on the NarrativeQA test set.
Similar to other methods, our main finding is that warmstarted initial policies are crucial for learning to generate answers that successfully use the input context.
B.7. Q UALITATIVE R ESULTS
We show sample generations from each of the algorithms for three randomly picked prompts below
[Table]
the fcc is called in to investigate.
during the fracas, it is revealed that the school’s principal (annie ross) has been expelling " problem students," namely, students with below-average standardized test scores, in an effort to boost the district’s test scores while still keeping their names on the rolls ( a criminal offense) in order to retain government funding.realizing he has started something huge, mark decides it is up to him to end it.
he dismantles his radio station and attaches it to his mother’s old jeep, creating a mobile transmitter so his position can’t be triangulated.
pursued by the police and the fcc, nora drives the jeep around while mark broadcasts.
the harmonizer he uses to disguise his voice breaks, and with no time left to fix it, mark decides to broadcast his final message as himself.
they finally drive up to the crowd of protesting students, and mark tells them that the world belongs to them and that they should make their own future.
the police step in and arrest mark and nora.
as they are taken away, mark reminds the students to "talk hard."
as the film ends, the voices of other students (and even one of the teachers) speak as intros for their own independent stations, which can be heard broadcasting across the country.
[Table]
that
[Table]
from the
[Table]
of his
[Table]
is
[Table]
, an
[Table]
, whose only outlet for his teenage angst and aggression is his unauthorized radio station.
his pirate station’s theme song is "everybody knows" by leonard cohen and there are glimpses of cassettes by such alternative musicians as the jesus and mary chain, camper van beethoven, primal scream, soundgarden, ice-t, bad brains, concrete blonde, henry rollins, and the pixies.
by day, mark is seen as a loner, hardly talking to anyone around him; by night, he expresses his outsider views about what is wrong with american society.
when he speaks his mind about what is going on at his school and in the community, more and more of his fellow students tune in to hear his show.nobody knows the true identity of "hard harry" or "happy harry hard-on," as mark refers to himself, until nora diniro ( mathis), a fellow student, tracks him down and confronts him the day after a student named malcolm commits suicide after harry attempts to reason with him.
the radio show becomes increasingly popular and influential after harry confronts the suicide head-on, exhorting his listeners to do something about their problems instead of surrendering to them through suicideÂ˘AÂˇTat the crescendo of his yelled speech, an overachieving student named paige woodward (who has been a constant listener) jams her various medals and accolades into a microwave and turns it on.
she then sits, watching the awards cook until the microwave explodes, injuring her.
while this is happening, other students act out in cathartic release.eventually, the radio show causes so much trouble in the community that the fcc is called in to investigate.
during the fracas, it is revealed that the school’s principal (annie ross) has been expelling "problem students," namely, students with below-average standardized test scores, in an effort to boost the district’s test scores while still keeping their names on the rolls (a criminal offense) in order to retain government funding.realizing he has started something huge, mark decides it is up to him to end it.
he dismantles his radio station and attaches it to his mother’s old jeep, creating a mobile transmitter so his position can’t be triangulated.
pursued by the police and the fcc, nora drives the jeep around while mark broadcasts.
the harmonizer he uses to disguise his voice breaks, and with no time left to fix it, mark decides to broadcast his final message as himself.
they finally drive up to the crowd of protesting students, and mark tells them that the world belongs to them and that they should make their own future.
the police step in and arrest mark and nora.
as they are taken away, mark reminds the students to "talk hard."
as the film ends, the voices of other students (and even one of the teachers) speak as intros for their own independent stations, which can be heard broadcasting across the country.
Zero
[Table]
Shot : mark hunter
[Table]
slater) starts an fm
[Table]
radio station that broadcasts from the basement of his parents’ house.
by day, mark is seen as a loner, an outsider, whose only
[Table]
hunter (slater) starts an fm pirate radio station that broadcasts from the basement of his parents’ house.
mark is a loner, an outsider, whose only outlet for his teenage ang
[Table]
mark hunter (slater) starts an fm pirate radio station that broadcasts from the basement of his parents’ house.
mark is a loner, an outsider, whose only outlet for his teenage ang Supervised : he was taking them to a psychiatric hospital.
Supervised
[Table]
PPO : addressing them
Supervised
[Table]
B. M ACHINE T RANSLATION B.8. S ETUP
WMT-16 We pick two languages, English and German
[Table]
and frame this task similarly to other machine translation tasks—requiring the models to translate from English to German.
We train models onrewards: SacreBLEU, chRF, TER
[Table]
and BertScore
[Table]
R ESULTS AND D ISCUSSION
Tables 30, 31 presents our benchmarking results with 4 reward functions along with supervised baseline performances on test set.
Our main finding is that NLPO + Supervised performs better than
PPO and supervised models
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
IWSLT
[Table]
[Table]
IWSLT
[Table]
B.8. Q UALITATIVE R ESULTS
We show sample generations from each of the algorithms for three randomly picked prompts from
IWSLT below.
[Table]
B. D AILY D IALOG B.9. S ETUP
We consider DailyDialog (Li et al., 2017) as the test bed for the dialogue generation task.
The dataset includes conversations written by human on various topics.
In addition, each utterance contains labels of intent and emotional information.
For simplicity, we focus only on generating the next utterance, given the dialogue context.
We chose a context window of size 5 , which results in 35 k training, 3 k and 3 k utterances.
The input to the model is dialogue history in which utterances are concatenated using a <
EOU> token.
We picked GPT-2 as the LM as they are more suited for text continuation than encoder-decoder LMs.
For a fair comparison, we use top-k sampling with k = 20 as the decoding method for all methods.
For RL methods, we use a linear combination of meteor score and intent match score (whether the generated text’s intent matches with the reference’s intent) as the reward function.
The coefficients for meteor and intent are chosen based on both lexical scores and intent accuracy on the validation set.
For this purpose, we trained an intent classifier (fine-tuned RoBERTa (Liu et al., 2019)) that classifies given text into intent categories such as inform , question , directive and commisive , etc.
Table 32 provides a summary of hyperparameters and implementation details.
[Table]
P ub li s h e d a s a c on f e r e n ce p a p e r a t
I C L
R
Tasks
LexicalandSemanticMetrics
DiversityMetrics Alg
RewardFunction
[Table]
Distinct
[Table]
Distinct
[Table]
H
[Table]
Unique
[Table]
Dialog
ZeroShot
GPT-
[Table] [Table]
[Table]
Supervised
[Table] [Table]
[Table] [Table]
PPO
Meteor
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
538 NLPO
Meteor
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Supervised+PPO
Meteor
[Table]
Intent
[Table] [Table]
[Table] [Table]
829 Supervised+NLPO
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
B.9. R ESULTS AND D ISCUSSION
Table 33 presents our benchmarking results of RL methods along with supervised baseline performances on test sets.
Our main finding is that RL methods generally achieve better intent accuracy and automatic metric scores, in particular NLPO variants perform better than all other methods
. B.9. H UMAN P ARTICIPANT S TUDY
Figure 10 shows the Daily Dialogue instructions and interface used for the human evaluation experiments.
Tables 34, 35 show averaged results, annotator agreement
[Table]
and the results of statistical significance tests to determine which models output better generations when rated by humans
[Table]
B.9. Q UALITATIVE A NALYSIS
We show sample generations from each of the algorithms for three randomly picked prompts below.
[Table] [Table]