F ANTASTIC R EWARDS AND H OW TO T AME T HEM : A C ASE S TUDY ON R EWARD L EARNING FOR T ASKO RIENTED D IALOGUE S YSTEMS
Anonymous authors
Paper under double-blind review
A BSTRACT
When learning task-oriented dialogue (TOD) agents, one can naturally utilize reinforcement learning (RL) techniques to train dialogue strategies to achieve userspecific goals.
Prior works mainly focus on adopting advanced RL techniques to train the TOD agents, while the design of the reward function is not well studied.
This paper aims at answering the question of how to efficiently learn and leverage a reward function for training end-to-end TOD agents .
Specifically, we introduce two generalized objectives for reward-function learning, inspired from the classical learning-to-rank literature.
Further, we utilize the learned reward-function to guide the training of the end-to-end TOD agent.
With the proposed techniques, we achieve competitive results on the end-to-end response-generation task on the
Multiwoz 2.0 dataset.
I NTRODUCTION
The bloom of pre-training language models ( e.g. , Devlin et al., 2018; Lewis et al., 2019; Radford et al., 2019; Zhang et al., 2022) have significantly pushed the boundaries of natural language processing (NLP) on real-world tasks.
Among all the promising potentials, one important application is the task-oriented dialogue (TOD) systems, which interact with the users in multiple turns via natural languages to accomplish tasks such as weather inquiry, ticket booking or schedule planning (Chen et al., 2017; Kwan et al., 2022).
Traditionally, the problem of TOD is decomposed into several sub-tasks (Smith & Hipp, 1994;
Young et al., 2013): natural language understanding (NLU) for understanding turn-level user intents or slot values (Tur & De Mori, 2011; Casanueva et al., 2020), dialogue state tracking (DST) for tracking user belief state across multiple dialogue turns (Zhang et al., 2019; Zhu et al., 2020), dialogue management (DM) for choosing system actions to take (Peng et al., 2017;
Zhao et al., 2019), and natural language generation (NLG) for mapping system actions to natural language responses (Wen et al., 2015; Zhang et al., 2020).
This pipeline approach, however, requires intensive structural designs and comprehensive data annotation for training (Kwan et al., 2022).
Recently, there has been a growing interest in building end-to-end TOD agents, which directly generate responses based on the natural language conversation mixing user utterances and past responses.
Apart from this structural simplicity, many of the end-to-end TOD models can utilize the pretrained language models and are simply trained by supervisely fine-tuning the pretrained models on the TOD datasets ( e.g. , Hosseini-Asl et al., 2020; Ham et al., 2020; Lin et al., 2020; Peng et al., 2021).
Due to the intrinsic similarity between dialogues and sequential decision-making, reinforcement learning methods are naturally employed to train dialogue systems and have achieved great success ( e.g. , Williams & Young, 2007; Georgila & Traum, 2011; Zhao et al., 2019).
Since interacting with users during the training process is mostly impractical, offline RL (Lange et al., 2012; Levine et al., 2020), i.e. , RL on static datasets, has been recently adopted to train end-to-end TOD models (
e.g. , Jaques et al., 2019; 2020; Ramachandran et al., 2021; Snell et al., 2022a;b; Jang et al., 2022).
Although this direction already presents promising empirical results, an open question exists on how to properly design the reward function for the underlying (offline) RL.
Existing works ( e.g. , Wu et al., 2019c; Jang et al., 2022; Snell et al., 2022b)
manually design a sparse reward function that only indicates whether the agent achieves the goal or not.
Unfortunately, due to the delayed feedback, learning from such a sparse reward-signal is itself challenging for RL agents (Andrychowicz et al., 2017; Liu et al., 2019; Durugkar et al., 2021).
When applying to train the more complicated TOD agents, the sparse reward-signal could lead to poor empirical performance (Takanobu et al., 2019;
Wang et al., 2020a).
To address the issue, we aim at answering the following question in this paper:
How to efficiently learn a reward function and leverage it for training end-to-end dialogue agents?
We answer the first half of the question by introducing two reward-learning objectives
RewardNet and RewardMLE , based on classical learning-to-rank literature (Cao et al., 2007;
Xia et al., 2008).
Our desiderata is a reward function that can “explain” some non-trivial preferencebased ordering among multiple alternative dialogue trajectories, thus potentially allowing the resulting RL-trained TOD agents to have better-than-demo performance.
We accomplish this idea by learning a parameterized reward function on dialogue turns, from which the accumulated reward of a dialogue trajectory can reflect the preference among the multiple alternatives.
We answer the second half of the question by utilizing the learned reward-function to guide the training of the end-to-end
TOD system, with special considerations on the training stability.
With the answers to the above question, we achieve competitive results on the end-to-end response-generation task on the widelyused dialogue benchmark MultiWOZ 2.0 (Budzianowski et al., 2018).
Several ablation studies and analyses are further conducted to provide insights on the proposed techniques.
B ACKGROUND
Task-oriented dialogue as reinforcement learning.
We formulate the problem of task-oriented dialogue system as a partially observable Markov decision process (POMDP) (Kaelbling et al., 1998), specified by M = ⟨ S , A , O , P , R , γ ⟩ , where state s ∈ S consists of the previous dialogue history h and the user intended goal g specified prior to the start of the dialogue; o ∈ O is the observation that can be the user utterance; action a ∈ A can be the system response or dialogue act, whose dimension is the size of the whole vocabulary (denoted by | A | ); P ( s ′ | s, a ) is the underlying transition probability; R ( h, a, g ) is the intermediate reward function for taking action a under dialogue history h and goal g ; and γ ∈ [0 , 1] is the discount factor.
The dialogue history h t at timestep t consists of all the previous observations and actions, i.e. , h t ≜ { o 0 , a 0 , . . .
, o t − 1 , a t − 1 , o t } .
Since the TOD agent cannot directly observe the user goal g , it makes decision based on the entire dialogue history h t
so far.
Specifically, the policy π is defined as a mapping from h t to a probability distribution over A , i.e. , π ≜ π ( a t | h t ) .
The training objective is to find a policy π that maximizes the expected (discounted) cumulative reward
[Equation]
where µ g is the sampling distribution of goals and T is the number of turns in a dialogue trajectory.
Reward design and learning in task-oriented dialogue systems.
Unlike the classical RL problems where the intermediate reward function is well designed and provided, in TOD systems we can only get the evaluation results at the end of the dialogue (Budzianowski et al., 2018).
Consequently, most of the existing works adopt the manually designed intermediate reward function that only gives binary reward to indicate whether the dialogue agent achieves the goal or not ( e.g. , Weisz et al., 2018; Wu et al., 2019c; Jang et al., 2022):
[Equation]
goal
[Equation]
achieved at timestep
[Equation]
if goal g is not achieved at timestep t , where R const is a positive constant that can be 1.
However, such a sparse reward signal can be one of the reasons that the TOD agents from RL often have poor performance (Takanobu et al., 2019;
Wang et al., 2020a).
Similar issue is also observed in goal-oriented RL (Andrychowicz et al., 2017).
To address the above issue, a few recent works focus on learning an intermediate reward function from demonstrations or mechanical dialogue assessments ( e.g. , Wang et al., 2020a;
Ramachandran et al., 2021), inspired by the reward-learning-from-preferences in RL ( e.g. , Christiano et al., 2017; Brown et al., 2019; 2020).
More precisely, suppose we are given two dialogue trajectories
τ i and τ j , taking the form τ i ≜ { g ( i ) , ( o ( i ) 0 , a ( i ) 0 ) , . . .
, ( o ( i ) T , a ( i ) T ) } , and we want to learn a parametrized reward function R
θ ( o t , a t , g ) with parameter θ , 1 such that (cid:80) Tt =
0 R θ
( o ( i ) t , a ( i ) t , g ( i ) )
> (cid:80) Tt =
R θ ( o ( j ) t , a ( j ) t , g ( j ) )
when τ i is preferred over τ j (denoted by τ i ≻
τ j for short) and vice versa .
Then one can follow the Bradley-Terry model of preferences (Bradley & Terry, 1952) to train the reward function by minimizing the loss
[Equation]
ℓ ( θ ) can be interpreted as a pairwise ranking loss, which is formalized as binary classification in the problem of learning to rank (Herbrich et al., 1999;
Freund et al., 2003; Burges et al., 2005).
M AIN M ETHOD
In this section, we first introduce objectives for reward-function learning based on the classical approach in the learning-to-rank (LTR) literature (Liu, 2009).
Then we use MinTL (Lin et al., 2020) as an example to describe how we can use the learned reward function as a plugin module to improve existing methods on training the end-to-end TOD model.
3. T WO G ENERALIZED O BJECTIVES FOR R EWARD L EARNING
We introduce two objectives RewardNet and RewardMLE , both of which can utilize multiple dialogue trajectories on each update for optimizing the reward function.
Our intuition is that, compared with the pairwise approach described in Eq.
(1), these two objectives consider more information at each training step, and thus can be more effective for reward learning and may lead to a better solution under the stochastic training setting.
Setup.
Assume there are N ≥ 2 dialogue trajectories denoted by D N ≜ ( τ 1 , τ 2 , . .
. , τ N ) , and each trajectory τ i has an automatic evaluation score S ( τ i ) .
For simplicity, we assume that these N dialogue trajectories are of equal length T and are already sorted by the automatic evaluation scores, i.e. , τ 1 ≻ τ 2 ≻ · · · ≻ τ N , or equivalently, S ( τ 1 ) > S ( τ 2 ) > · · · > S ( τ N ) .
We also denote the accumulated reward of the dialogue trajectory τ i from R θ as J ( τ i ; θ )
=
(cid:80) Tt =
0 R θ
( o ( i ) t , a ( i ) t , g ( i ) ) .
Our goal is to learn the reward function R θ ( o, a, g ) such that the accumulated reward of the trajectories can reflect the ranking order, i.e. , J ( τ 1 ; θ ) > · · · > J ( τ N ; θ ) .
RewardNet .
The proposed RewardNet objective for reward function learning is adopted from the ListNet loss (Cao et al., 2007) in the LTR literature.
Specifically, given the N trajectories and their associated scores, we define the RewardNet loss as the cross entropy between { J ( τ i ; θ ) } Ni = and { S ( τ i ) } Ni =1 :
[Equation]
(2) with
[Equation]
, where Φ( · ) is a monotonic and positive function defined on R + , and P S ( τ i ) is a normalized probability defined by the true evaluation score of trajectory τ i .
Note that when N = 2 and Φ is the identity function, RewardNet can be viewed as a soft version of the pairwise preference loss defined in Eq.
(1), where the hard binary preference labels are replaced with { P S ( τ i ) }
Ni =1 .
This soft pairwise loss is adopted for reward learning in the recent CASPI paper (Ramachandran et al., 2021).
RewardMLE .
The RewardMLE objective is based on the ListMLE loss (Xia et al., 2008), where we only utilize the ranking order in the batch dialogue trajectories D N , instead of the original metric scores { S ( τ i ) } Ni =1 .
Let y = rank( S ) be the random variable that represents the ranking order of the dialogue trajectories ( y ( τ i )
=
i, ∀ i , if the batch trajectories D N are sorted).
The RewardMLE objective is defined as the negative log-likelihood of the ranking order y under the Plackett-Luce choice model (Plackett, 1975; Luce, 2012) induced by the accumulated reward of each trajectory { J ( τ i ; θ ) } N i =1 .
Specifically, the loss is defined as
[Equation]
with
[Equation]
where trajectories in D N are assumed sorted as described in the problem setup, i.e. , τ 1 ≻ . . .
≻ τ N .
Since RewardMLE only uses the ranking information derived from the raw scores, it is potentially a more robust choice when the preference scores are inaccurate.
In Eqs.
(2) and (3), the monotonic and positive function
Φ transforms the unnormalized inputs { J ( τ i ; θ ) } Ni =1 to a N -dimensional probabilistic simplex.
In this work, we consider Φ as exponential function exp( · ) and power function ( · ) p , p ∈ N , which are known as the softmax and escort transforms respectively (Mei et al., 2020).
3. P OLICY G RADIENT E STIMATION WITH L EARNED R EWARD F UNCTION
With the learned reward function R θ ( o, a, g ) , the next step is to improve the parametrized dialogue agents π ϕ via policy gradient methods (Sutton & Barto, 2018) given a collected offline dataset ˆ D := { τ k } Kk =1 .
A classical approach to train the policy π ϕ is to estimate the policy gradient via the
REINFORCE method (Williams, 1992):
[Equation]
where G π ϕ ( h t , ˜ a t , g ) is the (discounted) accumulated reward that the agent π ϕ receives, starting from observation o t (part of h t ) and action ˜ a t , under the given goal g .
When the discount factor γ > 0 , estimating G π ϕ ( h t , ˜ a t , g ) requires Monte Carlo sampling (on-policy) or temporal difference learning (off-policy), both of which require learning an additional value-function network.
Empirically we observe that learning an additional action-value function could introduce additional instability to the subsequent training of the end-to-end dialogue model.
To simplify the training pipeline, we simply set the discount factor γ = 0 , thus G π ϕ ( h t , ˜ a t , g )
=
R θ ( o t , ˜ a t , g ) .
Though the policy gradient estimator defined in Eq.
(4) is unbiased, it tends to have high variance, especially when the action space is large.
Unfortunately, in the end-to-end TOD system, the action space is often defined to be the vocabulary space, whose dimension is larger than 30000 .
As a result, optimizing the agent π ϕ with the REINFORCE estimator may suffer from divergent training.
We illustrate this phenomenon via a toy example in Section 5.2.
To address the high-variance issue of the REINFORCE estimator, we utilize the Gumbel-softmax (GS) trick (Jang et al., 2016; Maddison et al., 2016) to reduce the variance, [Equation] with f ϕ ( h t ,ϵϵϵ ) = (cid:104) f
(1) ϕ
( h t ,ϵϵϵ ) ,...,f ( | A
[Equation]
) ϕ
[Equation] and f ( i ) [Equation] ( h
[Equation]
ϵϵϵ
[Equation] = [Equation] where { l i ( h t ; ϕ ) } | A | i =1 are the logits of the categorical distribution defined by the agent π ϕ , and λ is the temperature that we set as 1.
Besides, following the pessimistic principle in offline RL (Buckman et al., 2020), we add a weighted regularization such that the actions generated by the agent π ϕ are close to the actions in the dataset ˆ D ,
[Equation]
which is similar to the weighted behavior cloning loss in offline RL (Wang et al., 2020b), except that we directly use the intermediate reward as the weights, rather than the value function.
Combining the policy gradient and the weighted regularization, we have the following loss for the agent π ϕ :
[Equation]
where α is a constant coefficient balancing these two parts.
Note that the original supervisedlearning loss of MinTL (Lin et al., 2020) can be decomposed into two parts, respectively for the dialogue state tracking (DST) and the response generation.
We retain the DST loss ℓ DST ( ϕ ) in
MinTL, and replace its response-generation loss with Eq.
Our final loss for TOD training is
[Equation]
We illustrate our methods in Fig. 1, and provide an algorithm box in Appendix B.
Remark
Eq.
(6) for the learning of the dialogue agent π ϕ is essentially a generalized objective from several previous works.
Specifically, if we set α = 0 and set the reward function to be constant
R θ ( o t , a t , g ) ≡ 1 , Eq.
(6) reduces to the objective in MinTL, without any guidance for responsegeneration from the learned reward function R θ .
If we set α = 0 , and use the RewardNet loss with N = 2 and Φ = ( · ) 1 ( i.e. , the identity function) to train the reward function, Eq.
(6) reduces to the objective in CASPI (Ramachandran et al., 2021).
In Section 5, we demonstrate the advantages of our techniques proposed in this section, including the RewardNet and RewardMLE losses for reward learning, and the J GS ( π ϕ ) for agent learning.
R ELATED W ORK
Recent works on the end-to-end task-oriented dialogue systems ( e.g. , Wu et al., 2019b; Lin et al., 2020; Hosseini-Asl et al., 2020; Ham et al., 2020;
Peng et al., 2021; Yang et al., 2021) have significantly improved the overall system performance, and simplify the algorithmic design in earlier works, which require solving several pipeline based sub-tasks ( e.g. , Young et al., 2013; Gao et al., 2018; Zhang et al., 2020).
The reward function trained with our methods can be leveraged as the guidance to train existing end-to-end models, without changing the underlying structures.
We demonstrate the effectiveness of our proposed reward learning methods under the structure of the
MinTL (Lin et al., 2020), where we only add an additional reward-function-guided objective for the response-generation model, while keeping other components of the MinTL unchanged.
One line of related research is applying RL to learn TOD agents.
It is often unsuccessful to directly apply RL algorithms such as the DDPG (Lillicrap et al., 2015) or PPO (Schulman et al., 2017), since the training of agents could potentially diverge (Zhao et al., 2019;
Jang et al., 2022; Kwan et al., 2022).
Recently, a number of works consider offline (batch) RL (Levine et al., 2020) as a potential solution to stabilize the agent training on a static dataset ( e.g. , Jaques et al., 2020; Ramachandran et al., 2021;
Jang et al., 2022; Verma et al., 2022; Snell et al., 2022a;b).
Following the offline RL principle, we use a reward-weighted regularization to stabilize the dialogue-agent training.
Together with the incorporation of the Gumbel-softmax trick to estimate the policy gradient, our work retains the algorithmic simplicity while improving the training stability and the overall performance.
Finally, our work closely relates to works on reward learning for task-oriented dialogue systems ( e.g. , Takanobu et al., 2019;
Ramachandran et al., 2021).
This line of research differs from works that directly adopt the manually designed reward function, which only gives sparse signals to indicate whether the agent achieves the goal or not ( e.g. , Weisz et al., 2018;
Wu et al., 2019c; Jang et al., 2022; Snell et al., 2022b).
One thread of this research direction is utilizing inverse reinforcement learning (IRL) (Russell, 1998) to learn a dense reward function, by considering the collected data as the expert demonstration (Takanobu et al., 2019).
However, modern IRL techniques such as GAIL-style algorithms (Ho & Ermon, 2016; Fu et al., 2017) often require iterating between reward and policy learning (Finn et al., 2016), which are computationally expensive and less scalable to dialogue-generation models.
Besides, the IRL methods aim at justifying the data, while the reward-learning framework in our work seeks to explain the preference among multiple trajectories, potentially leading to better-than-demo agents (Brown et al., 2019; 2020).
Our work is more closely related to the research on reward learning from preferences, which is widely used in recent NLP tasks, including training language models (Ouyang et al., 2022) and fine-tuning (Ziegler et al., 2019), question and answering with verification (Nakano et al., 2021;
Menick et al., 2022),
[Table]
and task-oriented dialogue systems (Ramachandran et al., 2021).
These works adopt the pairwise preference-learning objective in Christiano et al.
(2017), which can be viewed a special case of the
RewardNet loss discussed in Section 3.1.
Our work mainly focuses on the TOD tasks, where we study reward-function learning and utilization for training end-to-end dialogue agents.
E XPERIMENTS
Dataset.
We evaluate our proposed methods on the MultiWOZ 2.0 dataset (Budzianowski et al., 2018), which is a representative TOD benchmark.
MultiWOZ 2.0 is a large-scale and multi-domain dialogue corpus with seven domains: attraction, hospital, police, hotel, restaurant, taxi, and train.
Each dialogue therein covers between one to three domains.
This dataset has 8438 dialogues for the training set and 1000 dialogues for the validation and test set respectively.
Evaluation Metrics.
Our proposed method is evaluated on the end-to-end dialogue-modeling task of the MultiWOZ 2.0 dataset.
Following the standard setup ( e.g. , Budzianowski et al., 2018; Mehri et al., 2019), we use four automatic evaluations metrics: 1) Inform rate: the fraction of the dialogues where the system has provided an appropriate entity; 2) Success rate: the fraction of the dialogues where the system answered all the requested information; 3) BLEU score (Papineni et al., 2002): measures the fluency of the generated response; 4) Combined Score (Mehri et al., 2019): an overall quality measure defined as Combined Score ≜ (Inform + Success) × 0 .
5 + BLEU .
Details on prepossessing and implementation of our response and reward models are in Appendix B.
5. M AIN R ESULTS
Main evaluation.
Table 1 compares the performance of our methods with several classical and recent approaches in the end-to-end response-generation task.
As shown in Table 1, our proposed methods not only improve the dialogue-task completion, measured by the Inform rate and the Success rate; but also generate fluent responses, reflected by the competitive BLEU scores.
Recall that
CASPI (Ramachandran et al., 2021) is a special case of RewardNet loss (Eq.
(2)) when we use escort transform ( Φ = ( · ) 1 , the identity function) with pairwise preference ( N = 2 ).
When we use three dialogue trajectories ( N = 3 ) to estimate the RewardNet loss and retain the same escort transform, the overall performance generally improves over CASPI.
As discussed in Section 3.1, our generalized RewardNet loss considers more information on each update of the reward model, and thus could learn a better reward function.
The performance is further gently improved by changing the RewardNet loss (Eq.
(2)) to the
RewardMLE loss (Eq.
(3)), with the softmax transform ( Φ = exp( · ) )
and N = 5 dialogue trajectories.
This again demonstrates the benefit of our proposal of using multiple trajectories to learn the reward model.
Section 5.2 conducts ablation study on the number of trajectories and choice of Φ .
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
So far, we follow the prior work to not utilize policy gradient to train the response-generation model, i.e. , α = 0 in Eq.
Extra performance gain can be obtained by the policy-gradient updates via the Gumbel-softmax trick (GS) in Section 3.2.
In particular, GS improves both the previous
RewardNet and RewardMLE models.
This shows the efficacy of directly optimizing the responsegeneration model w.r.t.
the learned reward function.
Further discussion is provided in Section 5.2.
Low-resource experiment.
We evaluate our method on the limited-data setting by following the testing strategy in Lin et al.
Specifically, we use 5% , 10% , and 20% of the training data to train our basic RewardNet and RewardMLE models in Table 1, without the GS component.
We compare them with the baseline scores in Lin et al.
Table 2 reports the results.
It is clear that our models outperform the baselines, MinTL and DAMD, showing the efficacy of our method.
Comparing with Table 1, our models trained with 20% of the data perform competitively with the baseline methods trained on the full training set.
5. A BLATION S TUDY
The ablation study considers the following four research questions to better understand our methods.
a
): What if we learn the reward functions via different number of trajectories?
In Fig.
2a and 2b, we vary the number of trajectories used for the reward-learning losses in Table 1.
To avoid unwanted interference, we use the basic version of models without the GS component.
The case of using two trajectories reduces to the pairwise-ranking loss discussed in Section 2.
As shown in Fig.
2a and 2b, the generalized approach of using multiple trajectories to learn the reward function provides the flexibility to outperform the classical pairwise preference.
This is more apparent in the RewardMLE models, which are less sensitive to the small errors in the ground-truth scores.
In general, we hypothesize that the optimal trajectory number depends on the scoring quality.
(b): Do different probabilistic transforms in
reward learning objectives affect the performance?
We modify the basic version of the RewardNet and RewardMLE models in Table 1 by using the softmax transform and by using different powers in the escort transform in the reward learning losses
Eqs.
(2) and (3).
For the escort transform, we consider Φ =
( · ) p , p ∈ { 1 , 2 , 3 , 4 } .
Figs. 2c and 2d plot the resulting Combined Scores.
We see that the RewardMLE model is less sensitive to the choice of probabilistic transform — all the considered variants have Combined Score at least 104 .
In fact, changing its softmax transform used in Table 1 to the escort transform with power two improves the performance to 106 .
Thus, the choice of probabilistic transform provides an additional angle to improve the learned reward function and the entire TOD model.
(c)
: Is our method sensitive to the coefficient α in the generation-model loss Eq.
[Figure]
To investigate the robustness of our method under different weights for the policy-gradient optimization of the response-generation model.
We select our best policy-gradient-based model in Table 1, the RewardNet + GS model, and vary the α coefficient in the generation-model loss Eq.
Fig.plots the resulting four automatic evaluation metrics.
We see that our model is relatively robust to the choice of α .
The five variants in Fig.
3 all have
Combined Scores at least 105 , higher than the best baseline result of 104 . 78 in Table 1.
In fact, by changing the α coefficient to 0 .
5 from 0 .
1 used in Table 1, we achieve a even better Combined Score of ≈ 107 . 2 .
Further, the capability of task-completion and the fluency of the generated responses are both relatively insensitive to the choice of α .
(d): How does the addition of the policy gradient method Gumbel-softmax help the performance?
Fig.
4 compares the performance of our models in Table 1, with error bar showing the standard deviation of the Combined Score over five seeds.
It is clear that the addition of the Gumbel-softmax method can not only improve the score, but also reduce the performance variation, which is apparent when comparing the RewardMLE model with the RewardMLE + GS model.
As discussed in Section 3.2, the Gumbel-softmax (GS) trick is more advantageous than the classical
REINFORCE method (Williams, 1992) for policy-gradient update.
As a demonstration, we conduct a toy experiment following Yin et al.
and plot the results in Fig.
The task here is to learn the parameter ψ of a D -dimensional categorical distribution to maximize a simple reward function.
Specifically, denote the sigmoid function as σ ( · ) , the goal is
[Equation]
where Cate( σ ( ψ )) denotes the categorical distribution with probability vector σ ( ψ ) , and D = R = 30 .
The optimal σ ( ψ ) is clearly (0 , . .
. , 0 , 1) , leading to the optimal expected reward ≈ 0 .
We initialize ψ = 0 and use one sample for stochastic gradient-ascent update, with learning rate 1 . 0 .
The first row of Fig.
5 traces the objective function during the training process when using the true gradient, REINFORCE, and the GS for policy-gradient updates.
We see that the REINFORCE method converges to a local maximum, while the GS method reaches to the global optimum, as using the true gradient for updates.
The second row shows the gradients for θ 1 and θ D , where we see that gradient estimates from the REINFORCE method is both unstable and vanishing, compared to the
GS method.
The learned probabilities { σ ( ψ ) 1 , . . .
, σ ( ψ ) D } is traced in the third row, where the red line is for σ ( ψ ) D that should ideally be 1 , and the shadowed lines are for the other components that ought to be 0 .
The learning process of the GS method closely resembles that of using the true gradient, while REINFORCE oscillates around a local minimum.
The last row of Fig.
5 plots the estimate of gradient variance via 500 samples, averaged over each component of the ψ vector.
The gradient variance of the REINFORCE method is on the order of 10 − 2 at the begining and converges to roughly 10 − 4 , while the GS is 10 − 6 throughout the training process.
This toy experiment shows that a low-variance method, such as the GS, is critical to the success of the policy-gradient training.
5. F URTHER A NALYSIS
Human evaluation.
For a more comprehensive evaluation of our method, we conduct a human evaluation on the quality of the generated responses, where our model and the top-two baselines in Table 1, GPT-Critic and CASPI, are compared.
We follow the evaluation protocol in prior work ( e.g. , Zhang et al., 2020;
Ramachandran et al., 2021; Jang et al., 2022) to evaluate on two metrics: 1)
Appropriateness : measures the appropriateness of the generated response under the context of the dialogue turn; 2) Fluency : evaluates the comprehensibility and coherency of the generated response.
We randomly pick 50 turns in the test set and show to 10 evaluators the responses generated from
[Figure] [Figure]
[Figure] [Figure]
each method, together with the dialogue history up to that turn.
The method names are anonymized.
The evaluators were asked to read the dialogue history and score the response on a 5 -Point Likert
Scale { 1 , 2 , 3 , 4 , 5 } , where the score 5 is the highest and 1 the lowest.
Fig.
6 summarizes the evaluation results.
We see that our model outperforms the baselines on both the appropriateness and fluency scores.
The human-evaluation results coincide with our comparatively good dialogue-task completion and BLEU score in Table 1.
Examples of the generated dialogues.
Tables 3 and 4 in Appendix A conduct two case study comparing the generated responses from our method with those from the baselines GPT-Critic and
CASPI.
We additionally annotate the generated responses to discuss the quality of the generations.
These examples show that the responses from our model compare favorably with the baselines on both task completion and the comprehensibility, aligning with the automatic and human evaluations.
Quality of the DST.
To further understand the performance gain of our models, we compare our basic RewardNet and RewardMLE model in Table 1 with the baselines UBAR and GPT-Critic on the quality of the generated dialogue states.
Fig.
7 plots the results on the dialogue state prediction, measured by the two metrics Joint (Goal) Accuracy and Slot Accuracy (Wu et al., 2019a).
We see that our two models have more accurate DST than the two baselines, which can be related to their better performance in Table 1.
Interestingly, the DST of the RewardMLE model is also better than that of the RewardNet model.
This may suggest that a better reward model not only benefits the learning of response generation, but also the DST, since these two losses are jointly minimized in training the TOD model, and thus a good response-generation loss from a better reward model helps the optimization of the DST loss.
C ONCLUSION
In this paper, we aim at answering the question of how to efficiently learn and utilize a reward function for training end-to-end TOD agents.
We answer the question by introducing two generalized reward-learning objectives, and use a stable policy-gradient method to guide the training of the end-to-end TOD agents.
Future work includes extending our reward-learning objectives to other applications, such as the question and answering with verification.
E THICAL S TATEMENT
We developed our methods based on the publicly available MultiWOZ 2.0 dataset (Budzianowski et al., 2018).
It is important to note that, like other task-oriented dialog models, our implementation will likely reflect the socio-economic and entity biases inherent in the MultiWOZ dataset (Qian et al., 2021).
We initialized model parameters from the pre-trained BART model, the comprehensive analysis of certain biases captured by BART is outside our scope.
Besides, we invited volunteers for human evaluation with transparent and detailed explanations and communications about data use, research intent, occupied hours, etc.
Appendix A E XAMPLES OF THE GENERATED DIALOGUES
Tables 3 and 4 show two case study comparing the generated responses from our method and from the baselines GPT-Critic and CASPI.
Our method outperforms the baselines in terms of both task completion and the quality of the generated expression.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
please
[Table]
B A LGORITHMIC D ETAILS
Prepossessing.
The raw corpus is prepossessed by common practice in the TOD literature.
Specifically, we represent the database (DB) query results as one-hot vectors following Budzianowski et al.
(2018), use domain-adaptive delexicalization proposed by Wen et al.
and generate delexicalized responses with placeholders for specific DST/DB information as in Zhang et al.
Implementation of the response model.
Our model is based on the MinTL TOD model (Lin et al., 2020), which uses the pre-trained BART-large model (Lewis et al., 2019).
MinTL directly works on the system response and does not explicitly output the dialogue act.
Our proposed method in Section 3 is applied to the response training, and we retain the DST-training loss in MinTL.
Our model is trained by fine-tuning BART on the training set and early stop by the validation set.
Implementation of the reward model.
Our reward model is implemented by the encoder part of the BART-base model, followed by a simple two-layer MLP.
The output of the reward model is scaled to [0 , 1] via a sigmoid function.
The input to the reward model is the concatenation of belief state, system response, and dialogue goal, at each turn of the sampled dialogue rollout.
The model outputs the reward of each turn in the rollout, which is summed and fed into the losses proposed in
Section 3.1.
We use the HuggingFace library (Wolf et al., 2019) to implement our reward model.
Algorithm 1 provides the pipeline of our methods.
Input: Reward function R θ
( o, a, g ) , TOD agent π ϕ ,
[Table]
ˆ D :=
(cid:110)(cid:16) g ( k
[Table]
, ( o (
[Table]
t , a (
[Table]
t
[Table]
Tt =
(cid:17)(cid:111) K k
=1 , number of iterations M
[Table]
and M 2 , probabilistic transform function Φ , hyperparameters N , α .
C C OMPARISON WITH SOME OTHER REWARD - LEARNING METHODS IN RL- BASED DIALOGUE AGENTS
As an additional discussion on related work, in this section we briefly compare our work with three other reward-learning methods in RL-based Dialogue agents, i.e. , Saito (2018), Hu et al.
(2018), and
Li et al.
These three papers all focus on the dialogue-management module of the pipeline design, wherein the action-spaces of the agents are some abstract dialogue acts rather than the human-language-like system response as in our paper.
The possible system responses are fixed in these three papers.
For example, Hu et al.
(2018) trains the agent to select from “the set of the indices to all available questions in the Q20 game”; and Li et al.
(2020) has an action space of size 300 .
As discussed in Section 1, such a pipeline design requires intensive structural designs, such as determining the possible questions in the Q20 games; and may not enjoy the language plurality and conversation elicitation that our end-to-end model could offer, e.g. , in Table 4.
Due to the complexity and much higher dimension of the system responses as our action space, the methods proposed in these three papers are not directly applicable to our setting, which will be discussed in details below.
The proposed method in Saito (2018) requires specially-designed curriculum data and hand-crafted decomposition of the entire task into sub-tasks, which are not readily available and non-trival for large-scale and multi-domain dialogue corpus such as our tested MultiWOZ 2.0 dataset.
The use of progressive neural networks to provide reward information in Saito (2018) requires additional computation and memory complexity and thus may not scale to transformers.
Meanwhile, our method scales well to transformers, as shown in our experiments (Section 5).
Further, the method in Saito (2018) may only be feasible on constrained information-retrieval, but not on some more general tasks such as booking requirement in the tested MultiWOZ 2.0 dataset.
Our experimental results show that our method is capable of such tasks.
Similar to our work, Hu et al.
(2018) proposes a neural network to approximate the reward function to deliver immediate rewards at each time-step.
Apart from the aforementioned simple action-space,
Hu et al.
(2018) uses the long-term return G t as a surrogate indicator of r t +1 to train the reward function (Eq.
(6) of Hu et al.
(2018)), which is lack of justification.
By contrast, as discussed in
Section 2 and Section 3, our method is based on the classical approach in the learning-to-rank (LTR) literature and extends the classical reward-learning-from-preferences in RL into utilizing multiple dialogue trajectories simultaneously to optimize the reward function.
As discussed before, Li et al.
(2020) considers a relatively-small action-space of size 300 and learns the reward model via the GAN structure, which may not stably scale up to high-dimensional actionspace such as the system response in our end-to-end TOD system.
The learned reward-function in Li et al.
(2020) only measures the probability that the input is from the real-data distribution, i.e. , only considers a pair of dialogue state s t and the corresponding system action a t .
This reward function does not consider the success of the entire dialogue, which is intuitively less favorable to end-to-end
TOD systems.
By contrast, our method trains a reward function that aligns with some evaluations on the entire dialogue trajectories, which is more directly related to the usage of the TOD systems.
We further note that apart from the reward-learning method, our paper also proposes using the
Gumbel-softmax trick as a more stable method to train the end-to-end TOD systems, and conduct a toy experiment in Section 5.2 to illustrate the advantage of the Gumbel-softmax trick over the classical REINFORCE method.
This is not covered in the three prior works Saito (2018), Hu et al.
(2018), and Li et al.
Finally, these three prior works use online RL methods such as DQN, REINFORCE, and PPO, which require environmental interactions and are thus less practical, as discussed in Section 1.
By contrast, our method allows training end-to-end response-generation models from static datasets.
D D ETAILED COMPARISON WITH CASPI
{+Table 5 further compares CASPI and our two methods: RewardNet +GS, N = 3 , Φ = ( · ) 
+}
and
RewardMLE +GS, N = 5 , Φ = exp( · ) , showing a detailed breakdown of the scores onto each of the five tested random seeds.
The performances of both our methods and CASPI are relatively stable across random seeds.
In particular, both our methods have higher Combined Score than CASPI on each of the five tested random-seeds .
This stable improvement of our methods over CASPI aligns with our intuition discussed in Section 3 and our main experimental results in Section 5.1.
We note that both the Average and the Median score across the tested random-seeds are valid metric for performance comparison.
However, there are some ambiguity in calculating the “Median” of the
Combined Score, namely, should it be the median of the Combined Scores on each random seeds, or should it be calculated as Median(Combined Score) ≜ (Median(Inform) + Median(Success))
× 0 . 5+Median(BLEU) ?
The first way aligns better with the definition of “Median” while the second way aligns better with the definition of Combined Score.
Such an ambiguity is cleared out when using the Average as the evaluation metric.
Besides, the metric Average score is widely used in prior work ( e.g. , Zhang et al., 2020; Lin et al., 2020; Jang et al., 2022).
With these considerations, we choose to report the Average over five random-seeds in our experimental section (Section 5).
[Table]
E E XPERIMENTS WITH THE GALAXY
To further demonstrate the efficacy and applicability of our approach
[Table]
we apply our reward learning and utilization methods onto the recently proposed GALAXY method (He et al., 2022), which achieves competitive performance on the end-to-end response generation task on the MultiWOZ 2.
dataset.
We note that the GALAXY paper does not disclose how many and which random seeds were used to obtain its main results; and the official codebase fixes the random seed as 10 .
This leaves us a doubt on if its reported results are only from this single seed 10 .
To mitigate the randomness in the optimization process, we re-run the vanilla GALAXY on the five random-seeds used to generate our main results and compare it on these seeds with the variants equipped with our methods
RewardNet +GS, N = 3 , Φ = ( · ) 1 and RewardMLE +GS, N = 5 , Φ = exp( · ) .
Table 6 shows a detailed breakdown of the scores onto each of the five random-seeds.
Note that since we use different random seeds than the original GALAXY paper, we are unable to get its reported scores.
We see from Table 6 that adding our reward learning and utilization methods improves the performance of the vanilla GALAXY, in almost all evaluations metrics, in both the Average and the Median scores.
In particular, adding our RewardMLE +GS method improves the average Combined
Score of the vanilla GALAXY by 3 . 27 , and adding our RewardNet +GS improves the vanilla
GALAXY by 2 . 11 .
We believe these relatively significant improvements can further demonstrate the effectiveness and general applicability of our proposed methods.
[Table]
F E XPERIMENTS ON THE M ULTI WOZ 2.1 DATASET
To test the efficacy of our proposed methods on additional datasets, we run our two methods
RewardNet + GS, N = 3 , Φ = ( · ) 1 and RewardMLE + GS, N = 5 , Φ = exp( · ) on the MultiWOZ 2.1 dataset.
Table 7 compares our two methods with the baselines SimpleTOD and UBAR in the main evaluation (Table 1) that also report results on the MultiWOZ 2.1 dataset.
Additionally, we also present our rerun of CASPI on this dataset.
As in Table 1, the Combined Scores of our methods are generally better than the baselines.
In fact, our methods achieve both good task-completion (Inform and Success rates) and fluent generated responses (BLEU score)
[Table]
Table 8 shows a detailed breakdown of the scores of CASPI and our two methods onto each of the five tested random seeds.
We see that the scores of our methods are generally robust across seeds.
In fact, on four out of the five seeds, at least one of our method performs better than CASPI.
Further, our methods have higher Average and Median scores than CASPI on each of the four evaluation metrics.
This set of experiments can further corroborate the efficacy of our proposed methods.
[Table]
Yang et al., 2021)
[Table]
(Ramachandran et al., 2021)
[Table]
[Table]
107.11 Median 91.
G I MPLEMENTATION DETAILS
Our implementation is based on the official codebase of MinTL and CASPI.
Apart from the hyperparameters in our method discussed in Section 5.2, most other hyperparameters and the training procedure of our models follow MinTL and CASPI.
In addition to the discussion of our method in Section
[Table]
we list the important hyperparameters for training our reward model in Table
[Table]
and the important hyperparameters for training our response generation model in Table 10.
Both Bart models uses the default token length 142.
We note that unlike CASPI that uses the dialogue acts as the actions, the action space of our reward model is the system response, which is combinatorial space of the vocabulary.
We use this action space because we want to pass gradients from the reward model to the end-to-end responsegeneration model during the training time, where the output of the end-to-end model is the humanlanguage-like system response.
For the reward-function learning
[Table]
we do not change the length of the trajectories in the dataset.
The reward model is updated by the preference scores/orderings among multiple trajectories of the same length.
Our intuition is that trajectories of the same length may roughly correspond to tasks of similar complexity, making the preference among them more comparable and meaningful.
This approach is also taken by the prior work CASPI.
Our tested MultiWOZ2.0 dataset is publicly available here
[Table]
and the MultiWOZ2.1 dataset is available here
[Table]
[Table]
[Figure]