A M IXTURE - OF -E XPERT A PPROACH TO RL- BASED D IALOGUE M ANAGEMENT
Dhawal Gupta
Moonkyung Ryu Mohammad Ghavamzadeh Craig Boutilier {yinlamchow, atulep, ofirnachum, dhawgupta, mkryu, ghavamza, cboutilier}@google.com
A BSTRACT
Despite recent advancements in language models (LMs), their application to dialogue management (DM) problems and ability to carry on rich conversations remain a challenge.
We use reinforcement learning (RL) to develop a dialogue agent that avoids being short-sighted (outputting generic utterances) and maximizes overall user satisfaction.
Most existing RL approaches to DM train the agent at the word-level, and thus, have to deal with a combinatorially complex action space even for a medium-size vocabulary.
As a result, they struggle to produce a successful and engaging dialogue even if they are warm-started with a pre-trained
LM.
To address this issue, we develop a RL-based DM using a novel mixture of expert language model (MoE-LM) that consists of (i) a LM capable of learning diverse semantics for conversation histories, (ii) a number of specialized LMs (or experts ) capable of generating utterances corresponding to a particular attribute or personality, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts.
Our MoE approach provides greater ﬂexibility to generate sensible utterances with different intents and allows RL to focus on conversational-level DM.
We compare it with SOTA baselines on open-domain dialogues and demonstrate its effectiveness both in terms of the diversity and sensibility of the generated utterances and the overall DM performance.
I NTRODUCTION
With the tremendous advancements in natural language understanding and generation, increasing attention has been directed to construct intelligent dialogue agents that can carry out engaging conversations with users.
Such interactions can be open-ended, contain different topics, and often involve an underlying task, such as negotiation, information exchange, and recommendation.
Therefore, to satisfy the user, a good dialogue agent should not only generate natural responses, but also be capable of pursuing the task’s objectives and adapting to the user’s feedback on-the-ﬂy.
A standard solution is to train the dialogue agent using behavioral cloning, where the agent is a language model (LM) that imitates the utterances in the training set (Ga š i´c et al., 2011; Fatemi et al., 2016).
By leveraging deep neural networks, e.g., RNNs (Sutskever et al., 2014) and Transformers (Vaswani et al., 2017), a LM encodes the conversation to a low-dimensional dialogue state and predicts an utterance, but steering such generation for particular purposes remains an open question.
Several works studied ways to ﬁne-tune a LM to generate texts with speciﬁc contexts (Ziegler et al., 2019; Ficler and Goldberg, 2017).
Other results learned a single steerable LM that is capable of generating utterances for multiple speciﬁc intents (Gu et al., 2017; Chen et al., 2018; Subramani et al., 2019; Dathathri et al., 2019).
While these LMs produce ﬂuent and relevant responses, it is unclear how to control them to systematically pursue goals during multi-turn dialogue conversations.
Another popular approach is to view dialogue management (DM) as a control problem and use reinforcement learning (RL) to optimize the agent’s policy (which is often a LM itself).
Using
RL for dialogue systems has a long history.
Earlier work relies on speciﬁc, hand-crafted semantic states (Levin and Pieraccini, 1997; Singh et al., 2002; Walker, 2000) or partially observable belief states (Williams and Young, 2007; Young et al., 2010), in which the agent chooses the best handcrafted dialogue act at each turn, with the goal of either satisfying the user (Shah et al., 2018), completing the task (Shi and Yu, 2018), or responding to the user’s query (Serban et al., 2017a).
However, the application of these approaches is limited to problems whose action space can be captured by hand-crafted representations, and they cannot handle complex conversations.
On the other hand, more recent approaches use deep learning to extract semantic representations from conversation histories, treat these representations as dialogue belief states, and apply RL to learn a word-level generative DM agent (Jaques et al., 2019; Li et al., 2016; 2017; Shin et al., 2020).
However, since there are innumerable possibilities of language utterances, and thus, the action space of the RL problem is extremely large, the agent often performs planning poorly and generates incomprehensible utterances (Zhao et al., 2019).
Another issue is that RL only optimizes a scalar reward, while the aforementioned methods often need to optimize for both the quality of the generated utterance, e.g., ease of answering (Li et al., 2016), ﬂuency (Li et al., 2017; 2019), and diversity (Yarats and
Lewis, 2018), and the goal, e.g., conversation length (Zhou et al., 2020), user’s sentiment (Hancock et al., 2019), and task completion (Verma et al., 2022; Jang et al., 2021).
Moreover, deﬁning the reward as weighted combination of these metrics is not ideal, since the hand-picked weights do not often reﬂect the underlying success criteria.
To address the above issues related to using RL in dialogue management (DM) systems, we propose an RL-based DM agent using a novel mixture of expert (MoE) approach.
Our MoE approach is based on a mixture of expert language model (MoE-LM), which consists of three main components: 1) a
LM (a probabilistic encoder and a decoder) capable of learning diverse semantics for conversation histories, and as a result generating diverse utterances, which we refer to as the primitive LM or LM 0 , 2) a number of specialized LMs (or experts ), { LM i } mi =1 , that each is constructed using the latent space learned by LM 0 , but has been trained such that it is capable of generating utterances corresponding to a certain intent or personality, and 3) an RL-based dialogue manager (DM) that at each turn, given the latent state shared by the experts { LM i } mi =0 and the utterance action(s) they suggest, chooses one among them for the agent to execute.
Our MoE-LM can be seen as a special case of hierarchical
LMs (e.g., Serban et al.
2017a;
Zhao et al. 2019;
Saleh et al. 2020), but it is different than them because it learns both the LMs (experts) and the DM.
Moreover, the DM in MoE-LM is a policy conditioned on both the latent state and the actions suggested by the experts, and not just the state as it is common in hierarchical RL.
The primitive LM ( LM 0 ) plays an important role in this model because it learns diverse semantics for conversation histories and allows the agent to generate a wide variety of utterances.
This diversity is also shared with the specialized LMs (experts) and gives them ﬂexibility in generating their (more) specialized utterances.
Another important feature of MoE-LM is its modularity that facilitates adding and removing specialized LMs (experts).
Moreover, this hierarchical architecture allows us to solve an RL problem with much smaller state and action spaces, which is quite important in the quality of the learned policy.
Finally, since the candidate utterances are generated by experts with different intents, instead of combining all agent-user signals into a single RL reward, our DM agent can focus on optimizing the speciﬁc goal of the conversation task.
We start the paper with a brief introduction of LMs and the use of Markov decision processes (MDPs) in modeling dialogue management problems in Section 2.
We then describe the overall architecture of our MoE-LM in Section 3, followed by the detailed implementation of each of its three main components (described in the above paragraph) in Sections 4 to 6.
Finally, in Section 7, we demonstrate the effectiveness of our MoE-LM in open-domain dialogues, in terms of both its ability to generate diverse and sensible utterances and its overall DM performance.
P RELIMINARIES
Language Models (LMs)
In this work, we employ seq2seq LMs to generate the next utterances in a dialogue.
We assume access to a dataset of the form D
= { ( X ( k ) , Y ( k ) ) } |D| k =1 , where each
X =
X ( k ) is a L -turn conversation history
X
= { X l } L  1 l =
and Y is its next utterance.
We denote by N X , an upper-bound on the length (number of tokens) of each utterance X l in X .
The role of a LM is to predict the probability of the next utterance Y , consisting of N tokens, conditioned on the conversation history X , i.e., p
Y = { [Equation]  .
In the transformer architecture ( ? ), the
LM ﬁrst encodes the conversation history X using an encoder  to a ( L ⇥ N X ) -length sequence of embeddings { ( z l, 0 , . . .
, z l,N X  1 ) } L  1 l =0 , where each z l,n is a vector in the latent space.
For notational convenience, we concatenate these embeddings into a single embedding z 2 Z ✓ R d and denote the overall dimension of the latent space as d .
In the RNN architecture (Serban et al., 2016), the
LM’s encoder  directly maps the conversation history X to a latent state z 2 Z ✓ R d .
In both architectures, the next utterance b [Equation]
y n } Nn =1 is sampled token-by-token from the decoder , [Table] [Table] [Table]
> µ
, b
[Equation]
= [Equation]
b
[Equation]
b
[Equation]
; z  , where b y 0 is a ﬁxed initial (start-of-sentence) token (Chien and Kuo, 2019), and the latent state is denoted as z =  ( X ) .
Markov Decision Processes (MDPs) have been used to model dialogue management problems in a variety of settings (Li et al., 2016; Asadi and Williams, 2016; Jaques et al., 2019).
In such
MDPs, denoted by M = ( S , A , P, r, s 0 ,  ) , the state space S represents the tokenized conversation history and the initial state s2 S is the initial user’s query.
The action space A is also the tokenized language space with each action a 2 A being the agent’s next utterance (which is a ﬁxed-length, N X , sequence of tokens).
The transition kernel P models the user’s response to the action taken by the agent (bot).
Finally, the reward function r measures the user’s satisfaction.
In these MDPs, we can think of the entire LM as a policy that maps conversation histories to next utterances, and solve them by ﬁnding a policy ⇡ ⇤ with maximum expected discounted return, i.e., ⇡ ⇤ 2 arg max ⇡ J ⇡ :=
E [ P 1 k {+=
 +} t r t | P, s 0 , ⇡ ] .
Note that the size of the tokenized state and action spaces grow exponentially with the size of the vocabulary.
This makes it intractable to solve the MDP even for a medium-size vocabulary.
As a result, it would quite desirable to develop a novel
MDP paradigm that is more amendable to RL-based DM systems.
M IXTURE OF E XPERTS (M O E) L ANGUAGE M ODEL
We start by explaining how a MoE language model (MoE-LM) can enrich the bot’s utterances and improve the overall performance of the DM.
While our approach is applicable to any DM system, we use open-domain dialogue (Sankar et al., 2019) as a running example to show how
MoE-LM-based agents can improve user satisfaction measured by an improvement on a sentiment or engagement.
Intuitively a good DM agent should possess different behaviors (e.g., inquisitive, explorative, relevant, soothing, empathetic, complimentary, provoking) and swiftly decide which intent to use to pivot a conversation, build rapport, pique the user’s interests, improve their mood, etc.
To achieve this goal, we require the LM to have a language representation (primitive discovery) that captures different semantics, in order to encode different conversations and avoid generating dull and repetitive responses.
We also need a machinery (expert construction) to embed different intents into sub-models of this LM, so that they can behave accordingly when prompted, and respond efﬁciently.
Finally, with various candidate utterances available, the DM module of this LM should understand the current level of user satisfaction and determine which response is the most appropriate.
Motivated by these observations, we construct our MoE-LM in three steps as shown in Figure 1.
We give the main idea behind each step here and leave their detailed descriptions to Sections 4, 5, and 6.
Step 1: Primitive Discovery.
We ﬁrst employ the dataset D , introduced in Section 2, and learn a language model LM
= (  , G 0 , ) consisting of a stochastic encoder (i.e., an encoder  and a latent space distribution
G 0 that maps the encoded conversation into a latent distribution), and a decoder .
The stochastic encoder (  , G 0 ) comprises an encoder  that maps tokenized conversation histories X to a latent space Z ✓ R d , i.e., z =  ( X ) 2 Z , which is then used to construct a parameterized d - dimensional [Table] distribution[-G 
-]
[Equation] ( z
[Equation]
z )
[Equation]
over R d .
The decoder predicts the next [Table] [Equation]
Y 0 (token-by-token) conditioned on the point z 0 sampled from the latent distribution, i.e., [Equation]
Y[-
-]
[Equation]
) 3 , z
[Equation] ⇠ [Equation] ( ·| z ) .
We denote by LM
( Y | X ) :
= E z {+
+}
⇠ G 0 ( ·| z ) ,z =  ( X ) [ ( Y | z 0 )] , the primitive and learn it using a loss function that in addition to predicting the next utterance accurately, encourages diversity and generalization in the learned latent space Z (see Eq. 1 and Fig. 2).
As we will explain in Section 4, our loss function is inspired by those in prior work, and more speciﬁcally by the one in OPAL (Ajay et al., 2020), i.e., an unsupervised learning method for discovering primitive skills in trajectories that are used by some downstream RL tasks.
Step 2: Expert Construction.
Given the latent space Z , encoder (  , G 0 ) , and decoder learned in Step 1, we now learn m latent distributions {
[Equation]
=1 , each deﬁned as G i ( z [Equation] | z
[Equation]
=
[Equation]
. Intuitively, each G i corresponds to an attribute, e.g., an intent or a personality (in case of a chatbot) and generates samples in speciﬁc parts of the latent space
Z .
This results in having m LMs, { LM i } mi =1 , LM i =
(  , G i , ) , each of them corresponds to a specialized version of the original LM, LM 0 , and serves as an expert in our MoE-LM.
Upon receiving a conversation history
X , each expert LM i generates a candidate (or more) for the next utterance b
Y i in certain parts of the language space that are compatible with its attribute (personality).
As we will explain in Section 5, each G i is learned using a loss function that encourages its corresponding LM, LM i , to generate utterances consistent with its attribute (see Eq. 2).
Step 3: Dialogue Manager (DM).
The dialogue manager , denoted by µ , takes as input the encoded conversation history z
=  ( X ) and the candidate action utterances generated by the experts { b
Y i } mi =0 , and selects one of them as the action for the bot to execute, i.e.,
[Equation]
[Equation] =0 ) .
We will describe how DM is trained using reinforcement learning (RL) in Section 6.
P RIMITIVE D ISCOVERY IN M O E-LM
Motivated by literature in the reinforcement and imitation learning ﬁelds (Ajay et al., 2020), we propose to learn the primitive LM, LM 0 , in our MoE-LM by solving the following KL-constrained optimization problem that aims at capturing diverse semantics:
[Equation]
where b
E is the empirical expectation over ( X , Y ) in the dataset D , ⇢ is a distribution over the latent space conditioned on the encoded conversation history z and the target utterance Y , and ✏ KL is a positive real-valued threshold.
Using (1), we learn LM
= (  , G 0 , ) by maximizing the log-likelihood, while enforcing consistency between the latent variable z 0 predicted by G 0 ( ·| z ) and ⇢ ( ·| z, Y ) via the KL constraint.
The distribution ⇢ ( ·| z, Y ) is a Gaussian N
[Equation]
,  ⇢ ( Y )) I d ⇥ d  in which  ⇢ is a pre-trained encoder for the target utterance Y , and mean µ ⇢ ( · , · ) and variance  2 ⇢ ( · , · ) are trainable models.
One reason for using a separate encoder  ⇢ for the target utterance Y is to avoid overﬁtting  (i.e., to avoid having back-propagation gradient of  with Y as input).
Connection to VAE-like objectives In practice, we implement the KL constraint in (1) as a penalty weighted by an appropriately chosen coefﬁcient.
Thus, one may interpret the objective in (1) as a variation of  -VAE (Burgess et al., 2018).
Due to the connection to VAEs, one may draw similarities between our method and existing dialogue approaches such as VHRED (Serban et al., 2017b) and
VHCR (Park et al., 2018).
However, we emphasize that there are key differences, and these may be explained by ﬁrst understanding how the objective in (1) encourages diversity , which is key to good primitive learning.
Namely, it is important that primitive discovery learns an encoder-decoder  , which can be modulated by the choice of z ; i.e., changing z 0 while ﬁxing X should lead to different distributions over generated utterances.
The objective in (1) encourages this diversity by conditioning the latent variable z 0 on both the target utterance Y and z =  ( X ) , i.e., z 0 ⇠ ⇢ ( ·| z, Y ) .
In contrast, the KL constraint is used to make sure that the stochastic encoder
G 0 ( ·| z ) of our primitive LM is not too varied for different Y , and thus has a limiting effect on diversity.
For example, in the extreme when ✏ KL = 0 (or,  ! 1 when used as a regularizer) there will be no specialization of the latent space for different Y .
Although  ! 1 is an extreme case, degenerate behavior can also happen when  = 1 , i.e., in the traditional variational loss used by VHRED and VHCR.
Speciﬁcally, it is well-known that the traditional VAE loss is an upper bound on the negative log-likelihood of the data under a stochastic encoder-decoder parameterization.
Thus if the data can be modeled by a single
LM then a VAE-optimal decoder can simply ignore G 0 , leading to a degenerated latent space as observed in previous work (Park et al., 2018).
This is precisely the reason that, in our approach, we weaken the KL constraint ( ✏ KL  0 or, equivalently,  ⌧ 1 ).
This enables our approach to more reliably guarantee that a unique z 0 represents each distinct conversation pair ( X , Y ) , thus capturing diverse semantic modalities and enabling easier downstream specialization.
In the mathematical results below, we formalize the claim above, namely, that the log-likelihood objective in (1) leads to a learned  , that can easily recover any arbitrary desired LM by specializing the latent space G .
We begin with a deﬁnition that characterizes the coverage of an arbitrary LM on the conditional conversation data distribution P
[Equation]
Deﬁnition 1.
LM D , ⇠ is a ⇠ -common LM of data D if E D [ TV ( LM D , ⇠ ( Y | X )
||
P D ( Y | X )))]  ⇠ .
Leveraging Theorem 4.1 in Ajay et al.
(2020), we now present the theoretical result characterizing the representational power of our primitive encoder-decoder pair (  , ) on data D .
Lemma 1.
Let (  , ⇢ , ) be the solution to (1) with
[Equation]
Then there exists LM :
= (  , G , ) such that E D [ TV ( LM D , ⇠ ( Y | X ) || LM ( Y | X ))]
 ⇠ + q 12 ( ✏ + H ) , where
G ( z 0 | z ) =
E Y ⇠ D [ ⇢ ( z 0 | z, Y )] , and H =
E D [log P D ( Y | X )] is a constant depending on D .
The result above shows that, as long as LM D , ⇠ is ⇠ -common in D , then there exists a specialization of the latent space
G that, when paired with  , , can approximately recover LM D , ⇠ .
The quality of the approximation is a function of ✏ — how well the objective in (1) was optimized — and ⇠ .
In practice, to construct the primitive by replacing G with G 0 , i.e., LM {+
+}
= (  , G 0 , ) , because G 0 ( z 0 | z ) can be viewed as an distillation of ⇢ ( z| z, Y ) .
This theoretical result also motivates the next section, where we explain our algorithm’s
“Step 2: Expert Construction”.
Speciﬁcally, we show how to use the trained encoder-decoder pair  , to learn a spectrum of different specialized experts parameterized by different latent distributions G i .
E XPERT C ONSTRUCTION WITH P LUG - AND - PLAY L ANGUAGE M ODELS
To complete the MoE framework one needs to systematically create a gamut of different experts
LM i , 8 i 2 { 1 , . . .
, m } , with each generating candidate utterances of different intents.
By viewing each expert as a distribution of particular behaviors in conversation data
D , we leverage the results of Section 4 and Lemma 1 and adopt a universal encoder-decoder (  , ) among all the experts.
Therefore, each expert i is only parameterized by an arbitrary d -dimensional latent distribution (e.g., Gaussian), and it samples certain regions of the latent space Z .
Following the terminology from Dathathri et al.
, these experts can all be catagorized as plug-and-play language models (PPLMs) .
Creating experts is handy because it only requires learning new latent distributions, while switching between experts amounts to sampling a different distribution.
Denote by ` i ( X , Y ) 2 R a real-valued label that characterizes the intent of expert i 2 { 1 , . . .
, m } , e.g., determined by an off-the-shelf sentiment classiﬁer.
We train the latent distribution
G i ( z ) of expert i by solving the optimization problem min
[Equation]
Unlike the weighted maximum likelihood approach considered in Dathathri et al.
(2019), which assigns weight ` i to training samples that correspond to expert i , we propose to learn each expert via reward-maximization and treat ` i as a reward signal w.r.t.
expert i to be maximized.
Interestingly, this approach is also linked to reinforcement learning (RL), in which both the “state” and “action” spaces are the latent space Z , and the “policy” is the latent distribution G i .
The main beneﬁt of our approach is that it does not require the target utterance Y from data
D and is thus less vulnerable to data-imbalance issues in
D on certain intents.
Notice from (2) that the reward-maximization problem is myopic, i.e., the above RL problem has a discounting factor of 0 .
The main motivation is that, unlike dialogue management that is a sequential decision-making problem, here we want each expert to possess particular behaviors, and this can readily be done via greedy maximization. Long-term dialogue optimization will be handled by the dialogue manager rather than these experts.
For example in the case of a Gaussian G i , we use the standard REINFORCE (Sutton et al., 1999) algorithm to learn the model parameters ( µ i ,  2 i ) of G i according to
[Equation]
where ↵ > 0 is the learning rate.
To reduce the variance of these estimates, we also adopt the baseline reduction technique (Greensmith et al., 2004) in policy gradient, which replaces ` i ( X , Y ) with ` i ( X , Y ) :
= ` i ( X , Y )  E Y ⇠ ( ·|  ( X ))
[ ` i ( X , Y )] .
Following arguments from Lemma 1 and Lemma 4.0.1 in Ajay et al.
(2020), in the following we quantify the sub-optimality of expert LM i .
Corollary 1.
Denote the i -th reward-maximizing objective as L i ( LM ) :
= [Equation]
Suppose an optimal LM for this objective LM i, ⇠ 2 arg max LM L i ( LM ) is ⇠ -common in D .
Moreover, let G ?
i be in the arg min of (2).
Then with expert LM i =
(  , G ?
i , ) and ( ✏ , H ) from Lemma 1, we have
[Equation]
While it may be obvious that optimizing G
i w.r.t.
(2) encourages expert LM i to capture the behaviors encouraged by ` i , this corollary has two further implications: (i) Since the sub-optimality of LM i compared to the oracle LM i, ⇠ is bounded by the quantity ✏ deﬁned in Lemma 1, it justiﬁes using the primitive ( ,  ) , which optimizes ✏ , for expert construction; (ii) Sub-optimality further depends on ⇠ , quantifying how well LM i, ⇠ is represented in the original dataset D .
R EINFORCEMENT L EARNING FOR M O E-LM D IALOGUE M ANAGER
We now describe the dialogue manager (DM) of our MoE-LM and propose RL algorithms to train it.
As mentioned in Section 3, the DM is a policy µ that takes the encoded conversation history
z
=  ( X ) and the m + 1 candidate action utterances generated by the experts { b
Y i } m i =0 , 4 and stochastically selects one of them to execute, i.e., b
Y ⇠ µ ( · | z, { b
Y i } mi =0 ) .
Note that each expert i 2 { 0 , . . .
, m } is a LM, LM i , that acts as a policy ⇡ i ( ·| X ) and maps each conversation history X to an utterance b
Y i .
With this architecture we address the large size of state and action spaces in the original MDP that grows exponentially with the size of the vocabulary.
As described in Section 2, the state and action spaces of the original MDP are the tokenized conversation history and the tokenized language space, respectively, while here the DM should choose among m + 1 actions (which is a much smaller and ﬁnite action space) given the latent space Z (which is a continuous state space) of encoded conversations.
It is important to note that our MoE-LM is different than other hierarchical architectures (Kulkarni et al., 2016) in which the decision at the high-level is to choose a low-level controller only based on the current state of the system.
In MoE-LM, the DM observes both the current state and the actions suggested by the experts and then chooses one among them.
We consider two RL settings to solve this specialized MDP.
The ﬁrst one is ofﬂine RL, in which the policy must be learned from the collected conversations D without further (online) interactions.
Ofﬂine RL requires optimizing a policy, while minimizing the deviation from the behavior policy to avoid errors due to data co-variate shift.
Among numerous ofﬂine RL algorithms (Kumar et al., 2020; Carta et al., 2021; Jaques et al., 2019), one effective algorithm to learn the DM policy µ is IQL (Kostrikov et al., 2021).
Given conversation data D , IQL ﬁrst computes the critic functions ( Q ✓ ⇤ ( z, a ) , V  ⇤ ( z )) via solving min ✓ ,  E ( z,a,r,z + ) 2 D [( r +  V  ( z + )  Q ✓ ( z, a )) 2 ]
+  · E ( z,a ) 2 D [ L ⌧ 2 (
[Equation]
[Equation]
))] , where z is the encoded conversation, a is the bot utterance, z
+ is the next encoded conversation, r is the conversation-level reward,  > 0 is a tunable weight, and L ⌧ 2 is the expectile regression operator (Koenker and Hallock, 2001) of estimating the top- ⌧ expectile statistics of the Q -function random variable (approximated by the value function V  ), and then extracts the DM policy µ via greediﬁcation over the ﬁnite set of MoE candidate utterances:
[Equation]
Intuitively, IQL leverages the generalization capacity of critic functions to estimate the value of the best action without directly querying the values w.r.t.
unseen actions.
This makes it less conservative than most ofﬂine RL methods that either constrain the policy’s actions to be in-distribution or solve a behavior-regularized policy optimization problem.
The second RL setting for learning the DM policy µ is via model-based RL (MBRL) (Shah et al., 2018; Wei et al., 2018).
While this paradigm can be applied to any online/ofﬂine RL algorithms we demonstrate it with the simple DQN (Mnih et al., 2013).
Here we ﬁrst learn a user utterance model P user ( X + | X , a ) :=
E z =
user ([ X ,a ])
[ user ( X + | z )] via maximum likelihood, then generate data D MB , whose next-state b s + encodes the next conversation generated from roll-outs and the corresponding candidate actions, solve the Bellman error minimization:
min
[Equation]
, and obtain the DM policy µ via the aforementioned greediﬁcation step.
The beneﬁt of MBRL over ofﬂine RL is two-fold.
First, one can easily obtain a high-ﬁdelity user utterance model (Peng et al., 2020) by simply ﬁne-tuning a large LM (e.g., GPT-3 (Floridi and Chiriatti, 2020)).
Second, with sufﬁcient dialogue roll-outs that captures many different scenarios, MBRL generally can be more data-efﬁcient and less prone to distributional shifts than ofﬂine RL.
E XPERIMENTS
We evaluate our MoE-approach on two open-domain benchmarks that are common within the RLbased dialogue management literature (Jaques et al., 2019).
The ﬁrst one is the Cornell Movie corpus (Danescu-Niculescu-Mizil and Lee, 2011), which consists of conversations between speakers in different movie lines and has a median conversation length of 3 utterances.
The second is the Reddit
Casual (Ghandeharioun et al., 2019) conversations, which is a subset of the Reddit corpus that only contains casual conversations on various topics of at least 3 turns and a median of 7 utterances.
We conduct several experiments to test the efﬁcacy of different parts in the MoE-LM, namely (i) the predictive power and diversity of the primitive, (ii) the quality of experts, and (iii) the overall DM performance.
For each metric, we report mean ± standard error over 100 conversations sampled from the evaluation set.
We also ran an ablation study on 4 transformer-based MoE-LMs, namely MoE-1,
MoE-2, MoE-3- MoE-4, to understand how performance is affected by different model architectures, language encoders, and latent generators.
MoE-1 and MoE-2 use a simpler architecture, while MoEand MoE-4 use the same encoder architecture as BERT (Devlin et al., 2018).
MoE-1 uses much smaller latent distribution models {G i } than MoE-2; MoE-3 uses the pre-trained BERT encoder, while MoE-4 trains that from scratch.
Details of these models can be found in Appendix B.3.
EXP 1: Comparing Primitive Models We compare the quality of latent representations learned by the 4 MoE-LMs (via optimizing Eq. 1) and 2 baselines (standard Transformer (Vaswani et al., 2017) and VHRED 5 (Serban et al., 2017b)).
To assess their quality, for each test conversation we generated 25 utterances and reported the following 3 metrics: (i) Diversity : The 1  Sparsity (Hurley and Rickard, 2009) of the singular values of the embedded utterances, i.e., Diversity ( { ˆ Y i } ) :
= 1  p d  k SVD k 1 / k SVD k 2 / p d  1 2 [0 , 1] , where SVD :
= SVD ( {  SE ( ˆ Y i } 25 i =1 ) , and  SE is a pre-trained sentence encoder (e.g., a USE (Cer et al., 2018)); (ii) Dist- { 1 , 2 , 3 } (Li et al., 2015):
Ratio of unique { 1 , 2 , 3 } -gram in the generated utterances; (iii) Perplexity : The perplexity score of the utterance w.r.t.
a GPT-2 LM, which is more correlated to human’s judgement on semantic ﬂuency (Pang et al., 2020).
These metrics measure both accuracy and semantic diversity.
Qualitatively, we also measure ﬂuency and diversity of LMs using human ratings (see Appendix B.8 for details).
The results of the above experiments are reported in Table 1 and 6 (Appendix A.1), and sample utterances generated by these LMs can be found in Appendix A.3.
Human evaluation on diversity/ﬂuency of different LMs are given in Table 4.
In comparisons with the baselines (Transformer and VHRED), generally (i) transformer-based LMs out-perform VHRED due to their attention mechanism that explicitly encodes sequential semantic information, and (ii) the MoE-LMs achieve way better diversity without sacriﬁcing much on accuracy (i.e., the perplexity scores are still quite low).
Qualitatively, the sample utterances generated the Transformer are closer to the targets than that by MoE-2 and MoE-4, likely because Transformer tends to memorize the corpus (Kharitonov et al., 2021).
Contrarily,
MoE-LMs generate utterances that have similar contexts with targets but paraphrased or similar structures but different contexts, demonstrating their generalizability.
Human evaluations also show that MoE-2 and MoE-4 generate more diverse utterances while retaining sufﬁcient semantic ﬂuency.
Among different MoE-LMs, MoE-2 and MoE-4 have the best performances, particularly MoE-4 has better diversity while MoE-2 has lower perplexity.
This corroborates with our hypotheses that (i) jointly training the encoder and decoder with Eq.
1 seems necessary to encourage semantic diversity (as opposed to using a pre-trained BERT encoder, which maximizes likelihood), (ii) sufﬁcient representation power is necessary for G 0 to match the posterior distribution ⇢ in order to capture different semantics in
D .
In Fig.
2a and 2b, we visualize the latent space of 400 conversation data samples for both Transformer and MoE-4.
The latent states of MoE-4 are much more dispersed across the embedding space, implying that most conversations get encoded uniquely.
In contrast, the latent space of Transformer has many clusters, suggesting it is more prone to generating similar utterances even with different input conversation and is thus less generalizable.
EXP 2: Quality of Experts We compare the performance of experts learned by the 4 MoE-LMs (where experts are separately trained by optimizing Eq. 2) and 2 baselines (WD (Holtzman et al., 2018) and PPLM (Dathathri et al., 2019)).
To study the sub-optimality gap in Corollary 1, we also include the performance of Transformer-based expert end-to-end LMs that are individually optimized with
REINFORCE (Li et al., 2016), using the expert labels { ` i } as rewards.
Inspired by Ghandeharioun et al.
(2019) on how bot behaviors are characterized, we use the following label functions to deﬁne the
[Table] [Table]
[Figure]
intents of experts: (i) ` pos-sent ( Y ) , ` neg-sent ( Y ) , ` joy ( Y ) , ` optimism ( Y ) , ` anger ( Y ) , ` sadness ( Y ) quantifydifferent sentiment tones and are constructed by a RoBERTa-based sentiment detector (Liao et al., 2021) that predicts whether an utterance is of positive or negative sentiment, and whether it falls into any of the 4 more-reﬁned emotions: {joy, optimisim, sadness, anger}; (ii) ` sent-coh ( X , Y ) measures empathy , i.e., bot’s sentiment coherence with user’s; (iii) ` question ( Y ) outputs 1 when a bot question is detected and 0 otherwise; (iv) ` exp ( X , Y ) quantiﬁes exploration , i.e., the tendency to avoid repetitive contexts (see Appendix B.7 for details).
Qualitatively, we also measure ﬂuency and expert skills of LMs using human ratings (see Appendix B.8 for details).
The results of the above experiments are reported in Table 3 and 8 (Appendix A.1), with sample utterances reported in Appendix A.4 to
A.10.
Results on human evaluation of different LMs w.r.t.
ﬂuency and different expert skills are given in Table 5.
Compared with the baseline LMs, generally the experts created under the MoE-LM framework, especially under MoE-2 and MoE-4, better capture all different language intents (where
WD and PPLM appear to capture negative sentiments and emotions much more effectively than behaviors), demonstrating the efﬁcacy of our approach which constructs specialized experts on a diverse language space via reward maximization (instead of weighted MLE).
Human evaluations also show that MoE-4 is most effective in generating semantically ﬂuent utterances that possess a wide range of expert characteristics.
Similar to the ablation study in EXP 1, all the experts associated with MoE-2 and MoE-4 are also among the best ones in capturing language intents.
Interestingly, with the Reddit data the experts in MoE-4 perform the best, while with much less data (Cornell) the best experts are built upon the simpler MoE-2 architecture.
We conjecture this difference is due to over-ﬁtting issues faced by the larger LMs (MoE-4) when there is insufﬁcient data for expert ﬁne-tuning.
In Fig.
2c and 2d we visualize the latent space of the sentiment-based experts in MoE-4, each with 400 conversation data samples.
Notice that the sentiment experts’ latent distributions are clearly separated (because positive and negative sentiments have opposite behaviors), while the emotion expert’s latent distribution have more gradual separations and even some overlaps (because e.g., joy versus optimism are quite similar, while joy versus anger are quite different).
This validates our MoE-LM represents different behaviors in separate regions of the latent space and justiﬁes our structural prior of modeling each expert as a specialized version of the primitive LM, whose latent distribution focuses on particular regions.
EXP 3: MoE-RL Against DialoGPT Simulated Users
We compare the dialogue management performance of MoE-LM, for which their MoE-based DMs µ are trained with different methods: (i) IQL (Kostrikov et al., 2021), (ii) Ensemble DQN (Carta et al., 2021), (iii) KL-control (Jaques et al., 2019), with 3 standard RL-based DM baselines using the VHRL architecture (Saleh et al., 2020): (i) REINFORCE (Li et al., 2016), (ii) KL-control, (iii) SAC (Haarnoja et al., 2018).
According to the results on expert quality in EXP2, we pick the MoE-2 and MoE-4 frameworks for the Cornell and Reddit tasks respectively.
For systematic evaluation, we perform the experiment by having these RL agents interact with a DialoGPT (Zhang et al., 2019) simulated user environment for a maximum of 5 turns.
The DM task is to maximize total user satisfaction in the conversation level, which is measured by both (i) user’s overall sentiment, and (ii)
user’s sentiment transition.
To construct an immediate reward that serves as a surrogate for user satisfaction, we set r ( X , a, X + ) =
1 ` sent ( X + ) +  2 ( ` sent ( X + )  1   1   L P
L  1 l {+=
 +} l ` sent ( X l )) , where the linear combi-
[Table] [Table]
[Table]
nation weights (  1 ,  2 ) =
25) correlate with Ghandeharioun et al.
(2019), and ` sent ( X ) is the same RoBerTa-based sentiment labeler as in EXP2, which assigns a score from [  1 , 1] that is proportional to the positive sentiment and inversely proportional to the negative sentiment prediction probabilities.
To ensure the baseline RL DM methods can also possess certain bot-level features, e.g., question, positive sentiment, etc., besides the above RL reward for user satisfaction we also optimize a linear combination of bot-based rewards when training the baseline models, see Appendix B of
Saleh et al.
(2020) for more details.
Since the DM problem lasts at most 5 turns, we use this as the effective horizon and set  = 1  1 / 5 = 0 .
The results of the above experiments (performed in both ofﬂine RL and MBRL settings) are reported in Table 2, Table 7 (Appendix A.1), Table 9 and 10 (Appendix A.2), with sample utterances reported in Appendix A.11.
Our experiments show that MoE-LMs outperform most baselines on DM performance.
We attribute this ﬁnding to three factors: (i) MoE-LM restricts the action space into a smaller set of candidate utterances generated by experts (whose qualities are validated in EXP2), the corresponding RL problem then becomes simpler and requires less data (especially in Cornell) to solve; (ii) Unlike the baseline RL methods, which need to optimize both bot-and-user signals, the
MoE DM agents focus on optimizing the user satisfaction goal and are therefore more effective; (iii)
Among different RL settings, MBRL, which ﬁrst learns a user utterance model (the model uses the same encoder from the primitive LM and learns a separate decoder for user-utterance prediction) then does DM, performs much better than ofﬂine RL methods that moderately improve upon the primitive
LM (behavior policy).
IQL-based dialogue managers are among the best across different settings potentially because IQL is more robust to co-variate shifts than standard RL methods, e.g., Ens-Q,
SAC, and yet it is less conservative than the behavior-regularized algorithms, e.g., KLC.
Interestingly, our MoE-LMs also have lower (better) perplexity scores than other methods.
This may be due to the fact that MoE-LM uses pre-trained encoder and decoder from the primitive LM, which are optimized for generalization and accuracy, while other RL methods may distort their language representations to create utterances that maximize reward but become less natural.
C ONCLUDING R EMARKS
We developed a mixture-of-expert (MoE) approach for RL-based dialogue management (DM).
Our
MoE language model (MoE-LM) comprises of three main components: (i) a LM that can generate diverse semantics for conversation histories, (ii) a number of specialized LMs (or experts) that can produce utterances corresponding to a particular attribute or intent, and (iii) a RL-based DM that performs dialogue planning with the utterances generated by the experts.
To understand how well our
MoE approach generates diverse and sensible utterances, and solves DM problems, we evaluated it using two open-domain dialogue tasks and compared it with SOTA baselines.
Our results showed that MoE-LM (i) improves diversity of text generation, (ii) can generate utterances with speciﬁc intents, and (iii) yields better overall performance.
We consider our work as a step forward in creating steerable LMs that possess different intents and in training RL-based DMs that can carry on rich conversations.
Future work includes improving the language representation with informationtheoretic approaches, ﬁne-tuning the experts based on the DM objective, extending the RL agent to track users’ behaviors (via abstract belief states) and plan upon them, preventing RL dialogue agents from generating harmful behaviors (via enforcing safety constraints), and evaluating our MoE-LM on more realistic problems, such as information retrieval, recommendation, and negotiation.