A N E QUAL -S IZE H ARD EM A LGORITHM FOR D IVERSE D IALOGUE G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Open-domain dialogue systems aim to interact with humans through natural language texts in an open-ended fashion.
However, the widely successful neural networks may not work well for dialogue systems, as they tend to generate generic responses.
In this work, we propose an Equal-size Hard Expectation–Maximization (EqHard-EM) algorithm to train a multi-decoder model for diverse dialogue generation.
Our algorithm assigns a sample to a decoder in a hard manner and additionally imposes an equal-assignment constraint to ensure that all decoders are well-trained.
We provide detailed theoretical analysis to justify our approach.
Further, experiments on two large-scale, open-domain dialogue datasets verify that our EqHard-EM algorithm generates high-quality diverse responses 1 .
I NTRODUCTION
Open-domain dialogue systems aim to generate natural language text utterances to hold open-ended conversations with humans (Li et al., 2017a;
Wang et al., 2021b).
These systems have shown great success, and are seamlessly integrated into our society in the form of chatbots.
For example, XiaoIce 2 , a chatbot from Microsoft, has gained over 660 million active users from all over the world across more than 40 platforms since its launch in 2014 (Zhou et al., 2020).
Recent advances in deep learning allow neural networks to be trained in an end-to-end fashion with large volumes of dialogue data, significantly improving the performance of open-domain dialogue systems (Serban et al., 2016; Wu et al., 2021).
However, neural dialogue systems tend to generate generic responses such as I don’t know (Li et al., 2016b; Wang et al., 2021b).
One of the possible causes is the one-to-many mapping phenomenon in the dialogue task, where a dialogue context may correspond to multiple plausible responses (Wei et al., 2019; Bao et al., 2020; Khan et al., 2020).
Learning to generate a dialogue utterance is thus analogous to learning a multimodal distribution of a continuous variable, where a mode refers to a peak in the distribution; in the dialogue task, a mode can be thought of as a set of similar responses.
The widely used cross-entropy training is not good at capturing different modes, as it encourages the prediction to cover all plausible responses, forcing the model to learn an overly smoothed distribution.
Consequently, these neural dialogue models resort to generating generic responses (Wei et al., 2019).
Previous studies address the generic responses mainly at training time or inference time.
Trainingtime approaches explore different training objectives: Li et al.
(2016b) apply reinforcement learning to optimize a customized reward function that discourages generic responses.
Khan et al.
(2020) train a generative adversarial network (GAN) for dialogue generation, as GAN discourages generic responses because they can be easily identified by the discriminator.
However, RL and GAN training is known to be difficult, and a recent study finds that “the generation performance of text GANs is not convincingly better, or even worse, than standard MLE training”
(He et al., 2021)
Wang et al.
(2021b) apply label smoothing to encourage less probable words to be generated.
Among inference approaches, Vijayakumar et al.
(2016) encourage diverse beam results by penalizing similar beams.
Holtzman et al.
(2020) apply nucleus sampling to allow plausible but less probable words to be decoded.
Another way to address the generic responses is to explicitly capture different modes with mixture models.
Shen et al.
(2019) train a multi-decoder model with the Expectation–Maximization (EM) algorithm
(Dempster et al., 1977) to address the one-to-many mapping phenomenon in the diverse machine translation setting.
However, their direct application of the standard EM algorithms suffers from the decoder-collapsing problem , where the multi-decoder model degenerates to a singledecoder model.
The authors attempt to alleviate this problem by disabling the dropout layers during the E-step of the EM algorithm, but their results show that the model is still susceptible to collapses.
To this end, we propose a novel EM variant to multi-decoder diverse dialogue generation.
A standard EM algorithm assigns a sample to all decoders by the posterior probability, thus known as
Soft-EM; it suffers from synchronous-training collapse, where the posterior probabilities tend to be similar and all decoders are also trained similarly.
The Hard-EM variant trains the decoder having the highest posterior probability; it suffers from non-training collapse due to the rich-gets-richer phenomenon.
Our proposed EM algorithm avoids both types of collapses by adopting hard assignments and imposing an equal-assignment constraint.
Hence, we call our approach Equal-size Hard
EM (EqHard-EM).
We conducted experiments on the Weibo (Gao et al., 2019) and OpenSubtitles (Lison et al., 2018) datasets.
Results show that our EqHard-EM algorithm alleviates the decoder-collapsing problem that occurs in both Soft-EM and Hard-EM, allowing the multi-decoder model to specialize and generate high-quality and diverse responses.
In addition, we provide in-depth theoretical and empirical analyses to better understand our EqHard-EM algorithm.
To sum up, our contributions are three-fold: 1)
We propose the EqHard-EM algorithm to alleviate the collapse issues in soft and hard EM variants; 2) We provide detailed theoretical analysis to justify our equal assignment; and 3) We conduct extensive empirical experiments to show the effectiveness of our approach.
R ELATED W ORK
Open-domain dialogue generation is a challenging task as neural dialogue models tend to generate generic responses such as “I don’t know”
(Li et al., 2016b; Wang et al., 2021b).
In addition to the training-time and inference-time approaches mentioned in Section 1, another line of work addresses the problem by using cue words to guide the response generation.
Yao et al.
(2017) enhance the gated recurrent unit to fuse cue words that are heuristically selected based on point-wise mutual information.
Gao et al.
(2019) jointly train a cue word inference model along with the decoder network to further improve performance.
Zou et al.
(2021) apply non-autoregressive generation to incorporate cue words in the generated response.
Incorporating commonsense knowledge is also investigated to tackle the generic responses.
Zhou et al.
(2018) use a large-scale knowledge graph to ground the generation process.
Wu et al.
(2020) make further improvements by retrieving and fusing relevant knowledge facts.
These approaches do not directly address the one-to-many phenomenon in open-domain dialogue generation, but rely on heuristics to make the responses more meaningful.
Expectation–maximization (EM, Dempster et al., 1977) is a principled approach to training mixture models.
The algorithm has been successfully applied in various fields (Dayan & Hinton, 1997; Sigg & Buhmann, 2008; Li et al., 2019).
Specifically in natural language processing, EM is applied to unsupervised or semi-supervised training of hidden Markov models for sequential labeling (Rabiner & Juang, 1986), known as the classic Baum–Welch algorithm
(Baum et al., 1970).
Hofmann (1999) apply the EM algorithm for probabilistic latent semantic analysis (pLSA), where the topic is an unobserved variable.
In the neural regime, the most related work to ours is Shen et al.
(2019), as they apply EM to diverse machine translation.
By contrast, we propose a theoretically justified
EqHard-EM variant for diverse dialogue generation.
Another related (but different) research topic is mixture of experts (MoE, Jacobs et al., 1991; Hampshire & Waibel, 1992; Shazeer et al., 2017), where an expert k predicts a real-valued feature vector e k , mixed by predicted probabilities p as (cid:80) k
p k e k .
The main difference between MoE and mixture models is that we mix decoders’ probabilities rather than features, as P ( r | c ) can be thought of as the marginal probability (cid:80) k P ( z = k |
c ) P ( r
| z = k, c ) .
However, we do not adopt such factorization because a uniform prior of P ( z | c ) makes the problem degrade to the optimization of P ( r | z = k, c ) individually for different k s. Admittedly, we might train a model to learn the mixing probability
P ( z = k | c ) , predicting which decoder is responsible for a context c .
This, however, violates our one-to-many assumption, which suggests one context requires multiple decoders.
Therefore, our EM formulation does not predict the decoder z by merely the context c , but determines which decoder fits a given training sample comprising both the context c and response r .
A PPROACH
In this section, we first present a multi-adapter architecture for the dialogue task (Subsection 3.1).
Then, we propose EqHard-EM, a novel EM variant, for training such mixture models (Subsection 3.2).
3. N EURAL A RCHITECTURE
We propose to address the one-to-many mapping phenomenon in the dialogue task with a mixture model.
Given a dialogue context, a shared encoder builds an input utterance’s hidden representations, based on which mutiple decoders generate a set of output responses.
Specifically, we adopt the adapter architecture (Houlsby et al., 2019), where all decoders share Transformer parameters but are inserted with thin adapter layers serving as a mixture component.
For multiple decoders, it is possible to instantiate each with a full-Transformer model (Shen et al., 2019), but this leads to a large number of parameters that may not fit into the memory of an ordinary
GPU.
To this end, we propose a multi-adapter architecture (Figure 1a), where different decoders share most Transformer parameters, but differ by a few inserted, thin adapter layers (Houlsby et al., 2019).
In fact, the adapter model is widely used for parameter-efficient domain adaptation (Artetxe et al., 2020; Wang et al., 2021a; Ngo Trung et al., 2021).
We propose to apply this architecture to mixture models.
Our preliminary experiments show that the multi-adaptor model results in less than one point degradation in BLEU scores, but reduces 76% parameters compared with full-Transformer decoders.
Therefore, we use the multi-adaptor model in our main experiments.
Specifically, an adaptor layer applies two projections of linear transformation and non-linear activation.
Further, a residual connection is included to ensure that the Transformer layers are not significantly disturbed.
This process can be formulated as
[Equation]
where x ∈ R
d is the input to the adapter layer; W 1 ∈ R d × d ′ and W 2 ∈ R d ′ ×
d are the projection matrices with d ′ < d ; and ϕ is a non-linear activation unit such as ReLU (Agarap, 2018).
3. T RAINING M ETHODS
We propose to train our multi-adapter architecture in two stages: 1) Pretraining of the shared
Transformer-layer parameters, and 2) EM training of adapter-layer parameters as the mixture model.
In the first stage, we pretrain a T5-small 3 encoder–decoder model (Raffel et al., 2020) with a standard cross-entropy loss since the Transformer parameters are shared among different mixture components; in the second stage, we adopt the standard treatment of adapter modules (Houlsby et al., 2019) and freeze the shared Transformer parameters to minimize the number of trainable parameters for effective adaptation.
In the rest of this subsection, we focus on the second stage, i.e., training the adapter parameters of the mixture model.
Consider a multi-decoder model with K decoders.
Given a dialogue context c and response r , our training objective is to maximize the marginal log-likelihood log (cid:80) k P ( r , z
=
k | c ; θ ) , where θ is the model parameters and the variable z ∈ { 1 , · · · , K } is the choice of the decoder.
Since z is unobserved in the training set, it can be viewed as a latent variable, and we apply the expectation– maximization (EM) framework (Dempster et al., 1977) as a principled approach to training latentvariable models.
Specifically, the EM algorithm alternates between the E-step and the M-step:
E-step: Estimate the posterior distribution of the latent variable by applying Bayes’ rule:
[Equation]
M-step: Maximize the expected log-likelihood with respect to θ :
[Equation]
We additionally assume a uniform prior for every context c as P ( z = k | c ; θ ) = 1 /K , following the one-to-many assumption that each context has multiple plausible responses.
This assumption simplifies Eqn.
(2) as
[Equation]
suggesting that the posterior is proportional to the likelihood under the uniform prior assumption.
Soft-EM.
The classic EM algorithm, as explained above, assigns a sample to a mixture component by the posterior distribution (2) in a soft manner, and thus is also known as Soft-EM.
However, we observe that Soft-EM performs poorly in our experiments due to synchronous-training collapse 4 , where all decoders are trained almost synchronously and generate similar responses.
In particular, the Transformer-layer parameters are pretrained.
Even with the inserted adapter layers, all decoders still behave similarly, yielding a similar posterior distribution (2).
This in turn yields a similar gradient for each decoder, and as a result, multiple decoders are virtually collapsed to one decoder.
Hard-EM.
Alternatively, the assignment in the E-step can be done in a hard manner by choosing the best-fit mixture component
[Equation]
Then, the M-step is approximated as follows
[Equation]
Unfortunately, the Hard-EM algorithm suffers from the non-training collapse , where only one decoder is trained due to the rich-gets-richer phenomenon.
In particular, an arbitrary decoder will be selected more often than the others by Eqn.
(5) due to the random initialization of adaptor layers.
Then, that decoder will be trained with more samples and perform better than the other decoders, making it even more likely to be selected by Eqn.
In practice, we observe that only one decoder is properly trained by the Hard-EM algorithm, making the mixture model ineffective.
be
In our development we have tried on
Our proposed EqHard-EM .
To this end, we propose an Equal-size Hard Expectation–
Maximization (EqHard-EM) algorithm to address the collapse issues.
Specifically, we adopt the hard assignments from Hard-EM to avoid synchronous-training collapse, whereas we impose an equal-assignment constraint to ensure that all decoders are adequately trained to avoid non-training collapse.
We formulate the E-step as a constrained optimization problem over an assignment matrix A ∈ { 0 , 1 } N × K , where N is the number of samples and K is the number of decoders.
The element
A nk indicates whether the n th sample is assigned to the k th decoder.
We define the cost matrix
C ∈ R N × K by the posterior probability, i.e., C nk =
− Q k
( r ( n ) , c ( n ) ) as in Eqn.
Our equal-size hard E-step aims to minimize A
[Equation]
subject to
[Equation]
n
[Equation]
Compared with Hard-EM, we impose the constraint (9) to ensure that every decoder is assigned with an equal number of samples.
In practice, this optimization problem can be transformed to a balanced assignment problem, and in turn solved by the classic Hungarian algorithm (Kuhn, 1955).
The algorithm was later improved by
Jonker & Volgenant (1987) to achieve a computational complexity of O ( N 3 ) , which is efficient in our batch implementation and contributes to less than 1% of the total training time.
As shown by
Figure 1b, the reformulation is accomplished by duplicating each decoder and its associated costs by a factor of NK to match the number of samples.
Here, we enforce the batch size N to be divisible by
K .
Note that the decoders are only conceptually duplicated when we apply the Hungarian algorithm.
The actual decoder network is not duplicated, and it does not add to computational cost.
For inference, we follow existing literature (Zhao et al., 2017;
Gu et al., 2019;
Shen et al., 2019) and consider the multi-response generation setting, where the goal is to generate a set of diverse responses given a dialogue context.
We collect the outputs of different decoders by greedy decoding.
Theoretical analysis.
We provide theoretical justifications for our EqHard-EM algorithm.
Our equal-assignment constraint makes sense intuitively, because the aggregated posterior should be the same as the prior distribution, suppose the posterior distribution is correctly estimated and we have enough samples.
This is given by (cid:80) c (cid:80) r P ( z | r , c ) P ( r , c )
= (cid:80)
c
P ( z | c ) P ( c ) =
(cid:80) c (1 /K )
P ( c ) = 1 /K for z = 1 , · · · , K .
In this part, we develop a theorem on how equal the assignments would be in the imperfect-posterior and finite-sample cases.
Theorem 1.
With probability at least 1 − δ , we have
[Equation]
for all z ∈ { 1 , · · · , K } .
Here, Q ( z | c , r ) is an estimated posterior and D is a dataset with finite samples.
The proof sketch is to consider finite samples (Lemma 1) and the estimated posterior (Lemma 2) separately, which can then be combined in a straightforward manner.
See Appendix A for the detailed proof.
Theorem 1 justifies the equal-assignment constraint, because with a high probability, the assignments will converge to uniform if we have a large dataset and a correct posterior.
More importantly, the difference is at most linear in the expected error of the estimated posterior Q ( z | c , r ) , such as the one in the E-step, and is sub-linear in 1 / |D| , supporting the use of our approach in practice.
While the term (cid:80) ( r , c ) ∈D Q ( z | r , c ) mainly works for soft assignment, we could not adopt soft assignment because it leads to synchronous-training collapse.
Nevertheless, the theorem is illuminating to our algorithm as well, as it is a hard-assignment approximation similar to Hard-EM.
E XPERIMENTS 4. S ETUP Datasets.
We evaluated our approach on two large-scale open-domain dialogue datasets:
Weibo (Gao et al., 2019) and OpenSubtitles (Lison et al., 2018).
The Weibo dataset is a compilation of posts and replies, crawled from a Chinese micro-blogging website.
The dataset features explicit one-to-many mapping because in most cases, a post corresponds to multiple replies.
On the other hand, the OpenSubtitles dataset is a collection of movie dialogues, which are parsed from online subtitle files.
The dataset is known to be noisy (Vinyals & Le, 2015;
Feng et al., 2020; Nakazawa et al., 2021), and therefore it demonstrates the implicit one-to-many phenomenon: a movie utterance may lead to many plausible responses even though the dataset only contains one reference response.
Admittedly, there exist other manually constructed, high-quality datasets such as DailyDialog (Li et al., 2017b), but they are usually much smaller and the one-to-many phenomenon is less severe.
Recently, Wen et al.
(2022) identify significant overlapping between training, validation, and/or test splits in various open-domain dialogue datasets (including OpenSubtitles) due to the overlights of data collection.
We use their cleaned OpenSubtitles dataset, which contains 1M, 12K, and 12K samples for training, validation, and test, respectively.
We investigated the Weibo dataset and found significant overlapping between the validation and test sets as well, so we followed their cleaning strategy and obtained 4M, 10K, and 10K training, validation, and test samples.
Overall, Weibo and
OpenSubtitles allow us to conduct comprehensive analyses in both explicit and implicit one-to-many mapping scenarios and in different languages.
Metrics.
Open-domain dialogue evaluation is currently an active area of research due to its oneto-many nature (Tao et al., 2018; Sinha et al., 2020).
In particular, the standard BLEU metric, which measures the n -gram similarity between generated and reference responses, has low correlation with human evaluation because a plausible response is not necessarily similar to the reference response (Liu et al., 2016).
Therefore, we followed previous work in open-domain dialogue (Zhao et al., 2017; Gu et al., 2019; Li et al., 2020) and generated 10 responses for each dialogue context to compute BLEU-P, BLEU-R, and
BLEU-F. Roughly, BLEU-P measures the average quality of the generated responses, whereas BLEU-R measures how well the references are covered.
BLEU-F is the harmonic mean.
We adopted uni-gram and bi-gram BLEU scores in our experiments, because
Liu et al.
(2016) show that higher-order BLEU scores are often zero, making the metric rely mostly on smoothing techniques.
For diversity, we adopt the Dist (Li et al., 2016a) and Pairwise-BLEU
(Shen et al., 2019) scores.
Dist- n is a widely used metric that measures the percentage of distinct n -grams among utterances.
We report Dist-1 and Dist-2 between the generated responses for each dialogue context.
In addition, we follow Shen et al.
and report Pairwise-BLEU, which measures the pairwise similarity (measured in BLEU) among the responses, i.e., a low Pairwise-BLEU indicates high diversity.
Implementation details.
We considered 10 decoders due to the setup of diverse generation evaluation (Zhao et al., 2017;
Gu et al., 2019; Li et al., 2020).
Each E-step involved 640 samples, assigning 64 samples to each decoder.
More implementation details are presented in Appendix B.
4. R ESULTS AND A NALYSES
Main results.
Table 1 shows our main results on Weibo and OpenSubtitles.
The T5-small model is a pretrained encoder–decoder Transformer (Raffel et al., 2020), where we experimented both beam search and sampling.
For beam search, we set the beam size to be 10, i.e., the number of samples needed for evaluation.
As known, beam search does not generate diverse responses (Vijayakumar et al., 2016; Holtzman et al., 2020), shown by low BLEU-R, low Dist scores, but high
Pairwise-BLEU.
The performance of nucleus sampling is inconsistent in the two experiments: on the Weibo dataset, nucleus sampling improves the diversity to a certain extent, but its BLEU-F and
Dist scores are inadmissibly worse than our approach; on OpenSubtitles, nucleus sampling gener-
[Table]
ates low-quality texts (very low BLEU-F), which are diverse among themselves.
6 We examined the samples and find two plausible reasons for the inconsistency: 1) The Weibo dataset is in Chinese and
BLEU scores measure the overlap of Chinese characters.
This is more lenient than English words, making it possible to retain relatively high BLEU-F scores.
Nucleus sampling highly depends on the distribution temperature and the nucleus size for sampling.
If the sampling scope is too small, then the diversity would not be improved much; if otherwise the sampling scope is too large, the quality would be deteriorated.
By contrast, our EqHard-EM approach is able to perform consistently well on both datasets by simple greedy decoding.
Our BLEU-R is much higher than most competing methods.
This shows that our EqHard-EM has a high coverage of potential responses, largely alleviating the one-to-many issue in dialogue generation.
Overall, we achieve the state-of-the-art BLEU-F scores, while at the same time largely improving diversity.
In fact, we outperform all non-sampling methods in terms of
Dist and Pairwise-BLEU scores.
We also experimented with recently proposed models: AdaLabel (Wang et al., 2021b) is an encoder– decoder Transformer, trained with adaptive label smoothing to improve diversity.
DialogBERT (Gu et al., 2021) features the use of a BERT encoder (Devlin et al., 2019) to capture contextual information.
They were alleged to be the state-of-the-art models when the overlapping problem was not realized.
However, they perform poorly on the deduplicated datasets; our results in the diverse generation setting are consistent with single-response generation (Wen et al., 2022).
Comparing EM variants.
We conducted controlled experiments to compare EM-like algorithms (Table 2).
We do not adopt the recurrent dropout trick (Shen et al., 2019) because we find them ineffective in our dialogue setting (see Appendix E).
We find Soft-EM suffers from synchronous-training collapse, as it achieves very high Pairwise-BLEU scores.
It achieves even lower diversity than beam search because of the synchronous-training collapse, which is characterized by its Pairwise-BLEU scores.
This can be best seen by decoder assignment statistics (Figure 2a), where every decoder is assigned with an equal fraction of samples with near-zero standard deviation.
On the other hand, the Hard-EM algorithm also performs poorly.
It suffers from the non-training collapse, due to the rich-gets-richer phenomenon.
As seen in Figure 2b, only Decoder 6 is trained in the experiment.
Hard-EM achieves near-zero Pairwise-BLEU and the lowest Dist scores because the untrained decoders tend to repeat random words, which can be verified in our case studies (Appendix D).
As a result, the generated sentences are not meaningful, shown by extremely low BLEU scores.
As shown in Figure 2c, our EqHard-EM guarantees that all decoders are trained with the same number of samples, preventing non-training collapse; since our algorithm adopts hard assignments (either 0 or 1), it also has a high variance among different samples, preventing from synchronoustraining collapse.
Overall, our EqHard-EM achieves the highest BLEU-F score, as well as the best diversity scores (excluding the improperly trained Hard-EM).
Appendix D presents sample responses generated by Soft-EM, Hard-EM, and EqHard-EM.
[Table]
We further investigate whether EM-style, posterior-based assignment actually helps.
We compare our EqHard-EM with two other equal-size hard assignment methods: EqRandom-Fixed and EqRandom-Dynamic .
The EqRandom-Fixed variant pre-assigns samples to a decoder before training; this is essentially training each decoder on a partition of the training set.
EqRandom-Dynamic assigns samples to decoders randomly and dynamically during training; this is essentially training a model with multiple initializations, if the number of epochs is large.
In both cases, we ensure each decoder gets the same number of samples, so that they are equal-size hard assignments.
As seen, these two methods perform similarly to each other.
They are slightly worse than beam search, but much worse than our EqHard-EM in terms of both quality and diversity.
These controlled experiments convincingly show that our EqHard-EM is able to take advantage of the principled EM algorithm to learn a mixture of decoders, and that our model does not suffer from the collapse issue.
Analyzing different decoders.
We analyze how decoders are specialized in certain aspects of dialogue responses.
Admittedly, the responses generated by different decoders would not exhibit clear clustering phenomena, because our inputs are open-domain dialogue utterances covering a wide range of topics.
The generated responses have to fit the input in a certain way, so the betweensample variance would be larger than decoder variance.
Despite this, we do find obvious patterns of certain decoders, including the length, 7 sentence structures (e.g., whether it is a question), and the preference of certain words (Figure 3).
For example, Decoder 7 tends to generate short question responses, whereas Decoders 9 and 10 tend to generate positive-sentiment utterances with the word “like/love”.
For comparison, we also show beam search results, which are nearly uniform among different samples in the beam.
Our decoder analysis, despite its simplicity, is conducted in a quantitative fashion, as opposed to previous case study-based claims (Gu et al., 2019;
Shen et al., 2019).
This provides rigorous evidence that our EqHard-EM is able to train multiple decoders to capture different “modes” of open-domain dialogue utterances.
Effect of the number of decoders.
We investigate how well our approach scales with the number of decoders.
Figure 4 shows a comparison between beam search and our multi-decoder model under 5-, 10-, and 15-generation settings.
Due to the limit of space and resources, we conducted the analysis only on the Weibo dataset.
As seen, our multi-decoder model performs similarly to beam search when there are only 5 decoders.
However, the performance gap increases when we have more decoders, which verifies the need for our multi-decoder model.
Additional results.
We conducted human evaluation (Appendix C), where results are highly consistent with automatic metrics, showing that our model generates high-quality and diverse responses.
Appendix D shows example outputs of different EM algorithms, confirming that our EqHard-EM variant has largely alleviated the collapse issues.
Appendix E shows an ablation study for additional EM variants, including the recurrent dropout trick (Shen et al., 2019).
C ONCLUSION
In this work, we propose an Equal-size Hard
Expectation–Maximization (EqHard-EM) algorithm for multi-decoder diverse dialogue generation.
Our
EqHard-EM adopts the hard assignment as in Hard-EM, but additionally imposes an equal-assignment constraint to ensure all decoders are well-trained.
We provide detailed theoretical analysis to justify our algorithm.
Extensive experiments on two large-scale datasets further confirm that our EqHard-EM algorithm enables decoders to specialize and generate high-quality and diverse responses.
Limitation and future work.
Our paper addresses the generic responses for dialogue systems by training a mixture of decoders.
This inevitably slows down the training process similar to standard
EM algorithms (Shen et al., 2019), because we need to perform inference for all decoders even though our EqHard-EM propagates gradient to one.
However, our Hungarian algorithm has little overhead (less than 1%) for the training process, which is negligible.
It should also be mentioned that, when the model is deployed, the inference speed would not be affected much, because different decoders can be parallelized if samples come one by one.
Moreover, our multi-adapter architecture saves 76% memory without hurting much performance compared with full-Transformer decoders.
In the future, we plan to explore more effective initialization by meta-learning to better specialize decoders during our EqHard-EM training.
R EPRODUCIBILITY S TATEMENT
We provided a detailed proof of Theorem 1 in Appendix A. For reproducing our results, we published the source code of our work as a Git repository (see the footnote from the abstract).
The implementation details are also thoroughly discussed in Appendix B. We further conducted training and inference efficiency experiments on an i9-9900X CPU and a TITAN RTX GPU.
Notably, the computational overhead of the Hungarian algorithm is negligible, as the E-step, Hungarian algorithm, and M-step take 75.69%, 0.97% and 24.31% of the total training time, respectively.
At inference, EqHard-EM is actually more efficient than beam search of a single-decoder model.
Our multi-adapter model processes 79.1 samples per second, whereas beam search processes 16.9 samples per second.
Our approach achieves a 4.7x speedup because inference for each decoder can be computed in parallel across multiple GPUs, whereas beam search cannot leverage the same level of parallelism.
A P ROOF OF T HEOREM
We introduce two lemmas for proving Theorem 1.
The first lemma controls the error between the empirical average and the true expectation.
It is a direct application of Hoeffding’s inequality.
The second lemma bounds the difference between the learned probability and the true underlying probability.
Following the notation in Eqn.
(2), we use Q ( z | c , r ) as the estimated posterior.
Lemma 1.
Given a dataset D consisting of context-response pairs ( c , r ) , for any z , and ϵ > 0 , we have
[Equation]
Proof.
Observe that Q ( z | c , r ) is real-value random variable.
The proof is then clear by using Hoeffding’s inequality.
Lemma 1 is a quantitative description of the law of large numbers.
Lemma 2.
Given any z and estimation Q ( z | c , r ) of the true P ( z | c , r ) , we have
[Equation]
Proof.
[Equation]
Here, Eqn.
(11) replaces 1 /K with the marginalization of P ( z, c , r ) over c and r .
Lemma 2 bounds the absolute difference between the estimation Q ( z | c , r ) and the true P ( z | c , r ) .
Theorem 1.
With probability at least 1 − δ , we have (cid:12)(cid:12)(cid:12)(cid:12)|D| (cid:88)
[Equation]
for all z ∈ { 1 , · · · , K } .
Here, Q ( z | c , r ) is an estimated posterior and D is a dataset with finite samples.
Proof.
Lemma 1 shows a high probability inequality for a single z .
The probability for the original inequality to hold for all z is then:
[Equation]
Choose an ϵ such that δ = 2 K exp (cid:0) − 2 ϵ 2 |D| (cid:1) .
We have
Pr
[Equation]
≤ δ
Then with probability at least 1 − δ
[Equation]
we have:
[Equation]
Lemma
[Equation]
(by Lemma 2) concluding the proof.
B I MPLEMENTATION D ETAILS
Our multi-adapter model used T5-small (Raffel et al., 2020) from HuggingFace (Wolf et al., 2019) as the backbone: 6 layers, 8 attention heads, an input/output dimension of 512, and an inner dimension of 2048 for both the encoder and decoder.
We implemented our multi-decoder model by inserting an adapter layer with an inner dimension of 256 after each Transformer decoder layer.
We adopted 10 decoders based on the common evaluation metric of diverse text generation (Zhao et al., 2017; Gu et al., 2019).
Table 3 shows a comparison between our multi-adapter architecture and the full-Transformer architecture.
As seen, our multi-adapter model retains a competitive BLEU performance while only using 24% of the parameters.
Since the goal of our research is not to push the scores, we adopted the multi-adaptor architecture in the main experiments, because it helped our development process much.
For training, we had two stages: 1) Pretraining of the shared Transformer model, and 2) EM training of the adapter layers.
In both stages, we used a learning rate of 0.001 with the Adam optimizer with β = (0 . 9 , 0 . 999) .
We used a batch size of 64 for pretraining.
For our 10-decoder EqHard-EM training, we considered 640 samples in an E-step, assigning 64 samples for each decoder.
C H UMAN E VALUATION
We conducted human evaluation to compare both the quality and diversity of the responses from
T5-small (beam search) and our multi-decoder model.
We again chose the Weibo dataset, because it is less noisy.
Five native Chinese speakers were provided with a dialogue context and generated responses of two systems, and were asked to select the better system or indicate a tie.
For quality, annotators were presented with only one response from each model, preventing them to identify the systems based on diversity.
For diversity, all 10 responses were presented.
In both cases, the presentation order of the two systems was randomly shuffled, so the annotators could not tell which system generated which response.
[Table] [Table]
Table 4 shows the results of human evaluation.
Our multi-decoder model shows statistically significant improvement in terms of overall quality and diversity.
Especially, annotators believe our approach generates more diverse utterances on 18x more samples.
The results are highly consistent with the automatic metrics in Table 4, confirming the effectiveness of our approach.
D C ASE S TUDY
Table 5 shows example responses from Weibo and OpenSubtitles.
As seen, Soft-EM exhibits the synchronous-training collapse, where all decoders generate highly similar responses.
Hard-EM, on the other hand, suffers from the non-training collapse, where a large number of decoders are not properly trained and generate low-quality text (e.g., certain words being repeated).
By contrast, our
EqHard-EM algorithm enables us to properly train all decoders to generate high-quality, diverse responses.
E A BLATION S TUDY FOR A DDITIONAL EM V ARIANTS
Shen et al.
(2019) address the collapse issues in the EM algorithm by implementing an ad hoc recurrent dropout trick: disabling dropout layers for the E-step while retaining dropout layers for the M-step.
They claim that the dropout layers in the E-step interfere with the specialization of different decoders, causing the mixture model to collapse.
We conduct an ablation study and show that their trick is ineffective in our dialogue setting.
As shown by Table 6, the recurrent dropout trick has minimum effect on the EM algorithms, and it does not prevent the non-training and synchronoustraining collapses for Hard-EM and Soft-EM.
This is not surprising because even Shen et al.
(2019) acknowledge that their recurrent trick does not work consistently, as shown in Table 7 of their
Appendix (red and blue numbers).
The one-to-many phenomenon in our dialogue setting is more severe than their machine translation setting, and it is understandable that such an ad hoc trick, without any theoretical justifications, may fail in the more difficult task.
We also compare EM variants with learned and uniform priors.
As shown by Table 6, learned priors perform worse than uniform priors due to the more severe non-training collapse problem, which is demonstrated by the extremely low BLEU scores.
This is understandable because a learned prior gives the model more opportunities to select the best decoder, further exacerbating the “rich-getsricher effect”.
Therefore, we do not consider the recurrent dropout trick and learned priors in our main experiments due to their ineffectiveness and poor performance.
[Table] [Table]