A Randomized Link Transformer for Diverse Open-Domain Dialogue Generation
Abstract
A major issue in open-domain dialogue generation is the agent’s tendency to generate repetitive and generic responses.
The lack in response diversity has been addressed in recent years via the use of latent variable models, such as the Conditional Variational Auto-Encoder (CVAE), which typically involve learning a latent Gaussian distribution over potential response intents.
However, due to latent variable collapse, training latent variable dialogue models are notoriously complex, requiring substantial modification to the standard training process and loss function.
Other approaches proposed to improve response diversity also largely entail a significant increase in training complexity.
Hence, this paper proposes a Randomized
Link (RL) Transformer as an alternative to the latent variable models.
The RL Transformer does not require any additional enhancements to the training process or loss function.
Empirical results show that, when it comes to response diversity, the RL Transformer achieved comparable performance compared to latent variable models.
Introduction
Open-domain dialogue generation refers to the task of generating coherent, natural and human-like dialogue given solely the dialogue context (also known as the dialogue history).
The development of opendomain dialogue agents that can engage humans in seamless general conversation (or chit-chat) is one of the main objectives of conversational AI.
Currently, however, agents display a tendency to generate repetitive and generic dialogue responses, which negatively impact both naturalness and contextual coherence.
Recently, researchers have turned to latent variable models, specifically the Conditional Variational Auto Encoder (CVAE)
(Sohn et al., 2015), to address this issue (Yang et al., 2021;
Gao et al., 2019; Zhao et al., 2017; Cao and Clark, 2017).
In addition to open-domain dialogue generation, latent variable models have also been applied to related tasks such as personalized dialogue (Lee. et al., 2022; Wu et al., 2020; Song et al., 2019), empathetic dialogue (Li et al., 2021, 2020b,a; Zhou and Wang, 2018), and topical dialogue generation (Wang et al., 2020).
These works have generally involved modelling the potential dialogue response intents as a latent Gaussian prior, which is typically generated by a Multi-Layer Perceptron (MLP).
During inference, a latent instance is sampled from the generated Gaussian prior via the reparameterization trick and fed to the decoder.
Stochasticity is induced during the response generation via such random sampling process.
However, even though latent variable models are effective at improving response diversity, they are notoriously hard to train primarily due to the Kullback-Liebler (KL) vanishing problem.
Usually, this problem is addressed via
KL annealing or incorporating a Bag-of-Words loss.
While KL annealing requires tuning the weighting hyperparameter β , attaining the Bag-of-Words loss involves defining an additional task of predicting the response bag-of-words.
This results in additional complexity during training.
Several other approaches to promoting response diversity which involve introducing an alternate loss function such as the Maximum Mutual Information (MMI) objective (Li et al., 2016a), the Inverse Token Frequency (ITF) objective (Nakamura et al., 2018), and the Inverse N-gram Frequency (INF) objective (Ueyama and Kano, 2020), require considerable additional computation steps.
On the other hand, adversarial learning-based (Li et al., 2017a) and embedding augmentation-based (Cao et al., 2021) approaches require extensive modifications to the standard training process, resulting in a significant increase in training complexity.
Hence, inspired by randomization-based neural networks (Suganthan and Katuwal, 2021) and the
Random Vector Functional Link (RVFL) neural network (Pao and Takefuji, 1992) in particular, we introduce a novel Randomized Link (RL) Transformer.
An alternative to latent variable models which does not require any modifications or enhancements to the standard training process or loss function.
In other words, the RL Transformer can be trained via standard gradient descent solely on the standard negative log-likelihood loss.
Experimental results on the DailyDialog (Li et al., 2017b) and EmpatheticDialogues (Rashkin et al., 2019)
corpora show that our RL Transformer successfully improves the diversity of the generated response, achieving comparable response diversification relative to latent variable approaches.
In addition, compared to the latent variable models, responses generated by our RL Transformer are noticeably more fluent and contextually coherent.
The remainder of this paper is organized as follows: Section 2 provides additional background information regarding open-domain dialogue generation and RVFL neural networks; Section 3 describes the proposed RL Transformer in detail; Section 4 provides details regarding our implementations and experiments; Section 5 presents the experimental results as well as our analysis of the results; Section 6 concludes the paper.
Related Work 2. Neural Open-domain Dialogue Generation
In recent years, generative neural models have been commonly applied to the task of open-domain dialogue generation.
Influenced by advances in machine translation, popular approaches to this task featured a sequence-to-sequence (seq2seq) architecture (Sutskever et al., 2014).
Recurrent neural networks such as the Long Short Term Memory (LSTM) and the Gated Recurrent Unit (GRU) have been often utilized in both the encoder and decoder of a seq2seq model (Shang et al., 2015; Sordoni et al., 2015).
More recently, Transformer-based models (Vaswani et al., 2017) have taken centre stage.
Multiple works have leveraged Transformerbased pretrained language models such as BERT,
GPT and GPT-2 to improve the overall language understanding and generation capabilities of their dialogue agents (Gu et al., 2021; Zhang et al., 2020;
Zhao et al., 2019).
In addition to latent variable models, different learning approaches such as reinforcement learning (Saleh et al., 2019; Li et al., 2016b) and adversarial learning (Li et al., 2017a) have also been applied to this task.
2. Random Vector Functional Link Neural Networks
The Random Vector Functional Link (RVFL) neural network (Pao and Takefuji, 1992) is essentially a single-layer feed forward neural network with a direct link between the input and output layer.
The optimal weights of an RVFL can be obtained iteratively, or through a closed form solution via regularized least squares or the Moore-Penrose pseudoinverse.
Prior work has mathematically proven that the RVFL is an efficient and effective universal approximator (Needell et al., 2020).
Over the years, multiple RVFL variants including (but not limited to) the deep RVFL (Shi et al., 2021), ensemble deep
RVFL (Shi et al., 2021), sparse Pretrained-RVFL (Zhang et al., 2019) and Rotation Forest-RVFL (Malik et al., 2021) have been proposed.
Recently,
RVFL neural networks have been applied to a broad range of practical tasks across multiple domains such as remote sensing (Dai et al., 2022), malware classification (Elkabbash et al., 2021), medical image classification (Nayak et al., 2020; Katuwal et al., 2019) and even Covid-19 spread forecasting (Hazarika and Gupta, 2020).
Methodology 3. Randomized Link (RL) Transformer
In this paper, we propose a Randomized Link (RL)
Transformer for open-domain dialogue generation.
Our proposed model generates the dialogue response based only on the dialogue context.
Similar to the standard Transformer, the RL Transformer consists of an encoder and decoder.
The encoder maps an input sequence X
= { x 0 , ..., x J − 1 } ( J refers to the length of the input sequence) to an intermediate representation Z
= { z 0 , ..., z J − 1 } .
The decoder then accepts Z as input and generates the final output Y =
{ y 0 , ..., y K − 1 } ( K refers to the length of the output sequence) token by token, in an auto-regressive manner.
The proposed RL Transformer leverages linear layers with randomly initialized weights to incorporate stochasticity into the response generation process.
Our work is largely inspired by the Random Vector Functional Link (RVFL) neural network (Pao and Takefuji, 1992), a single-layer feed forward randomization-based neural network consisting of a fixed randomized hidden layer and a direct link from the input to the output layer.
The
[Figure]
Encoder Decoder Randomized Link Self Attention
Decoder
Dialogue Context
Generated Response
Linear (Softmax)
Add & Layer Normalization Add & Layer Normalization
Add & Layer Normalization Add & Layer Normalization
Add & Layer Normalization
[Figure]
direct link is implemented by concatenating the input with the randomized hidden layer output, before being fed to the output layer.
Only the weights in the output layer are trainable.
We incorporate the RVFL architecture into the self-attention and feed forward networks of a Transformer.
Essentially, the Randomized Link Transformer encoder consists of a Randomized Link SelfAttention (RLSA) network and a Randomized Link
Feed Forward (RLFF) network.
The decoder, on the other hand, consists of a RLSA network, followed by a Encoder-Decoder (ED) RLSA network and a regular Feed Forward (FF) network.
Similar to the original Transformer architecture, residual connections and layer normalization is introduced after every RLSA, RLFF, ED RLSA and FF network.
An overview is provided in Figure 1.
3. Randomized Link Self-Attention
In order to incorporate stochasticity into the response generation process, we introduce a novel
Randomized Link Self-Attention (RLSA) network featuring feed forward linear layers with random weights.
Similar to to the original Transformer architecture, our randomized link self-attention involves mapping a query vector, denoted by Q , and a set of key-value vector pairs, denoted by K and
V respectively, to an output.
The output is derived by computing the weighted sum of the values, where the weight assigned to each value is computed by taking the dot product of the query with the corresponding key.
For each attention head in the standard transformer architecture, the input is passed to three distinct linear layers, resulting in the query, key and value vectors Q n , K n , and V n , where n refers to an arbitrary attention head.
In RLSA, each input will be fed to a single random linear layer.
The sizes of all random linear layers used in the randomized Transformer, denoted by d rand , are identical.
Similar to the RVFL, we introduce a direct link by concatenating the output of this layer with the original input.
The resultant representation is fed to a separate linear layer with trainable weights.
Similarly, for multi-headed
RLSA, a distinct Q n , K n , and V n vector is defined for each of the N attention heads.
The dimensionality of the Q n and K n vectors is denoted by d k , and the dimensionality of the V n vector is denoted by d v .
This can be expressed as follows:
[Equation]
where W Q , W K , and W V represent the weights of the trainable linear layers corresponding to the
Q , K and V vectors respectively.
We use the superscript () r to denote the randomized matrices, whereby W rQ , W rK , and W rV represent the weights of the randomly initialized linear layers corresponding to the Q n , K n and V n vectors respectively.
X denotes the input sequence, and [ · ] represents the concatenate operation.
The randomized linear layers W rQ , W rK , and W rV are initialized every epoch with Xavier normal initialization (Glorot and Bengio, 2010) i.e., W rQ , W rK , W rV ∼ N (0 , σ 2 ) where σ = γ × (cid:113) 2 d hidden + d rand .
The gain value γ = 1 . 0 .
The selection of the standard deviation or the variance of the initialization is vital to model performance.
This is because an excessively large variance would result in model divergence, while an excessively small variance would result in a drop in stochasticity.
For the randomized layers in RLSA, weights initialized from a normal distribution with a suitable standard deviation would allow the model to converge while maintaining stochasticity.
As seen in the equation, for the Xavier normal initialization, the standard deviation of the Gaussian distribution from which the initial weights are sampled is a function of the total number of inputs and outputs.
Empirically, we found that the standard deviation value utilized during Xavier normal initialization would generally outperforms other standard deviation values across all randomized layer sizes.
Then, following the standard Transformer architecture, the dot product of all corresponding Q and K vectors are computed to obtain the attention maps.
Then, to attain the score, the softmax function is applied over the dot products divided by the square root of the dimension of the Q and
K vectors i.e., √ d k .
Each of the V vectors is then multiplied with the attained score.
This results in the following expression:
[Equation]
where T represents the transpose operation.
The output of each attention head Z n is concatenated to form Z :
[Equation]
where N represents the number of attention heads.
Then, the resulting representation Z is then passed to a single linear layer with randomly initialized weights.
Once again, to obtain the encoder output Z , the output of the random layer is concatenated with the original input ¯ Z , and fed to a separate linear layer with trainable weights (direct link).
[Equation]
where W Z and W rZ represent the weights of the trainable linear layer and randomized linear layer used to obtain the encoder output Z respectively.
Similarly, the randomized linear layer W rZ is initialized every epoch with Xavier normal initialization i.e., W rZ ∼ N (0 , σ 2 ) where σ = γ × (cid:113)
[Equation]
The gain value γ = 1 . 0 .
For the Encoder Decoder (ED) RLSA in the decoder, the encoder outputs and prior decoder outputs are used as input.
The output of the prior decoder layer is used to generate the queries, and the encoder outputs are used to generate the keys and values.
An overview of the RLSA network is presented in Figure 2.
3. Randomized Link Feed Forward Network
The feed forward network in the standard Transformer consists of a two-layer fully trainable feed forward neural network.
Likewise, the Randomized
Link Feed Forward (RLFF) network is a two-layer feed forward neural network which features a randomly initialized fixed linear layer with a ReLU [Figure] activation function followed by a trainable linear layer.
Similarly, the direct link is introduced by concatenating the output of the randomized layer with the original inputs, and passing the resultant representation to the trainable layer.
In contrast to the RLSA network (Section 3.2), no additional randomized layers are introduced in the RLFF network.
The first linear layer is randomized instead.
This can be expressed as:
[Equation]
where W r 1 and W 2 refer to the first random linear layer and the second trainable linear layer respectively.
The size of the randomized linear layer W ris represented by d ff , and the size W 2 is d hidden .
Unlike the RLSA network, for the RLFF network, the randomized linear layer W r 1 is initialized every epoch with Xavier uniform initialization (Glorot and Bengio, 2010) i.e., W rZ ∼ U ( − a, a ) where a
[Equation]
6 d hidden + d ff .
Since the ReLU activation is applied to the layer output, the gain value. The gain value γ = √ 2 .
We found that utilizing a uniform initialization in the RLFF network instead of a normal initialization would result in a slight increase in response diversity.
Also, it should be noted that the RLFF network is only utilized in the encoder of the Randomized Transformer.
An overview of the RLFF network is presented in Figure 3. [Figure]
Experiment 4. Data
We evaluate the Randomized Link Transformer on the DailyDialogs (Li et al., 2017b) and EmpatheticDialogues (Rashkin et al., 2019) corpora.
For the EmpatheticDialogues corpus, the agent is expected to generate an appropriately empathetic response given the dialogue context and emotion label, which is not used in our experiments.
The training, validation and test set consists of 19,533, 2,770, and 2,547 dialogues respectively.
The DailyDialog corpus consists of general human-written dialogue examples covering a wide range of topics and emotions.
Similarly, the provided intent and emotion labels are not used in our experiments.
The training, validation and test set consists of 11,118, 1,000 and 1,000 dialogues respectively.
4. Implementation
In our experiments, we implement the Randomized
Link (RL) Transformer with 4 encoding layers,decoding layers, with 4 attention heads ( N = 4 ).
Since the 300 dimensional GloVe embedding (Pennington et al., 2014) is used, the hidden dimension d hidden =
The size of all randomized layers in the RLSA network d rand is fixed at 512.
d k , d v and d z were set to 64 for experiments of the
DailyDialog corpus, and 256 on the EmpatheticDialogues corpus.
For the RLFF network, d ff is set to 2048.
Inputs to the RL Transformer consists of the dialogue context (limited to 4 dialogue turns).
Responses are generated via greedy decoding.
During training, the Adam optimizer (learning rate = 0.00015, batch size = 32) is used.
4. Baselines
We implement the following four models in our experiments:
Transformer.
We implement a Transformer (Vaswani et al., 2017) with standard self attention and feed forward components.
The Transformer parameters are identical to the RL Transformer as described in Section 4.2.
CVAE.
Similar to Zhao et al.
(2017) and Lin et al. (2020), we implement a Transformer-based CVAE where the latent variable sampled from the latent
Gaussian is combined with the output of the encoder before being fed to the decoder.
Following
Lin et al.
(2020), the latent Gaussian is generated by a three-layer MLP with 512 node hidden layers, and the size of the random latent variable is fixed at 300.
The remaining Transformer parameters are identical to the RL Transformer as described in
Section 4.2.
SVT.
We also implement the Sequential Variational
Transformer (SVT) proposed in Lin et al.
The SVT replaces the standard Transformer decoder with a variational decoder layer which implicitly generates a distinct latent variable for each position.
Similarly, the latent Gaussians are generated via three-layer MLPs with 512 node hidden layers, and the size of the random latent variable is fixed at 300.
The remaining Transformer parameters are identical to the RL Transformer described in Section 4.2.
RL Transformer.
We implement the proposed
RL Transformer with configuration described in
Section 3.2.
4. Evaluation 4.4. Objective Measures
Distinct-N. We use the Distinct-1 and 2 scores, denoted by D-1 and D-2 respectively, to quantify the inter-response diversity of the generated responses.
Essentially, the Distinct- n score involves computing the number of distinct n -grams in a given text, and dividing the number by the total number of tokens.
The Distinct score is derived based on all generated responses.
MTLD & MATTR.
Additionally, we also utilize lexical diversity measures such as the Measure of Textual Lexical Diversity (MTLD) and
Moving-Average Type–Token Ratio (MATTR) score.
MATTR and MTLD are essentially text length invariant variants of the Token-Type Ratio (TTR).
MTLD involves computing the Token-Type
Ratio (TTR) for sequentially larger segments of the sentence until a predefined threshold h .
On the other hand, deriving MATTR requires averaging [Table] Transformer (Vaswani et al., 2017) [Table]
CVAE (Zhao et al., 2017)
[Table]
SVT
[Table]
Lin
[Table]
al., 2020)
[Table]
Transformer
[Table]
Transformer (Vaswani et al., 2017)
[Table]
CVAE (Zhao
[Table]
al
[Table]
[Table]
the TTR of successive segments of the generated response with a fixed window size w .
In this paper, h and w were fixed at 0.72 and 4 respectively.
Both MTLD and MATTR are derived based on all generated responses.
ROUGE-L & METEOR.
Both ROUGE-L and
METEOR compares the generated response to the response label.
Computing the ROUGE-L score involves first identifying the Longest Common Subsequence (LCS) between the generated response and response label, followed by computing the harmonic mean of the precision and recall between the LCS and the generated response.
METEOR is similarly based on the harmonic mean between precision and recall calculated based on the generated response and response label, with more emphasis placed on recall.
4.4. Human Evaluation
For human evaluation, we engage five graduate students (native English speakers) to evaluate the Fluency, Diversity and Coherence of the generated responses.
The Fluency criteria measures the naturalness and human-likeness of the generated response.
For the Fluency criteria, the evaluators were told to regard the response in isolation, without regard for the dialogue context.
The Diversity criteria accounts for the diversity on terms of vocabulary in the generated response.
Coherence refers to the contextual coherence of the generated response i.e., the relevance of the generated response in relation to the dialogue context.
The evaluators were given the dialogue context, and told to consider the appropriateness of the generated responses with regard to the dialogue context.
For each example, they were told to compare a response generated by RL
Transformer and a response generated by either the base Transformer, CVAE or SVT.
The superior response in terms of either Fluency, Coherence or Diversity is selected by the evaluator, and either a ‘Win’, ‘Lose’ or ‘Tie’ is assigned to the corresponding model.
The percentage of wins, loses or ties for each pair is then computed.
Each evaluator was given 50 randomly sampled dialogue contexts and the corresponding responses generated by the implemented models for evaluation.
Results and Discussion 5. Quantitative Analysis
Automatic metric scores attained on the DailyDialog and EmpatheticDialogues copora are presented in Table 1.
Human evaluation was only conducted on the DailyDialog corpus.
Results are presented in Table 3.
Based on the results attained, it can be concluded that the performance of the RL Transformer is comparable to that of latent variable models such as the CVAE and SVT.
On the DailyDialog corpus, RL
Transformer outperformed all implemented baselines in terms of Distinct-1 and 2 scores.
On the
EmpatheticDialogues corpus, RL Transformer outperformed all implemented baselines in terms of
Distinct-1, MATTR and MTLD.
When it comes to response diversity, the human evaluation results largely corroborate the automatic metric scores. In terms of Diversity, the RL Transformer achieved a high percentage of ‘Wins’ over the standard Transformer, and a relatively high percentage of ‘Ties’ against the latent variable models. It can also be observed that the RL Transformer also achieved a high percentage of ‘Wins’ over la-
[Table]
I think so , you know there is a great deal of opportunity for business there.
tent variable models in terms of Coherence.
This is corroborated by the qualitative analysis of the responses provided in the following section.
We suspect that low Coherence scores attained by the latent variable models could be partially attributed to the random sampling process.
Since the latent
Gaussian models the potential dialogue response intents, sampled random variables which deviate significantly from the mean could encompass an irrelevant dialogue intent.
When this random variable is fed to the decoder, an incoherent response would likely be generated.
Similarly, it can also be observed that the RL Transformer also achieved a high percentage of ‘Wins’ over latent variable models in terms of Fluency.
This could be potentially attributed to the random linear layers introduced in the RLSA networks, which serve to improve the overall capability of the RL Transformer.
Additionally, as reported in Liu et al.
, we note that the ROUGE and METEOR scores do not correlate with any aspect of human evaluation.
5. Qualitative Analysis
We conduct a qualitative analysis by examining the responses generated by each of the implemented [Table] D [Table]
RLFF(D)
[Table] models. The qualitative analysis largely support observations from our quantitative analysis.
The responses generated by the standard transformer were short, generic and repetitive.
As expected, responses generated by the latent variable models CVAE and SVT as well as our proposed RL
Transformer were noticeably more diverse and less repetitive.
However, numerous responses generated by
CVAE and SVT were relatively unnatural due to the relatively poor fluency and contextual coherence.
A large number responses generated by the latent variable models had grammatical or structural issues, and a significant number were irrelevant in relation to the dialogue context i.e., out of context.
On the other hand, the responses generated by the
RL Transformer were significantly more natural and human-like, displaying far fewer grammatical or structural issues and greater relevance with regard to the dialogue context.
Samples of the generated responses along with the corresponding dialogue contexts are provided in Table 2.
5. Ablation Study
We also conduct an ablation study, using the DailyDialog corpus, to investigate the contributions of each of the RLSA and RLFF components in the encoder and decoder.
Additionally, to examine the importance of the direct links, we implement a variant of the RL Transformer without any direct links.
The results of the ablation study are [Table] presented in Table 4.
In the same table, we also provide the diversity scores for two variants of the
RL Transformer where the randomized layers are initialized via Normal initialization ( N (0 . 0 , 0 . 1) ) and Uniform initialization ( U (0 . 0 , 0 . 01) ) respectively.
Based on the results attained, we can observe that the RLSA network in the encoder, and the
RLSA and ED RLSA networks in the decoder have a relatively large impact on response diversity.
Substituting the RLSA or ED RLSA networks for the standard self-attention network results in a significant drop on all diversity measures.
However, substituting the RLFF in the encoder for a standard feed forward network results in a relatively minor decrease in diversity.
Hence, we conclude that stochasticity introduced in the self-attention networks contribute to overall response diversity to a much larger extent compared to the RLFF network.
Although, it should also be noted that, substituting the standard feed forward network in the decoder with a RLFF network would result in a slightly lower diversity scores.
Also, the importance of the direct links cannot be overstated.
From the results attained by the RL Transformer without links, it is apparent that removal of the direct links would result in extremely low response diversity.
Upon closer inspection of the responses generated by this variant of the RL Transformer, we notice that a majority of the generated responses are short, generic and highly repetitive.
In addition, we present the results attained by varying the number of hidden nodes in the randomized layers in Table 5. Intuitively, this implies varying levels of stochasticity. From Table 5, we can observe that increasing the number of neurons in the randomized layer would generally result in an increase in diversity. This can be attributed to an increase in stochasticity due to an increase in the number of randomized weights.
However, there is a significant drop in diversity when the size of the randomized layers exceed 512.
In this case, the model fails to learn effectively, and generates meaningless, generic, and repetitive responses instead.
Conclusion
In this paper we have proposed a novel RL Transformer which successfully improves response diversity in the task of open-domain dialogue generation.
This is achieved by inducing stochasticity in the self-attention and the feed forward networks of a
Transformer via randomized layers and direct links.
Experimental results on the DailyDialog and EmpatheticDialogues corpora show that, compared to latent variable models, the RL Transformer achieved comparable levels of diversification while further improving on contextual coherence and fluency.
In the future, the RL Transformer can be adapted for related dialogue generation tasks such as personalized, empathetic or topical dialogue generation.