Context Matters in Semantically Controlled Language Generation for Task-oriented Dialogue Systems (2021)

Abstract: This work combines information about the dialogue history encoded by pre-trained model with a meaning representation of the current system utterance to realise contextual language generation in task-oriented dialogues. We utilise the pre-trained multi-context ConveRT model for context representation in a model trained from scratch; and leverage the immediate preceding user utterance for context generation in a model adapted from the pre-trained GPT-2. Both experiments with the MultiWOZ dataset show that contextual information encoded by pre-trained model improves the performance of response generation both in automatic metrics and human evaluation. Our presented contextual generator enables higher variety of generated responses that fit better to the ongoing dialogue. Analysing the context size shows that longer context does not automatically lead to better performance, but the immediate preceding user utterance plays an essential role for contextual generation. In addition, we also propose a re-ranker for the GPT-based generation model. The experiments show that the response selected by the re-ranker has a significant improvement on automatic metrics.

## introduction in a conversation, speakers are influenced by previous utterances and tend to adapt their way of speaking to each other @xcite . furthermore, generating the responses that fit well to dialogue context facilitates successful conversation and strengthens the user's impression of spoken dialogue systems (sdss). several previous works @xcite @xcite have explored the impact of previous dialogue information on the generated language in task-oriented dialogue. however, how to efficiently infuse the mr: inform (@xmath0; @xmath1: my friend told me about a place called the carolina bed and breakfast. do you know anything about it? typical nlg contextual nlg it is a 4 star guesthouse. yes, it is a guesthouse with a 4 star rating. dialogue context into a semantically controlled generator for improving the contextual interactive experience is still challenging. such as, the contextual generator proposed in @xcite has no big improvement without the help of an additional re-ranker. the empirical study in @xcite demonstrated that both recurrent and transformer-based seq2seq model can not effectively consider previous dialogue history for generation. in this work, we propose two contextual generators, which both utilize pre-trained models to encode dialogue context. and the experiment results show that context does matter in semantically controlled task-oriented natural language generation (nlg). the function of nlg in task-oriented sds is to generate meaningful output in the form of natural language with the guidance of meaning representation (mr). the mr is a formalism of response semantics and generally represents a dialogue action (da), such as inform or request, along with one or more slots and their values (see the mr in figure 1 ). however, this typical nlg only takes the input mr into account and has no clue on how to adapt to the dialogue history. this results in coarse and flat responses (see the blue part in figure 1 ). to enable the contextual interaction in sds, task-oriented nlg should not only contain the desired mr information, but also have access to the dialogue history. the example shown in figure 1 taken from our experiments: compared with noncontextual generation, the response with context guidance adapts better to the preceding dialogue and is more like a sentence from a real human. in addition, the contextual nlg models are prone to generate more diverse responses. in summary, the main contributions of this paper are as follows: with the powerful multi-context encoding of convert, we also analyse the impact of variable context size on csc-lstm. to the best of our knowledge, we are the first to utilize the pre-trained conversational model convert for contextual generation in task-oriented dialogue system. adapted from gpt-2 @xcite , the user utterance and da guide the contextual generation as context and semantic information respectively. we call contextually and semantically conditioned gpt csc-gpt for short in this paper. the experiments of gpt-based contextual model show that generation benefits from dialogue context, even if only immediate preceding user utterance is taken into account. the remainder of this paper is structured as follows: section 2 shows the related works of our research. section 3 introduces the dataset and the automatic metric scores which are used in this work. section 4 describes our proposed three models: csc-lstm, csc-gpt and bert-r, as well as the experiment details. section 5 shows the experiment results of all models in automatic metrics and human evaluation. the last section 6 concludes and outlines future research. ## related works for task-oriented nlg, semantically controlled neural models play a significant role. @xcite introduced a semantically conditioned model by adding an additional semantic cell in lstm to control the da, which is defined as the combination of intent and slot-value pairs, for generation. @xcite improved the rnnbased generator by considering latent information using the semantically conditional variational autoencoder architecture. as the major advantage and superior performance of pre-trained lms @xcite , @xcite proposed a semantically controlled generation model based on gpt-2; @xcite and @xcite presented an end-to-end task-oriented sds based on the pre-trained gpt-2. even though there has been plenty of works on semantically guided nlg, most approaches fail in utilizing information of the preceding interaction. dušek and jurcicek (2016) stood out as they extended the idea of nlg from mrs by adding one preceding user utterance to their recurrent encoder. however, we are more interested in the influence of bigger context sizes in csc-lstm contextual generation. and the model in @xcite was not able to show any improvement for contextual generation without an additional n-gram match re-ranker. however, our proposed contextual generators outperform both the baselines even without re-ranker and the re-ranker in our work can further highly improve the generation performance on automatic metrics. @xcite made an empirical study to understand how models use the available dialog history for generation and found that both recurrent and transformer-based seq2seq model can not consider previous dialogue history effectively. however, the dialogue history in our work is encoded by pre-trained models and experiments show that the generation in task-oriented dialogue benefits from dialogue history. @xcite also examined the role of context and demonstrated that the generation benefits from the dialogue history. while their approach highly relies on manually pre-defined templates which are costly to create, in this work, all responses are directly generated without the need of any templates. ## dataset and automatic metrics in our work, the automatic metric scores are not only used for performance evaluation, but also used for the bert-r training in section 4.3. hence, we introduce the dataset and the automatic metrics beforehand in this section. ## the proposed models and experiment details in this section, our proposed models (two contextual generators, one response re-ranker) and corresponding experiment details are introduced. ## the experiment results the experimental results of all models introduced in section 4 are presented in this section. to ensure a consistent performance comparison, we compute the metric scores based on the top one generation of all models. ## conclusion and future work in this paper, we propose two contextual generation models: csc-lstm trained from scratch and csc-gpt adapted from pre-trained gpt-2. both integrate dialogue context information into nlg for generating more variant and contextual response in task-oriented dialogue systems. in the experiment of csc-lstm against sc-lstm and csc-gpt against sc-gpt, our proposed contextual models both improve the generation performance in automatic metrics, thus showing that csc-lstm and csc-gpt are able to capture better the contextual needs resulting in a higher similarity to the data. this is further underpinned by the number of variations. more variant responses are generated per da in csc-lstm, while the sc-lstm only generates the same utterances each time. furthermore, the variation size of gpt-based generators is higher than sc-lstm based models. the possible reason is the pre-trained gpt-2 contributes to more diverse responses by default. the human evaluation results in an investigation of the impact of context size for dialogue response generation in csc-lstm indicates that longer context does not automatically result in better performance. however, all variant csc-lstm models have better performance than baseline, which means the immediate preceding user utterance contains the most contextual information for generation. this is also verified in gpt-2 contextual generators, even only immediate preceding user utterance is taken into account, the csc-gpt model outperforms sc-gpt both on automatic metrics and human evaluation. in addition to the above mentioned two contextual models, we also present a re-ranker for csc-gpt contextual model. adapted from pretrained bert, the bert-r continually train on multi-domain dialogues and fine-tune on a multiple regression task to learn the relationship between user context and system response by the metric guidance of bleu-4, meteor, bertscore and convert cosine similarity. finally, the top one generation selected by bert-r has significant superiority in bleu-4, meteor, bertscore and convert cosine similarity compared with top one generation in csc-gpt. this means, that our proposed bert-r works from the guidance of metric scores and can choose the generation with highest score. however, csc-gpt slightly outperforms bert-r in variation size and human evaluation. the possible reason is that the existing automatic metrics still have bias with human judgments @xcite . in the future, we will further explore the performance of bert-r with the guidance of other automatic metrics, which have higher correlation with human judgements. furthermore, there seems to be a link between the variation size metric and the human evaluation scores, which will also be part of future work.