Investigating the Representation of Open Domain Dialogue Context for Transformer Models (2023)

Abstract: The bulk of work adapting transformer models to open-domain dialogue represents dialogue context as the concatenated set of turns in natural language. However, it is unclear if this is the best approach. In this work, we investigate this question by means of an empirical controlled experiment varying the dialogue context format from text-only formats (all recent utterances, summaries, selected utterances) as well as variants that are more structurally different (triples, AMR). We compare these formats based on fine-tuned model performance on two downstream tasks—knowledge selection and response generation. We find that simply concatenating the utterances works as a strong baseline in most cases, but is outperformed in longer contexts by a hybrid approach of combining a summary of the context with recent utterances. Through empirical analysis, our work highlights the need to examine the format of context representation and offers recommendations on adapting general-purpose language models to dialogue tasks.

## introduction the bulk of existing work in adapting transformer models to open-domain dialogue represents the dialogue context as the concatenated set of turns in natural language @xcite @xcite . while the selfattention mechanisms of these models are able to capture the context from these flat representations, it remains unclear if this is the best approach @xcite . studying the format of context representation would help improve performance on downstream tasks such as response generation and external knowledge selection and could also potentially inform the pretraining of general-purpose dialogue models. additionally, as the length of conversations increases @xcite , these are truncated based on the limit imposed by the positional encodings on transformers. we also know that not all of the utterances are equally relevant so succinctly representing the relevant information in the context given the current conversation state and filtering out the noise from prior interactions would help to model provide more coherent responses. in this work, we empirically investigate the dialogue context representation in the text space for using sequence-to-sequence models. to prioritize broad coverage, we vary the the format of the context using both natural language-only formats (e.g., using all recent utterances or summaries) as well as formats that are more structurally different (e.g., extracting knowledge triples from the utterances) (section 2) and compare these based on downstream task performance. we find that concatenating all recent utterances is a strong baseline. however, in longer dialogues, combining recent utterances with a summary of the past context obtains the best performance. this shows the benefit of the complementary long and short view of dialogue context. we also observe that improving summary quality and introducing external elements about the coherence of the context result in a further gain of downstream performance. this study and related findings can be extended to combine with elements from the broader definition of context @xcite , such as social cues and guidelines @xcite , which were previously not included in dialogue datasets. ## approach we study the effect of the representation of dialogue context on downstream dialogue tasksknowledge selection and response generation. in order to do so, we run a controlled experiment finetuning sequence-to-sequence models on the two tasks verbalised into the text-to-text setup, while varying only the format in which the dialogue con-text is represented. the first broad category of representations consists of directly using the dialogue utterances. we include the concatenated past dialogue utterances, truncated when necessary, as plaintext representation. this includes all the past turns delineated using a special token when applicable. we also include windows of recent turns where we only use the most recent n utterances as the context. to test if models require only the knowledge items within the dialogue utterances, we extract (subject, object, relation) triples from the utterances as the context. to see if models benefit from more structured information, we convert the utterances into amr graphs @xcite . finally, we examine if the information from the context can be distilled using summarization @xcite @xcite . one method is to convert the utterances from both speakers into an abstractive summary using a separate summarization model. 1 and while a summary might contain all the required high-level information from the dialogue context, it loses the local discourse-level information from recent utterances. to mitigate this, we create a hybrid summary + utterances format by appending the summary with windows of turns. we also include an extractive summary in the form of selected turns from the context using pointwise mutual information, a proxy for relevance, with respect to the most recent turn @xcite . we provide further implementation details about each of the methods in appendix c and illustrate an example converted to each of them in figure 1 . ## results plaintext is a strong baseline, which is outperformed by summaries+utterances on longer dialogues from for both knowledge selection and response generation. when we examine the last turn and 3-turn columns, we see the trend that increasing the window size predictably improves performance, but these lag behind plaintext. this shows that transformers are able to leverage the additional information from more recent utterances in the context. however, we see that plaintext is outperformed by the summary + 5-turn method on the longer dialogue datasets, msc and tcs. this shows that past the limit imposed on current transformer encoders by the positional embeddings, summarizing all available information outperforms a truncated set of recent utterances. finally, we see that summary + 5-turn outperforms summary alone on all the datasets. these findings highlight the complementary long and short views of dialogue context from summaries and recent utterances respectively. ## related work when adapting transformers to dialogue tasks, the most common approach is to simply concatenate dialogue utterances @xcite @xcite @xcite . for longer dialog datasets where the entire conversation cannot be encoded, summaries of past sessions are a helpful way to provide all the relevant information needed to continue the conversation @xcite . while amr graphs have been used to perturb individual utterances in order to evaluate coherence in dialogue @xcite , to the best of our knowledge, amr and knowledge triples have not been used to represent the context. we include them for wider coverage. in the dialogue space, retrieval has largely been used to identify relevant knowledge items to be included for response generation @xcite . prior work has examined matching candidate responses with multiple utterances for selection, the weighting learned in effect attending to 'relevant' turns @xcite , however, we explicitly select turns as a means of representing the dialogue context across both of our open domain dialogue tasks. to our knowledge, ours is the first controlled experiment to evaluate different textual context representation methods for sequence-to-sequence models. ## conclusion in this work, we present an empirical controlled study examining dialogue context representation for transformer models on open-domain dialogue tasks. while concatenating all previous turns, as is often adopted, is a strong baseline, combining summaries of the overarching context with recent utterances yields the best results in longer dialogues. additionally improving the quality of the summaries being used and introducing further background information into the context further improve performance. this provides us with new directions to work on including dialogue summarization and considering the broader definition of context for use in open-domain dialogue. the european chapter of the association for computational linguistics: main volume, pages 300-325, online. association for computational linguistics. kurt shuster, spencer poff, moya chen, douwe kiela, and jason weston. 2021. retrieval augmentation reduces hallucination in conversation. in findings of the association for computational linguistics: emnlp 2021, pages 3784-3803, punta cana, dominican republic. association for computational linguistics. kurt shuster, jing xu, mojtaba komeili, da ju, eric michael smith, stephen roller, megan ung, moya chen, kushal arora, joshua lane, et al. 2022. blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arxiv preprint arxiv:2208.03188. yu wu, wei wu, chen xing, ming zhou, and zhoujun li. 2016. sequential matching network: a new architecture for multi-turn response selection in retrievalbased chatbots. arxiv preprint arxiv:1612.01627. canwen xu, wangchunshu zhou, tao ge, ke xu, julian mcauley, and furu wei. 2021. beyond preserved accuracy: evaluating loyalty and robustness of bert compression. in proceedings of the 2021 conference on empirical methods in natural language processing, pages 10653-10659, online and punta cana, dominican republic. association for computational linguistics. jing xu, arthur szlam, and jason weston. 2022. beyond goldfish memory: long-term open-domain conversation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 5180-5197, dublin, ireland. association for computational linguistics. yang xu. 2021. global divergence and local convergence of utterance semantic representations in dialogue. in proceedings of the society for computation in linguistics 2021, pages 116-124, online. association for computational linguistics. tianyi zhang, varsha kishore, felix wu, kilian q weinberger, and yoav artzi. 2019a. bertscore: evaluating text generation with bert. arxiv preprint arxiv:1904.09675. yizhe zhang, michel galley, jianfeng gao, zhe gan, xiujun li, chris brockett, and bill dolan. 2018. generating informative and diverse conversational responses via adversarial information maximization. advances in neural information processing systems, 31. yizhe zhang, siqi sun, michel galley, yen-chun chen, chris brockett, xiang gao, jianfeng gao, jingjing liu, and bill dolan. 2019b. dialogpt: large-scale generative pre-training for conversational response generation. arxiv preprint arxiv:1911.00536.