Open-domain Dialogue Generation: What We Can Do, Cannot Do, And Should Do Next (2022)

Abstract: Human–computer conversation has long been an interest of artificial intelligence and natural language processing research. Recent years have seen a dramatic improvement in quality for both task-oriented and open-domain dialogue systems, and an increasing amount of research in the area. The goal of this work is threefold: (1) to provide an overview of recent advances in the field of open-domain dialogue, (2) to summarize issues related to ethics, bias, and fairness that the field has identified as well as typical errors of dialogue systems, and (3) to outline important future challenges. We hope that this work will be of interest to both new and experienced researchers in the area.

## introduction being an empathetic, entertaining, and knowledgeable dialogue partner can be difficult even for humans. unsurprisingly, the task of dialogue generation, i.e., creating a system that is able to hold an intelligent conversation in a way a human would, constitutes a hard challenge for the natural language processing (nlp) community. in recent years, partially due to the development of powerful natural language understanding (nlu) and natural language generation (nlg) models @xcite , the quality of dialogue systems has been improving. systems fall into two broad categories, depending on if they support task-oriented or open-domain dialogues. task-oriented dialogue systems are built for specific purposes, such as booking a flight, and the topic of conversation is limited to the domain of interest. while a narrow scope reduces the complexity of the task, the fact that misunderstandings can have severe consequences adds to it: exact understanding of the user's intentions is crucial. in contrast, open-domain dialogue systems have the ability to talk about a wide variety of arbitrary topics. thus, conversations with opendomain dialogue systems more closely resemble human-human conversations. users often do not have any specific goal beyond enjoying the conversation. over the last few years -boosted by the development of deep learning models for text -the nlp community has seen rapid advances in the area of dialogue generation. a consequence of this success, as well as of the general growth of the nlp community, has been an abundance of publications on the topic: 275 submissions made dialogue systems the fourth largest track at acl 2021 in terms of submitted papers. 1 to assist researchers in keeping up with the fast progress and to provide a starting point for newcomers, we aim at providing a comprehensive overview of what we as a field currently can do (existing research), what we yet cannot do (common errors of dialogue systems) or believe must not do (problems related to ethics, bias, and fairness), and what we should do (open challenges for open-domain dialogue generation). our work complements @xcite , @xcite , and huang et al. (2020) -surveys of dialogue datasets, evaluation techniques, and model architectures, respectively, by providing a holistic view of the field. ## open-domain dialogue generation we use the following definition for open-domain dialogue generation, the task of a social chatbot or socialbot: given zero or more previous dialogue turns between itself and one or more other participants, a system must output a fluent, engaging, and meaningful natural language response. task evaluation. evaluation strategies can be sorted into two broad categories: automatic metrics and human evaluation. automatic metrics are cheap, but do not always correlate well with human judgments @xcite . common metrics for generative systems are perplexity @xcite , bleu @xcite , or dist-n @xcite . for retrieval systems, recall at position k in n candidates (r n @k), mean average precision (map), mean reciprocal rank (mrr) and precision at position 1 (p@1) are used @xcite . human evaluation is expensive, but done frequently, due to a lack of good automatic alternatives @xcite . for instance, @xcite propose to evaluate models by determining from which point in a conversation on one can tell they are not human. a detailed description of open-domain dialogue evaluation goes beyond the scope of this paper. we refer the interested reader to a recent survey on the subject by @xcite . ## open-domain dialogue datasets english datasets. the twitter dataset @xcite consists of roughly 1.3 million twitter conversations with 2 to 243 posts each. @xcite generalize it to the twitter triples corpus, which contains context-message-response triples. the context represents previous dialogue turns, and the response is the user's reply to the message. @xcite mine the meena dataset, which consists of about 867 million context-reply pairs from public posts. each context consists of all previous utterances in the conversation that a reply is participating in. the personachat dataset @xcite consists of chats and personas which are collections of five or more sentences that describe a personality. the dataset also contains revised personas, which are rewritten versions meant to prevent models from using simple word overlap to learn a persona. the chats are dialogues between two workers who each emulate one persona. the target guided conversation dataset @xcite is derived from the personachat corpus and leverages keywords for transitions between turns. the persona information is removed, and a rule-based keyword extractor is used to find keywords. this dataset allows for models to proactively guide the user towards a target topic. similar to the personachat dataset, the wizard of wikipedia dataset @xcite consists of dialogues between two crowdworkers: now, one worker is a "wizard" and the other an "apprentice". the wizard is given text about a topic from wikipedia, and the two are told to converse about it. the wizard labels each of their utterances with a sentence in the article that provides the knowledge used. the dataset is meant to aid creating dialogue systems that are able to use knowledge in retrieving or generating responses. opendialkg @xcite is created by asking two workers to converse about a topic using facts from a kg. one worker is given an entity and told to start a conversation about it. the second worker is given facts and told to respond using the most natural and relevant-sounding fact. as the conversation evolves, kg entities are surfaced to allow workers to use them in their responses. another grounded dataset is the cmu document grounded dataset @xcite . the authors give workers a wikipedia article on a movie, and ask them to converse about it for at least 12 turns. 2 experimental scenarios are considered: in the first, only one worker is given the article, and is told to convince the other person to watch it; in the second, both workers are given the article, and they are instructed to talk about the content. in a similar vein, @xcite create a large corpus of grounded conversations by scraping comments between users on reddit. they consider threads where users are discussing entities found in a linked web document. due to the common use of anchors to relevant information in the urls of linked documents, the authors use this dataset to train systems which can take advantage of machine reading comprehension models. the topical-chat corpus @xcite is a grounded corpus built using 300 entities across 8 topics. two workers are given reading sets, which are a collection of crowdsourced fun facts, washington post articles, and condensed wikipedia lead sections. different reading set configurations allow for a potentially asymmetrical amount of information to be given to each person. conversations are required to have a minimum of 20 turns, and workers are asked to annotate the sentiment of their utterances, where they found the information they spoke about, and the quality of their partner's utterances. the dailydialog dataset @xcite is created by scraping text from conversations held on an english learning website. each utterance is labeled with a dialogue act and an emotion. the empatheticdialogues dataset (rashkin et al., 2019) contains conversations grounded in situation descriptions. to get these situation descriptions, crowdworkers are asked to write about an emotional situation. subsequently, two workers are paired up and given a situation to roleplay. the goal of the dataset is to help to train systems that can identify user emotion from dialogue text. @xcite also give workers roles in order to create the antiscam dataset. it consists of dialogues between crowdworkers, where one worker is assigned the role of an attacker and the other the role of a user. in their conversations, the attacker poses as an amazon customer service agent and attempts to collect the user's information. the persuasion for social good dataset @xcite contains conversations between two crowdworkers, one of whom is trying to convince the other to donate to a specific charity. 300 of these conversations are annotated with one of ten persuasion strategies, or marked as a non-strategy. the objective of collecting this data is to improve the persuasiveness of dialogue agents. ## open-domain dialogue systems we sort approaches into three categories: (1) retrieval systems, which get their responses from a dataset; (2) generative systems, which generate responses automatically; and (3) comprehensive systems, which consist of a dialogue manager (dm), at least one system from the aforementioned categories, and optionally other functional modules. ## training and data augmentation retrieval-based systems are commonly trained with a cross-entropy loss @xcite , comparing a prediction against the gold standard from a training set. as an alternative, using a ranking loss, where a model is trained on distinguishing suitable from unsuitable responses, has been proposed @xcite . in comprehensive systems, the individual components are usually trained separately. several algorithms to train generative systems have been proposed. given a training set @xmath0, c 1 , b 1 ), . . . , (r n , c n , b n )} with n examples consisting of context c i , background information b i , and response r i , models are most commonly trained using maximum likelihood estimation @xcite . however, it has been shown that this encourages boring responses @xcite . as a remedy, several ways to weight training examples have been proposed @xcite . contrastive learning @xcite @xcite ) -where a model is trained to assign higher and, respectively, lower conditional probabilities to positive and negative samples than a reference model -and curriculum learning -during which examples are presented to a model in a specific order -have also been employed @xcite . finally, dialogue systems can also be trained via reinforcement learning @xcite @xcite or adversarial learning @xcite . pretraining. large pretrained models such as bert @xcite or gpt and its successors @xcite @xcite have improved the state of the art for a variety of nlp tasks. pretraining has also been used for open-domain dialogue generation. two different strategies exist: one option is to pretrain a model on large unlabeled corpora to then finetune it on dialogue data. @xcite , for instance, initialize parts of their generative system with a pretrained bert model, and @xcite finetune bert for multi-turn response selection in retrievalbased chatbots. @xcite introduce an english language-learning chatbot based on gpt-2. @xcite condition a gpt-2 model for dialogue generation on several previous conversations of a single individual to get it to use that individual's style. further, plug and play language models consist of pretrained language models in combination with one or more simple attribute classifiers, which control various aspects of its behavior, such as style or dialogue content @xcite . the second option is to pretrain a model on large dialogue corpora, such that it can then be finetuned on out-of-domain dialogue data. dialogpt @xcite ) is such a model. its architecture resembles gpt, i.e., it is a transformer @xcite language model. for training, specifically collected reddit data is used. like gpt, dialogpt is publicly available. the authors also experiment with gpt-2 as a basis for dialogpt and, similar to the work mentioned in the last paragraph, find pretraining on raw text to be beneficial. convert @xcite is another model which is pretrained on dialogue data: pretraining is done on a response selection task using reddit. data augmentation. data augmentation, i.e., the creation of artificial training examples, can help in the low-resource setting. @xcite augment paired dialogue data using unlabeled data in the form of unpaired dialogue data. a dialogue pair consists of a social media post and a corresponding response. their method starts by randomly selecting a sentence from the unpaired dataset. then, posts that are semantically similar to the randomly selected sentence are retrieved from the paired dataset. next, responses corresponding to the posts are collected from the paired dataset. finally, sentences that are semantically similar to the responses are pulled from the unpaired data. each of these newly pulled sentences are matched with the original randomly selected sentence, to create a set of candidate pairs. those candidate pairs are then ranked, and the top-ranked pairs are saved for later use. other approaches differ from the aforementioned in that they do not require unlabeled data. @xcite propose a conditional variational autoencoder as a generative data augmentation model. they combine this with a discriminator, which decides whether the generated responses are suitable for a given query. @xcite design a data augmentation and instance weighting model which is trained using gradient descent and the model's performance on development examples. ## common errors of dialogue systems we now discuss errors common across multiple systems, considering mistakes at the turn level, the conversation level, and the system level. turn level. at the turn level, errors consist mostly of system responses being either ungrammatical or nonsensical. both types of problems are more common in generative systems, as those commit errors seen in other nlg tasks, such as highly repetitive, nonsensical, or insignificant replies @xcite . models which are motivated by semantic similarity may resort to constantly echoing the user, rather than returning a coherent response @xcite . conversation level. problems arising at the conversation level are arguably more substantial than those at the turn level. potential solutions will most likely rely heavily on advancements in other areas of nlp, such as reasoning and information extraction. a common issue consists of replies being fluent, but either not relevant in the overall context of the conversation or too generic @xcite . off-topic replies can often be attributed to a failure to recognize entities or previous dialogue acts. another common problem are answers that are inconsistent across turns @xcite . system level. at the system level, researchers and model developers face the difficulty of incorporating world knowledge and common sense into models @xcite , as models still frequently generate responses that are factually incorrect @xcite . there exists a trade-off between the range of topics a system can cover and the depth of knowledge it can leverage for any individual topic. currently, especially comprehensive systems frequently rely heavily on curated content and static, handwritten conversation paths to talk intelligently and deeply about specific topics. however, the more a system relies on handwritten paths, the more brittle it becomes. similarly, curated content is impossible to scale to a truly open-domain setting. conversely, leaning more towards dynamically structured conversations gives models more flexibility and allows them to cover a wider range of topics, but often results in less meaningful responses. ## ethics, bias, and fairness the nlp research community is becoming increasingly aware of the ethical challenges around the systems we are building, and the area of dialogue generation is no exception to this. we now summarize prior work around safety and unwanted biases. safety. dialogue systems should avoid being unintentionally offensive or harming the user @xcite . therefore, attempts have been made to detect sensitive language around religion, race, violence, or contentious news as well as profanity @xcite . however, how to respond when sensitive topics are being identified is still an open question. as some of these topics shape our identities and our lives, an ideal system might not completely avoid them, and the best response strategy depends on the objectives of the system. when gpt-3 @xcite and blender @xcite detect toxic language in a user utterance, they stop producing output @xcite . while this is an ad-hoc solution, in the long term, a graceful reaction could potentially carry the conversation to healthier places as shown by @xcite . @xcite identify three potentially dangerous behaviors a dialogue system can exhibit: first, it can act as an instigator and provoke the user using negative language, as has infamously happened with the microsoft tay chatbot. second, even if a system exclusively uses non-harmful language, it can cause harm to the user by being a so-called yea-sayer, i.e., by being overly eager to agree with the user on wrong or inappropriate statements @xcite . third, a dialogue system can unintentionally impose as an expert and provide harmful advice. biases. an abundance of recent work has shown that nlp models are learning undesirable biases from the data they are being trained on @xcite @xcite . dialogue systems are no exception to this: @xcite investigate fairness in dialogue models and find that dialogue models exhibit significant prejudice against some genders and races. they propose two debiasing methods based on data augmentation and word embeddings regularization. @xcite point out that there are three types of gender bias in chat bots: the first one being due to the gender of the person that speakers are talking about, the second being due to the gender of the speaker, and the last being due to the gender of the addressee. liu et al. (2020b) aim at mitigating the former via adversarial learning. similarly, dinan et al. (2020a) propose to reduce gender bias via data augmentation, targeted data collection, and biascontrolled training. barikeri et al. (2021) introduce redditbias, a dataset grounded in conversations from reddit, which enables the measurement and mitigation of gender, race, religion, and queerness bias, and use it to explore dialogpt with and without debiasing. 8 open challenges for future research model evaluation and analysis. surveying research on open-domain dialogue generation (cf. section 4) as well as research on system evaluation (finch and choi, 2020), it is clear that a good automatic metric (or even manual evaluation strategy) has not yet been found. what the field needs are metrics that (1) evaluate different aspects of dialogue systems (cf. furthermore, while entire surveys are necessary to summarize work on the analysis of bert @xcite , we still know little about what dialogue systems, including dialogpt @xcite , learn from their training data. prior work on the analysis of dialogue models (with the exception of still non-exhaustive investigations of their biases) is limited; e.g., @xcite . we argue that learning more about dialogue models, which are likely to directly interact with users, is crucial. we should investigate the following: (1) what world knowledge do models acquire during training? (2) what linguistic knowledge do dialogue models learn? (3) which potentially harmful biases do models learn from real-world data? multi-party dialogue. how to extend systems to handle multi-party dialogue, as posed by @xcite , remains an underexplored area of research. having such systems will potentially contribute to creating richer social interactions in both online and offline communities. it will further increase our understanding of the dynamics behind turn taking @xcite . ## conclusion recent years have seen a drastic improvement in the quality of open-domain dialogue systems as well as in the amount of research in the area. therefore, we first presented an overview of the state of the field of nlp for open-domain dialogue. then, we outlined important future challenges: better model evaluation and analysis, multi-party dialogue, and multilingual dialogue.