More Diverse Dialogue Datasets via Diversity-Informed Data Collection (2020)

Abstract: Automated generation of conversational dialogue using modern neural architectures has made notable advances. However, these models are known to have a drawback of often producing uninteresting, predictable responses; this is known as the diversity problem. We introduce a new strategy to address this problem, called Diversity-Informed Data Collection. Unlike prior approaches, which modify model architectures to solve the problem, this method uses dynamically computed corpus-level statistics to determine which conversational participants to collect data from. Diversity-Informed Data Collection produces significantly more diverse data than baseline data collection methods, and better results on two downstream tasks: emotion classification and dialogue generation. This method is generalizable and can be used with other corpus-level metrics.

## introduction it is well-documented that neural dialogue models struggle with generating engaging, relevant responses @xcite and often produce banal responses such as "yeah." while this may be an appropriate response to a chitchat conversation, to keep a human participant engaged, diversity of responses is important. diverse models vary the language used and the content referenced, and the generated utterances differ from the most typical conversation responses some proportion of the time. a model which only generates "yeah," "no," and "i don't know" is not diverse and is not be engaging to converse with. past work has improved model diversity with innovation on model architectures and decoding strategies @xcite @xcite @xcite @xcite . we build upon this work to propose a novel method to collect and determine more diverse data to train these models with. our method can be used in conjunction with existing generation-specific model innovations. some prior work on data collection processes has prioritized diversity. for instance, @xcite prompts crowdworkers to choose an underused emotion class to generate dialogue. this work encourages coverage of emotion classes, but does not consider the likelihood that some crowdworkers are better at producing certain types of data than others. this paper introduces diversity-informed data collection (didc), a new strategy for creating a dataset of conversational utterances via selecting which participants' data to include in the collection. the strategy progressively builds up a more diverse sub-corpus from an existing larger collection. the main idea is to grow the sub-corpus by adding conversations sequentially and to assess the contribution of a new participant's utterances to the diversity of the entire sub-corpus. this strategy is also applicable to on-the-fly collection of new datasets via crowdworking or similar methods. we implement didc with three diversity metrics: outlier, entropy, and mean-idf. diversity-informed data collection also provides a new method for finding an upper bound on a current corpus's diversity via a corpus-wide oracle which has access to information about which utterances are most diverse across the corpus. prior work has not used corpus-level statistics to enhance the diversity of the collected data. instead, when collecting data with crowdworkers, researchers have sought more diverse responses by altering the task @xcite or by altering the stimulus @xcite . prior work that trains neural dialogue models has not made use of subsets of existing datasets that exhibit properties of diversity. our experiments show this strategy yields significantly more diverse data than baseline collection processes. it also yields better, more diverse model output on two downstream tasks. additionally, this method can be implemented for other metrics which are defined relative to the corpus. ## related work past work in neural dialogue generation investigates how to improve diversity in conversational responses. additionally, past work in crowdsourcing data collection has explored optimizing crowdsourcing data collection processes. ## diversity-informed data collection we propose a method, diversity-informed data collection, which progressively builds up a corpus, and while doing so, identifies which conversation participants produce more diverse utterances compared to the rest of the in-progress corpus. more formally, our task is to progressively build a subcorpus, sub c , of a given size from a larger, precollected corpus, c, where utterances are tied to ids of specific participants. our approach is aimed at building a diverse subcorpus sub c . our approach chooses which population of participants to collect data from for a given round. this population changes dynamically depending on calculated participant's diversity scores. when utilizing a human-created, pre-existing corpus, we assume responses of the dataset are well-formed and of acceptable quality. with this assumption, we can maximize diversity scores without worrying that quality will be sacrificed for this diversity. however, when using this approach to collect data on-the-fly, additional quality controls may be necessary to ensure diverse data does not come at the cost of quality. we assess two experimental conditions: simulated data collection and corpus-wide oracle upper-bound. simulated data collection is set up to mimic crowdsourcing data collection processes leveraging a large pre-collected corpus, while corpus-wide oracle upper-bound gathers an maximally diverse sub-corpus of utterances. ## evaluation we evaluate the collected corpora both in terms of how diverse each sub-corpus is as well as performance on two downstream tasks: conversation emotion classification and dialogue generation. ## discussion diversity considerations: compared to a random baseline, diversity-informed data collection results in more diverse data than random population, which is shown to be more effective on downstream tasks. future work can explore the effect of simultaneously optimizing multiple desirable measurements of diversity. however, we acknowledge that maximum diversity might not be what is desired and does not always result in the best downstream task performance, as indicated by the low corpus-wide oracle downstream task performance. while we have not examined the tradeoff between diversity and quality, this can be explored in future work. generalizability: diversity-informed data collection is generalizable to metrics other than diversity. concretely, didc should be used when a desired metric (1) can compare one sample (or set of samples) to the in-progress dataset and (2) has variation among the participant population. additionally, diversity-informed data collection can be applied to areas outside of dialogue data collection. for instance, didc could apply to collecting data with different emotions or sentiment. another extension is to a specialized application domain, such as collecting dialogues for educational tutoring purposes, where our method could be used to collect more data from students who generate text consistent with certain types of misconceptions. crowdworking deployment: we evaluated on simulated crowdworking data by leveraging an existing corpus. this choice stems from the desire to test multiple runs of methods in a controlled environment, to reliably determine significance, and to work with data with an assumed level of quality. that said, our approach can be applied to real crowdworking tasks. data can be gathered from several participants in parallel, where crowdworkers are added and offered new tasks or assigned qualifications based on their diversity. if our method is deployed in paid crowdworking tasks, diverse population might be more costeffective. in this particular investigation, we find diverse population requires 100-200 fewer participants than above mean population to create a dataset. due to the time required to train new participants, there is a tradeoff between training a new worker and collecting more data form current participants. caution should be taken in using this method on-the-fly without a quality check. standard quality control methods (e.g., crowdworker qualifications, manual examination, crowdworker verification) should be deployed for from-scratch data collection. crowdworker fairness: another important consideration for a live deployment is the crowdworker's perspective of fairness. because some crowdworkers are retained for more data collection than others, communicating this possibility to crowdworkers is essential @xcite . crowdworking best practices involve disclosing which quality metrics are being used to workers to set clear expectations @xcite . additionally, combining our method with a method which alters the task crowdworkers complete @xcite as opposed to restricting the crowdworking population could be a way to balance fairness with crowdworkers. different task and population combinations could allow for all crowdworkers to participate in more tasks. ## conclusion we propose a method, diversity-informed data collection, which leverages this to produce more diverse datasets than the standard approach, and which performs better on downstream tasks. we define diversity of an utterance compared to the other utterances in a corpus. this allows for measurement of the impact of adding each utterance to the corpus. working under the same assumption that a subset of participants produce diverse data compared to the corpus, our method can be extended to other diversity measures and can be modified to work with other corpus-level metrics.