Iterative Feature Mining for Constraint-Based Data Collection to Increase Data Diversity and Model Robustness (2020)

Abstract: Diverse data is crucial for training robust models, but crowdsourced text often lacks diversity as workers tend to write simple variations from prompts. We propose a general approach for guiding workers to write more diverse text by iteratively constraining their writing. We show how prior workflows are special cases of our approach, and present a way to apply the approach to dialog tasks such as intent classification and slot-filling. Using our method, we create more challenging versions of test sets from prior dialog datasets and find dramatic performance drops for standard models. Finally, we show that our approach is complementary to recent work on improving data diversity, and training on data collected with our approach leads to more robust models.

## introduction crowdsourcing is widely used to collect data, including cases where workers are writing new text, such as questions @xcite , dialog @xcite , and captions @xcite . to avoid repetition of short labels for images, von @xcite proposed using a taboo list, preventing workers from writing labels that previous workers had written. this idea has since been applied to emotion annotation @xcite and word association @xcite . however, in all of these cases the constraint is that there cannot be an exact match with another label. this limits the approach to tasks where workers write a single word or a short phrase. meanwhile, recent work on dialog has found that crowdsourced data can have limited diversity @xcite @xcite . this limited diversity has dramatic consequences, as models trained on such data may not generalize well to unseen or uncommon inputs. we present a generalization of the taboo list idea that can be applied to longer text like sentences. first, rather than features in the taboo list being complete labels, we allow them to be anything, e.g., for intent classification, each feature in the list is a single word that the worker cannot use in their new utterance. to create the taboo list, we propose using a simple model to find over-represented features in the data collected so far. second, rather than having a 1-1 mapping of taboo lists to examples we allow any mapping, e.g., for intent classification we have a taboo list for each intent. to show how this idea improves diversity for longer text, we apply it to crowdsourcing paraphrases for two standard dialog tasks: intent classification and slot-filling. we evaluate our approach in two ways. first, we generate new test sets for several standard intent classification and slot-filling dialog datasets. we find that results on our new test sets are dramatically lower than on the standard test sets, indicating these standard datasets do not provide data of sufficient diversity to train robust models. second, we compare our approach to another recent effort to improve diversity in dialog data @xcite . we collect data with both approaches, a baseline, and a mixture of all three, then evaluate models on all combinations of training and test sets. the mixed approach performs best, indicating that the two approaches complement each other by encouraging different types of diversity. simply collecting enormous datasets may be a way to develop robust models, but it is certainly not sample efficient. without any guidance, workers will mainly write examples that are in the head of the distribution of expressions, only slowly filling in the long tail (if at all). this work provides a method to encourage crowd workers to cover the long tail by using constraints to promote diversity. our results show that by collecting more diverse data, we can produce more robust and therefore useful models. taboo lists: von ahn and dabbish (2004)'s esp game introduced the taboo list idea that we extend. in their game, a pair of players label an image with a single word up to 13 characters long. if they write the same label, it becomes a label for the image and is added to a taboo list for future players looking at that image. of the papers in the acl anthology that cite their work, 38 cite the general idea of a game-with-a-purpose, but do not use the taboo idea; 25 cite the dataset released with the paper; two have the paper in the references but not in the main text; three use the taboo idea in new games. two of the new games use static taboo lists defined by the researchers @xcite , while the third uses the esp game approach, but applies it to a new task @xcite . being based on exact matching limits the range of tasks the taboo idea can apply to. our work overcomes this limitation. concurrent work by @xcite also uses taboo lists to encourage diversity in paraphrases, but they use simple frequency-based taboo word selection, and do not apply their approach to intent classification and slot-filling data. adversarial methods: our work is related to generation of adversarial examples. recent work has shown that inserting text can confuse question answering models @xcite , as can one-word changes to sentences that require world knowledge @xcite , and changing syntax can confuse pretrained models @xcite . the methodology of our first experiment is similar to this work, as we show that models trained on existing crowdsourced datasets perform poorly on the more diverse test sets that we collect. ## taboo data collection we propose a general iterative algorithm for data collection that encourages diversity. by introducing constraints, we can force writers to go beyond the most obvious response to a prompt. this increases the diversity of data, which is crucial for the creation of robust models. the general idea works as follows: ## experiments to demonstrate our approach we consider two experiments. first, we show that existing datasets from prior work are brittle, with training sets that are not sufficiently diverse to train robust models. second, we show how our approach can be used to collect more robust training data from scratch. ## conclusion this paper presents a novel way of guiding data collection away from over-represented areas in the sample space. we show how the approach is a generalization of prior work in crowdsourcing and present a new form of it for dialog data. in experiments on a range of datasets, we show that prior data collection approaches fail to capture diverse examples, leading to brittle models. finally, we show our approach is complementary to other efforts to increase data diversity, producing higher quality datasets. collecting data by combining the standard approach, outlier-based collection, and our taboo-based approach produces better training data that in turn leads to more robust models.