Do We Need to Create Big Datasets to Learn a Task? (2020)

Abstract: Deep Learning research has been largely accelerated by the development of huge datasets such as Imagenet. The general trend has been to create big datasets to make a deep neural network learn. A huge amount of resources is being spent in creating these big datasets, developing models, training them, and iterating this process to dominate leaderboards. We argue that the trend of creating bigger datasets needs to be revised by better leveraging the power of pre-trained language models. Since the language models have already been pre-trained with huge amount of data and have basic linguistic knowledge, there is no need to create big datasets to learn a task. Instead, we need to create a dataset that is sufficient for the model to learn various task-specific terminologies, such as ‘Entailment’, ‘Neutral’, and ‘Contradiction’ for NLI. As evidence, we show that RoBERTA is able to achieve near-equal performance on 2% data of SNLI. We also observe competitive zero-shot generalization on several OOD datasets. In this paper, we propose a baseline algorithm to find the optimal dataset for learning a task.

## introduction large scale datasets such as imagenet @xcite in vision, and squad @xcite and @xcite in nlp have accelerated our progress in deep learning. the general trend has been to create large scale datasets for various tasks such as abductive nli @xcite , drop @xcite , and swag @xcite . the process of creating big datasets involves heavy investment in resources, that further increases when models are developed in response to these datasets, and trained to top leaderboards. this makes deep learning research and development inaccessible to * equal contribution communities where resources are scarce. additionally, the heavy computation involved in training models adversely affects the environment on a broader scale @xcite . this leads us to the question: do we always need to create big datasets? we probe this question with motivation from the process by which we learn a certain topic/task. even though we have access to hundreds of materials available online, we do not need to go through all of them in order to learn the specific topic. in fact, we intentionally avoid certain materials which are noisy, distracting, or irrelevant to the topic. humans have deep background knowledge about the world which makes this possible. with the recent developments in language modelling, pre-training on huge datasets have imparted linguistic knowledge to models like bert @xcite and roberta @xcite . with this knowledge, models need not learn everything from scratch; instead they should just learn task specific terminologies such as 'entailment', 'neutral', and 'contradiction' for nli, which might not necessitate the use of big datasets. there are certain other factors that recommend against creating big datasets. a growing number of recent works @xcite @xcite @xcite have exposed the presence of spurious bias in many popular benchmarks. spurious bias represents unintended correlations between input and output (e.g.: the word 'not' is most often associated with the label 'contradiction' @xcite ). spurious bias makes a task easy for models, allowing them to exploit instead of learning generalizable features like humans. models finetuned on these benchmarks fail to generalize in out of distribution (ood) and adversarial settings. since the sources of these spurious biases: data collection, and crowdsourcing are hard to control, carefully selecting a smaller and optimal dataset may be a viable alternative. we propose a baseline in this paper to find the optimal dataset for learning a task. our approach is inspired by the human tendency to first make a rough estimate of the presence of relevant materials by glancing at various parts of an entire set of available materials. after selecting a slice of existing materials in the first phase, they remove redundant/easy/already known/possible distracting content from the slice. finally, they use heuristics based on their background knowledge about the task to sort based on relevance and select the most optimal content based on the priority of the task and availability of time to learn it. we utilize several recently proposed modules in our baseline. we prune snli @xcite to ∼ 2% of its original size using our baseline. our results show that roberta on training with this pruned set achieves near-equal performance on the snli dev set and competitive zero-shot generalization on three ood datasets (i) nli diagnostics @xcite , (ii) stress tests @xcite , (iii) adversarial nli @xcite . our analysis shows that big datasets not only prevent generalization, but also impact iid testset performance. interestingly, we find the annotation of those data to be correct and not noisy. this indicates that, certain data samples might be distracting a model by acting against the inductive bias created by rest of the dataset. our finding opens up the possible existence of such distractors in real datasets, encouraging nlp community to explore the optimal selection of certain samples in a dataset instead of trying to dominate a leaderboard with the entire dataset. ## proposed algorithm we mimic the relevant material selection process in humans to propose algorithm for selecting the optimal dataset necessary to learn a task, as illustrated in figure 1 . we use robotics terminology @xcite to explain the stages of learning (i) coarse action (ii) fine action. algorithm 1 details our approach. we briefly explain each stage below. formalization: let d represent the entire dataset, s represent samples, m be the model, s be the pruned set, e(s), c(s) and p (s) be the evaluation score, correct evaluation score and predictability score of each sample s respectively. in this preliminary work, we just explore the first term of dqi c1 . expanding this to other terms will be the immediate future work. ## conclusion we propose a baseline approach to find the optimal set of samples required to learn a task. our approach mimics humans in identifying relevant materials for learning a task. in the first stage, our algorithm finds a rough estimate as part of the coarse action. the second stage leverages two recently proposed modules aflite (for adversarial filtering of dataset biases) and dqi (for quantifying the quality of data) to perform fine action. we show the efficacy of our baseline by pruning snli to 2% of its original size. our results show that roberta on training with this pruned set achieves near-equal performance on the snlidev set and competitive zero-shot generalization on three ood datasets. our analysis shows that big datasets not only prevent generalization, but also impact iid performance. our findings about distracting samples will encourage community to look for the possible existence of such distractors in real datasets and subsequently explore the optimal selection of samples in a dataset instead of trying to dominate a leaderboard with the entire dataset. studying the effect of our algorithm on model training time, memory footprint, model interpretation research and better understanding of how deep learning models work in general are some of the potential future directions to explore.