When More Data Hurts: A Troubling Quirk in Developing Broad-Coverage Natural Language Understanding Systems (2022)

Abstract: In natural language understanding (NLU) production systems, users’ evolving needs necessitate the addition of new features over time, indexed by new symbols added to the meaning representation space. This requires additional training data and results in ever-growing datasets. We present the first systematic investigation into this incremental symbol learning scenario. Our analysis reveals a troubling quirk in building broad-coverage NLU systems: as the training dataset grows, performance on a small set of new symbols often decreases. We show that this trend holds for multiple mainstream models on two common NLU tasks: intent recognition and semantic parsing. Rejecting class imbalance as the sole culprit, we reveal that the trend is closely associated with an effect we call source signal dilution, where strong lexical cues for the new symbol become diluted as the training dataset grows. Selectively dropping training examples to prevent dilution often reverses the trend, showing the over-reliance of mainstream neural NLU models on simple lexical cues.

## introduction broad-coverage natural language understanding (nlu) systems that simultaneously support a wide range of user requests are critical for developing general-purpose natural language interfaces. such systems are already being deployed and reach millions of users worldwide: as of 2021, amazon alexa contains more than 80,000 skills @xcite , and microsoft has deployed a new conversational interface for outlook that uses over 300 composable functions to represent fine-grained semantics in task-oriented user-agent dialogues (semantic @xcite . these broad-coverage nlu systems do not acquire their full capability on day one: new features (e.g., intents or functions), and the new training examples for learning them, are added incrementally. however, there has been little research on the data and learning dynamics during such incremental development. the wide deployment and high cost of such systems make this research critical, and we aim to call attention to this important problem. we consider two prototypical nlu tasks: intent recognition and semantic parsing. nlu is generally concerned with mapping utterances into symbolic meaning representations (e.g., intent labels for intent recognition and sequences of functions/arguments for semantic parsing). we consider the following incremental symbol learning scenario: given a set of existing symbols and their training data, we want to learn a new symbol, which entails adding new labeled data for the symbol. as the system supports more and more symbols, its training data size continually increases. this relates more broadly to domain adaptation @xcite if we consider the new symbols to be a new domain with a very small number of examples, and differs from continual learning @xcite , which learns a new domain using only training data for the new domain (with replay of limited data from previous domains). we instead train models from scratch on all the existing data and the new symbol data simultaneously, which is typical in practice to guarantee accuracy on both old and new features. at first blush, dataset growth may seem positive, in holding with a common assumption of supervised learning: more data is generally better @xcite . however, our analyses reveal a troubling quirk: as the training data size increases, the performance on the new symbol decreases. to investigate this, we simulate the incremental symbol learning scenario by trying to learn a new symbol from a fixed number of new training examples when the overall training data size gradually increases. we examine an english intent recognition dataset @xcite and an english semantic parsing dataset @xcite , testing 5 symbols per dataset. fig. 1 shows the overall test accuracy of our best models (cf. §2) as well as the accuracy on examples containing the new symbol. as the dataset grows, the average test accuracy across all symbols increases monotonically. however, the accuracy on the new symbol's examples generally decreases. this decrease could lead to an increased demand for annotations, with more training examples needing to be collected to achieve adequacy for each new symbol as the dataset grows. class imbalance is one obvious candidate explanation for the performance decrease: as the number of examples n in the dataset grows and the number of examples for the new symbol ŷ stays fixed, the prior probability of the new symbol p(ŷ) ∼ count(ŷ) n in the training data decreases. if this were true, simply upsampling the new symbol's annotations should revert the decrease. with a view to addressing class imbalance, we explore two common solutions: upsampling and group distributionally robust optimization (dro) @xcite . however, neither of them fully attenuates the accuracy drop nor delivers a satisfactory solution. the failure of these solutions leads us to identify a different force associated with the performance decrease, source signal dilution, whereby the reliability of the signal coming from indicative tokens in the user utterances for the new symbol is increasingly diminished in larger datasets. this force is illustrated in fig. 2 . at low data settings, some tokens are highly correlated with the findmanager symbol, but as the dataset grows, the correlation with these tokens is diluted by competing examples that, often by coincidence, contain the same tokens. we show that when diluting examples (shown in red) are removed, the accuracy drop largely disappears, indicating that our state-of-the-art neural nlu models are overly-reliant on simple lexical cues for learning the new symbol. thus, when such lexical cues become less reliable in larger datasets, the models struggle to learn the new symbol. we later argue that while the removal of diluting examples may address the symptoms expressed under increasing dataset sizes, it falls short of an adequate or scalable solution. our main contributions are threefold: ## datasets and models intent recognition and task-oriented semantic parsing share multiple common features. they both translate user utterances to symbols in a certain meaning representation and they are both commonly used in production nlu systems. this makes them ideal testbeds for exploring the dynamics in incremental symbol learning. ## discussion firstly, our results show that, as the training dataset grows, the performance on a new symbol with a fixed number of examples often decreases, suggesting that an increasing amount of data for the new symbol will be required as the dataset expands. the longer the life-span of a system, the more new symbols may be added to it, meaning that as an nlu system extends its coverage, the data annotation cost may spiral up. simple solutions like upsampling and group dro do not suffice in this case: even with these in place, the performance remains decreasing or non-monotonic. our results demonstrate that removing diluting examples largely removes the performance decrease. however, treating the removal of diluting examples as a solution is unsatisfying in three ways. first, we now face a new challenge as we iterate the addition of new symbols. perhaps for the first symbol added, we can successfully remove offending examples from the training set; but as we iterate this process, we may find ourselves removing increasing percentages of our training data, with increasingly disparate subsets of annotations for each symbol. this makes removal an unattractive solution. secondly, while we can intervene on the training distribution, we cannot control the user distribution. fig. 7 shows that the performance on test examples containing trigger tokens but labeled with other intents does decrease after removal. this suggests that the model's ability to capture the full range of user utterances may be reduced on some axes after removal, even if the accuracy on the new symbol is increased. finally, treating removal of diluting examples as a solution means accepting that our models rely overly on (and are fooled by) simple lexical cues when learning from a small number of examples. despite the fact that our models leverage the power of recurrence and multi-head attention, and often use large pre-trained contextualized encoders, this reliance on simple non-contextual cues is reminiscent of simpler non-contextual models like naive bayes classifiers. we would ideally hope that a contextualized representation would be sensitive to the difference between, for example, "who is my manager?" and "invite my manager's wife." we speculate that the model's difficulty in successfully learning examples such as these simultaneously may be explained a sensitivity to the presence and absence of individual tokens (e.g. "manager") rather than a compositional and contextualized analysis of the input; we leave further examination of this hypothesis to future work. despite these unsatisfying observations, the removal of diluting examples also leads to a more promising conclusion, namely that the models we investigate seem to be able to handle a high degree of class imbalance, provided that the source signal is strong enough for the minority class. in the "no dilution" setting, the class imbalance remains exactly the same as for the baseline and group dro settings. the strong performance under the "no dilution" setting suggests that the models can cope with large class imbalances (e.g., 100,000 total examples to 100 new symbol examples) provided that the lexical cues for the minority class in the training data are strong. both the intent classifier and the transductive parser are capable of handling extremely large class imbalance ratios if the sourcetarget mapping is reliable. this helps explain why the models' performance can improve even as the class imbalance increases. ## related work our learning setting relates closely to work on learning with imbalanced data as well as analyses of spurious correlations. @xcite find that over-parameterized networks display a similar trend to our trends on the worst-performing group as model capacity is increased: minority-group performance decreases as overall performance increases. they conclude that large models tend to memorize minority-class data and rely on spurious correlations, leading to worsening accuracy. our work examines the accuracy on specific symbols as the size of the dataset (rather than of the model) grows. different solutions have been proposed to improve generalization on minority data, such as distributionally robust optimization (dro; @xcite @xcite as well as other training and re-weighting strategies @xcite . these solutions are typically applied to image classification tasks; in a space more closely related to nlu, @xcite explore several upsampling and re-weighting strategies for discourse relation classification with imbalanced data, and @xcite investigate the effect of imbalanced data for detecting out-of-scope intents. @xcite argue that simple lexical features, such as the ones we highlight, represent spurious correlations in the data; the models investigated here would be prone to over-reliance on correlations considered spurious by that account, with the removal of diluting examples strengthening them. in a similar vein, @xcite present evidence that natural language interface models rely upon spuriously-correlated features, and present a challenge dataset with such correlations mitigated. @xcite connect a model forgetting an example during training (i.e., the example was correctly classified, and then became incorrectly classified) with spurious correlations and propose a two-stage finetuning solution for improving model robustness. this past work in learning with spurious correlations and imbalanced data has focused on singlelabel multi-class classification problems; we follow this trend in our experiments with intent recognition. however, we go beyond the single-label setting in our semantic parsing experiments, where we investigate class imbalance in a highly structured multi-label multi-class output space. the challenging setting we present differs also from never-ending learning @xcite and domain adaptation/continued training in that for each iteration of the dataset, a new model is trained, rather than continued training on a single model. @xcite investigate few-shot learning for semantic parsing via continued training, where a trained model is exposed to a small set of annotations for a new predicate. while we also attempt to learn from relatively few annotations, we do not adapt learned models, instead simulating the common production setting where models are re-trained on datasets as a whole. previous parsing approaches for smcalflow have followed both modeling paradigms used here: semantic machines et al. (2020) present a seq2seq baseline for smcalflow, following previous work in seq2seq semantic parsing @xcite @xcite . @xcite outperform that baseline with a transductive seq2graph model using explicit type constraints. treating semantic parsing as a a seq2graph transduction problem has proved to be a strong paradigm for parsing abstract meaning representations @xcite , semantic dependencies @xcite @xcite , universal conceptual cognitive annotation @xcite , universal decompositional semantics @xcite @xcite , and gqa @xcite . ## conclusion we examined the effect of a growing dataset on the ease of learning new symbols for nlu, finding that it often becomes harder to learn a new symbol as more data is collected. this trend holds across models and settings, and could pose significant problems as nlu systems increase in lifespan and coverage. we found that the weakening of simple lexical associations as the datasets grow is closely tied to the decrease in performance, indicating that the neural models tested in this study may be overly reliant on simple lexical cues. we end by encouraging others to examine these effects in the problems tested here and also in similar problems, where similar effects are likely to be found.