Chasing the Tail with Domain Generalization: A Case Study on Frequency-Enriched Datasets (2022)

Abstract: Natural language understanding (NLU) tasks are typically defined by creating an annotated dataset in which each utterance is encountered once. Such data does not resemble real-world natural language interactions in which certain utterances are encountered frequently, others rarely. For deployed NLU systems this is a vital problem, since the underlying machine learning (ML) models are often fine-tuned on typical NLU data, and then applied to real-world data with a very different distribution. Such systems need to maintain interpretation consistency for both high-frequency utterances and low-frequency utterances. We propose an alternative strategy that explicitly uses utterance frequency in training data to learn models that are more robust to unknown distributions. We present a methodology to simulate utterance usage in two public NLU corpora and create new corpora with head, body and tail segments. We evaluate several methods for joint intent classification and named entity recognition (IC-NER), and use two domain generalization approaches that we adapt to NER. The proposed approaches demonstrate upto 7.02% relative improvement in semantic accuracy over baselines on the tail data. We provide insights as to why the proposed approaches work and show that the reasons for observed improvements do not align with those reported in previous work.

## introduction in academic research, natural language understanding (nlu) tasks are typically defined by creating annotated data, and then that data is used to train and evaluate machine learning models designed to solve that task. in such datasets, each utterance is typically encountered only once. but real-world natural language interactions do not look like that -in the real world, frequency matters. when people interact with each other "in the wild", some things are said often ("time to go to bed!"), others are infrequent to the point of being unique. the same holds for how people interact with digital assistants such as alexa, siri, or google assistant, which we use as the case study in this paper. the backbone of such commercial systems is the task of joint intent classification and named entity recognition (ic-ner) @xcite @xcite . the goal of this task is to identify the intended action (play music, open calendar, etc) and actionable slots (names, places, objects, etc) from a user utterance. the underlying joint ic-ner models must correctly handle both the frequently occurring requests and a long tail of less common entities. but in the common ic-ner corpora such as snips @xcite , there is no way to distinguish between requests for generic entities ("play music from youtube") and requests for a low-frequency entity ("help me locate a game called the master of ballantrae"). ic-ner models are fine-tuned on all training data, and then applied to real-world data with a very different distribution. in order to mitigate this issue, this work proposes a method for creating annotated data which explicitly factors in utterance frequency. we divide an nlu dataset into three disjoint segments: head (most frequent utterances), tail (least frequent utterances) and body (all remaining utterances). in this work, we define a segment as a subset of the dataset with similar characteristics, for example the head segment contains utterances with high frequencies in the real world. we then develop learning strategies which benefit from the token and label distributions in the head, body, and tail segments of the resulting frequency-enriched datasets. we simulate utterance usage patterns using two common public corpora for the ic-ner task: snips @xcite which con- ## limitations obtaining search volumes using the google adwords api cannot disambiguate between different context-based semantic interpretations of the same word, especially when there are no additional tokens to provide context. for instance, search volumes for apple will combine volumes related to the corporation and the fruit, while apple phone and apple juice will return only the relevant search volumes. further, this work did not address availability concerns for tail utterances/entities which may be more expensive or labor intensive to collect and annotate. ## conclusions we presented a methodology to estimate utterance frequency information in public datasets for ic-ner task. we create two new corpora: snipsesv and topesv which use the frequency information to segment the original corpora into head, body and tail segments. we adapt two dg approaches for ic-ner and compute performance on each segment as well as the original test set, which represents an unseen segment. our experiments show improvement in tail entity recognition by each dg approach as well as their combination. our followup analyses validate the segment-invariant representation learning by ot and suggest that dmg provides enhanced generalization using segmentspecific masks. to assist future research in this direction, we will release the snipsesv and topesv datasets used in this work upon publication.