Continuous Model Improvement for Language Understanding with Machine Translation (2021)

Abstract: Scaling conversational personal assistants to a multitude of languages puts high demands on collecting and labelling data, a setting in which cross-lingual learning techniques can help to reconcile the need for well-performing Natural Language Understanding (NLU) with a desideratum to support many languages without incurring unacceptable cost. In this work, we show that automatically annotating unlabeled utterances using Machine Translation in an offline fashion and adding them to the training data can improve performance for existing NLU features for low-resource languages, where a straightforward translate-test approach as considered in existing literature would fail the latency requirements of a live environment. We demonstrate the effectiveness of our method with intrinsic and extrinsic evaluation using a real-world commercial dialog system in German. Beyond an intrinsic evaluation, where 56% of the resulting automatically labeled utterances had a perfect match with ground-truth labels, we see significant performance improvements in an extrinsic evaluation settings when manual labeled data is available in small quantities.

## introduction 1.motivation and background voice-controlled personal assistants such as amazon alexa or google assistant have scaled to a large number of languages and see a constant influx of new functionalities that are exposed via the natural language interface. as a result, they have seen much interest around the development of multi-lingual and cross-lingual learning techniques that take this setting into consideration. beyond a setting where no target language data is available (language expansion, or crosslingual bootstrapping), ongoing development also involves use cases where new functionalities from a resource-rich language (typically english) as the source language have to be integrated into existing training sets in the target language (feature expansion), or even settings where target language training data of current functionalities exists in small quantities, but accuracy falls short of its aim and an influx of unlabeled data in the target language exists and could be used for continuous model improvement (feature improvement). in this work, we consider feature improvement use case for natural language understanding (nlu) in low-resource languages. we define the task of nlu as the combination of: (1) intent classification (ic), which classifies an utterance into a fixed set of intent labels (e.g. applianceoff), and (2) slot labeling, which classifies slot values into a predefined set of slot types (e.g. songname) @xcite . for example, as shown in figure 1 , a valid nlu annotation for the english utterance "turn off the light in the hallway" would be: applianceoff: (turn, actiontrigger), (off, actiontrigger), (light, device), (hallway, location), where applianceoff is the intent label, and actiontrigger, device and location are the slot types. we leverage machine translation to automatically annotate unlabeled utterances with intent and slot labels. collecting and labeling data for nlu is an expensive and time-consuming process, hardly scalable to an increasing number of languages without automation. ## method given unlabeled utterances in a target language e.g. german (de), for example "mach das licht im flur aus", our goal is to automatically annotate them with an intent label, and slot types for every token, as shown in the example in figure 1 . to this end, we consider the pipeline shown in figure 2 , which consists of three components: (1) machine translation system, (2) nlu model, and (3) label projection model. first, the mt system translates the unlabeled utterances into a reference language e.g. english. for the german utterance above, a valid english translation would be "turn off the light in the hallway". note that we do not make any assumption on the architecture of the mt system, be it statistical or neural, or on the way it is trained. we assume, however, a label projection model trained on the same data as the mt system. in a standard mt bootstrapping setting @xcite this is usually a word alignment model, either embedded in the mt system itself (as in phrase-based mt we used in section 3) or trained as a stand-alone component. after translation, we use an english nlu model on the translated utterances in order to get predictions for the intent label and the slot types. foot_0 for the example above, the result of this step would be the following annotated utterance: [applianceoff turn/actiontrigger off /actiontrigger the/other light/device in/other the/other hallway/location]. finally, we use the word alignment model to project the slot types from the (en) labeled utterances onto the unlabeled (de) utterances. for example, if the two words 'light' and 'licht' are aligned, the slot label of 'light' is copied over onto 'licht'. in our experiments (section 3), we make use of alignment models trained for the mt system to avoid building standalone alignment models. for the intent label, we simply copy it over from the english labeled utterance to the german unlabeled one. for reasons of simplicity and better interpretability, we used statistical machine translation (smt) as well as linear models (crf and maximum entropy) for the nlu component, however we believe that the improvements gained with this method would carry over to a case where neural mt and transformer-based nlu components are used. ## experimental setup in our experiments, we translate target-language (german) utterances from live conversational agent usage using an existing mt system ( §3.1), tag these using the english nlu system ( §3.2) and project the labels back onto the target language using word alignments. we report results using first intrinsic evaluation (how well does the translate-test approach perform in labeling the utterances?) and then a full evaluation in a feature improvement setting, and we evaluate these using a semantic error rate metric (semer, §3.3). ## conclusion this paper presents a new method to automatically annotate utterances with intents and slot types, leading to faster and cheaper early improvement of features. our method harnesses existing mt, english nlu and word alignment models which have been trained on general-domain data but adapted to our specific use case through preprocessing and fine-tuning. intrinsic evaluation results show that a translate-test approach is a viable way to get data labels in a way that is independent from the target language production system, whereas our extrinsic evaluation results suggest that the approach is es-pecially useful when a given feature has not seen extensive use yet. we plan to address in future work whether certain properties of a given feature can predict the viability of a translate-test approach in general and data augmentation with translated examples in particular, and whether the use of neural machine translation models would suggest modifications to this approach, as translations are often better but alignment results can be less clear-cut.