Training data reduction for multilingual Spoken Language Understanding systems (2021)

Abstract: Fine-tuning self-supervised pre-trained language models such as BERT has significantly improved state-of-the-art performance on natural language processing tasks. Similar finetuning setups can also be used in commercial large scale Spoken Language Understanding (SLU) systems to perform intent classification and slot tagging on user queries. Finetuning such powerful models for use in commercial systems requires large amounts of training data and compute resources to achieve high performance. This paper is a study on the different empirical methods of identifying training data redundancies for the fine tuning paradigm. Particularly, we explore rule based and semantic techniques to reduce data in a multilingual fine tuning setting and report our results on key SLU metrics. Through our experiments, we show that we can achieve on par/better performance on fine-tuning using a reduced data set as compared to a model finetuned on the entire data set.

## introduction in recent years, a variety of smart voice assistants such as apple's siri, samsung's bixby, google home, amazon echo, tmall genie, have been deployed and achieved great success. these voice assistants facilitate goal-oriented dialogues and help users to accomplish their tasks through voice interactions. one component of such spoken language understanding (slu) systems is natural language understanding (nlu) which aims to extract the intent of the query (intent classification) and semantically parse the user utterance (slot tagging). as an example, if a user requests "play madonna" to the voice assistant, slu would classify the intent * equal contribution † work done while interning with alexa ai as playmusic with slot filling of tokens "play" as action and "madonna" as artist. as in many other language processing fields, pretrained language models have seen major success for natural language understanding. pre-trained language models @xcite @xcite are generic language models learned in a semi-supervised fashion whose underlying large scale knowledge is then leveraged for fine-tuning towards down-stream tasks @xcite . bert @xcite is one such example of a language model based on the transformer network architecture @xcite , pre-trained on a corpora of 3300m words extracted from publicly available unannotated data and then fine-tuned on smaller amounts of supervised data for specific tasks, relying on the induced language model structure to facilitate generalization beyond the annotations. it provides powerful and general-purpose linguistic representations, triggering strong improvements and significant advances on a wide range of natural language processing tasks. @xcite also observed the success of bert to jointly learn intent classification (ic) and slot filling (sf) tasks by leveraging pre-trained representations. leveraging this joint ic and sf set up to interpret user utterances in commercial slu systems requires large volume of annotated training data (ezen-can, 2020), compute resources (such as gpus) and model build time to cover the variability of customer utterances. such resources (human and compute) are not only expensive but also time consuming, unscalable and may not be fully optimized to achieve the same performance. in this paper, we perform an empirical analysis on identifying subsets of multilingual training data which can achieve on par or better performance as compared to the same model trained on the full data-set for the same task. in particular, we fine-tune a lean-bert sized model (conneau et al., 2020), on the ic-sf task for hindi and english slu data. such mutlilingual large scale pre-training is known to effectively promote crosslingual generalization @xcite giving rise to an opportunity to exploit latent space similarities of such languages to identify redundancies in training data for fine-tuning. we experiment with various semantic and rule based data reduction approaches and report fine-tuning performance on key slu metrics. ## related work finding the right data reduction technique for bert fine tuning while maintaining evaluation performance can be considered as a part of two major classes of problems -fine tuning regime in the low resource setting to leverage insights from incorporated best practices, and the few shot classification class of problems where the model is trained using only a few samples from each class. note that the above two classes of problems are not disjoint and is concurrently explored in this work. ## method in this paper, we first describe the slu architecture used for the ic-sf task and the four methods for data reduction. ## results & discussion for each domain, we build four slu models trained on the combined english and hindi data, each named after the data reduction approach applied to the training data fed to it: baseline, unique, log n, clustering 70%, and singular score. we report the performance for each model on semer and ic accuracy metrics in ## conclusion in this paper, we investigated various inexpensive approaches for identifying data redundancy in training data used to fine-tune bert based models in slu systems for the ic and sf task. to the best of our knowledge, this work is the first step in the direction of identifying inexpensive techniques to fine-tune bert model without affecting offline metrics. we presented empirical results on a realworld slu dataset, showing that data reduction techniques benefit semer and intent classification metrics. in particular, we proposed a novel data redundancy identification and reduction technique which we call the singular score approach. this method helps jointly filter utterances based on their frequency and semantic representation and also helps achieve one of the best results among the techniques experimented with. future work may target more complex forms of identifying training data redundancy such as influential instances @xcite or active learning @xcite