Combining Weakly Supervised ML Techniques for Low-Resource NLU (2021)

Abstract: Recent advances in transfer learning have improved the performance of virtual assistants considerably. Nevertheless, creating sophisticated voice-enabled applications for new domains remains a challenge, and meager training data is often a key bottleneck. Accordingly, unsupervised learning and SSL (semi-supervised learning) techniques continue to be of vital importance. While a number of such methods have been explored previously in isolation, in this paper we investigate the synergistic use of a number of weakly supervised techniques with a view to improving NLU (Natural Language Understanding) accuracy in low-resource settings. We explore three different approaches incorporating anonymized, unlabeled and automatically transcribed user utterances into the training process, two focused on data augmentation via SSL and another one focused on unsupervised and transfer learning. We show promising results, obtaining gains that range from 4.73% to 7.65% relative improvements on semantic error rate for each individual approach. Moreover, the combination of all three methods together yields a relative improvement of 11.77% over our current baseline model. Our methods are applicable to any new domain with minimal training data, and can be deployed over time into a cycle of continual learning.

## introduction virtual assistants are becoming ubiquitous, their expansion fueled by third-party developers who build voice-enabled applications for an increasingly diverse array of domains. however, oftentimes these independent developers don't have the wherewithal to produce sufficient amounts of labeled data, and consequently their applications suffer from poor nlu performance. large pre-trained language models have mitigated but not eliminated the need for domain-specific training data, and therefore unsupervised and ssl techniques continue to form an important direction of work in the field. while a number of such methods have been previously tried in isolation, in this paper we explore the synergistic use of a number of ssl techniques with a view to improving nlu accuracy in low resource settings, specifically for third-party (3p) applications. nlu in our setting is understood as the joint task of intent classification (ic) and named entity recognition (ner). alexa is an ai virtual assistant developed by amazon. by default, alexa is enabled to work across many amazon-built domains, including news, music, calendar, weather, etc. but alexa also includes a third-party skill toolkit, which enables external developers to implement new voiceenabled functionality in the form of external skills, such as quiz and trivia games, food-ordering skills, voice interfaces to a host of devices ranging from vacuum cleaners to automobiles, and so on. developers can build skills by creating skill definitions, consisting of carrier phrases annotated with intent labels and slot labels (a.k.a named entities). for example, for a pizza ordering skill, a developer could provide the following carrier phrase: i would like a size pizza with topping and topping, with an intent such as orderpizza. the values of the size and topping slots are specified in the form of catalogs: catalog(size) = {large, medium, . . .} catalog(topping) = {bacon, peppers, . . .} a full utterance can be realized from the carrier phrase and the slot catalogs, e.g., i would like a medium pizza with peppers and bacon. every token in this utterance would be labeled with an other slot, except for the tokens corresponding to the slots in the carrier phrase, which would be labeled as size, topping and topping respectively. in most cases, skill definitions only contain a few carrier phrases per intent, resulting in very small training datasets that lack linguistic diversity and do not always resemble user queries. the main goal of this paper is to improve nlu model performance by incorporating unlabeled, anonymized and automatically transcribed user utterances from skills into the training pipeline, without human intervention or annotators. we explored three different methods: 1) data augmentation via maximal fstmatching; 2) data augmentation via tri-training ensembles; and 3) injecting auto-encoder sentence embeddings into our baseline dnn (deep neural network) model architecture. the first two techniques automatically obtain labels for live user utterances, which are then used to augment the baseline training data. the third technique performs unsupervised pre-training of an auto-encoder (ae) on user traffic, and this ae is then used to inject sentence embeddings into our models as additional signals. the nlu task we focus on in this paper is the joint task of intent classification (ic) and named entity recognition (ner), also referred to as slot labeling. the rest of this paper is organized as follows: section 2 gives an overview of data augmentation for nlu skills; section 3 introduces the three methods; section 4 details our experimental design; section 5 summarizes the results for all skills, including cumulative results for the three methods. finally, section 6 presents our conclusions. ## related work learning strategies focused on addressing the scarcity of labeled data within a specific domain can be grouped into two approaches: one focused on leveraging models and resources from other domains for which there is a wealth of resources; and approaches that aim to leverage unannotated data for the target domain. in the first group, transfer learning strategies @xcite focus on pre-training unsupervised models on large amounts of unlabeled data and then fine-tuning that model on a small quantity of labeled data. the most recent successful example of transfer learning are bert models @xcite , where a large transformer language model is pre-trained on either the task of masked language modeling or next sentence prediction, and whose encoder can be later fine-tuned as a feature extractor on other nlu tasks @xcite . in the second group, active learning algorithms use individual classifiers or ensembles of classifiers to select data points for human annotation based on different criteria: least-confidence for examples that are assigned low confidence scores by the classifiers @xcite , query-by-committee for examples that are assigned a diverse set of labels by the individual classifiers in the ensemble @xcite , or, more recently, a majority-crf that relies on majority voting from an ensemble of binary classifiers @xcite to select data for new nlu domains. similarly, semi-supervised learning (ssl) @xcite algorithms aim to use models trained on small amounts of annotated data to assign soft labels to unseen examples which can later be incorporated into the training set. self-training @xcite @xcite does this by using the same classifier or a teacher-student pair of classifiers to select and annotate data, whereas tri-training @xcite creates an ensemble of three diverse classifiers that augments unlabeled datasets during an iterative process based on classifier agreement. in this paper, we use (a) transfer learning to pre-train an auto-encoder that is later used to inject sentence embeddings into our ic-ner models (section 3.2) and (b) ssl to augment data with ic-ner annotations obtained by maximal fstmatching and tri-training ensembles (sections 3.1 and 3.3 respectively). on the topic of data augmentation via partial parses, similar to our maximal fst-matching approach, @xcite proposes to extract slot tagging annotations from web logs using a weakly supervised approach based on conditional random fields; and in @xcite the authors use regular expressions to augment training data for the task of stance detection. more recently, @xcite has proposed a semi-supervised framework to leverage unlabelled data and weak classification rules for improved text classification. ## experiment design we compare our methods against our production 3p dnn model, which is depicted in figure 1 . it consists of a pre-trained shared component and a skill-specific component. the shared component is pre-trained on alexa unsupervised traffic and external nlp corpora and computes bpe embeddings. the specific component consists of a large bi-lstm encoder whose input is the shared embeddings and a small bpe embedding followed by a short bi-lstm encoder. the ic layer output takes the summary vectors from both encoders and outputs an ic prediction. the ner layer takes as input a sequence of vectors, where each vector is composed of the concatenation of the two encoder representations and the gazetteer feature of each token. gazetteer features are mappings from sequences of strings to named entities, e.g., the sequence the beatles would be mapped to artist_name. this model is referred to as base in what follows. this study was carried out on 86 english skills from the top 100 alexa skills with the highest amount of user traffic. baseline training datasets for 3p skills consist of ten thousand carrier phrases sampled with repetition. for the 26 skills that include an out-of-domain (ood) intent in their skill definition, we sample the training sets to contain 50% of ood intent examples. for the 86 skills included in this study, the average number of unique carrier phrases in a baseline dataset for a skill is 3,302. our test sets for every skill are comprised of live user traffic annotated in-house with intent and slot labels. on average, the number of utterances in a skill test set is 816, with 425 of those being unique. ## results we use two different metrics, semer (semantic error rate) and irer (interpretation recognition error rate) @xcite . semer combines intent and slot classification accuracy into a single score. it computes a modified edit distance that takes into account the number of substitutions (s), incorrect predictions (i), and deletions (d) in the sequence of slots, and the intent prediction. for a sequence of l tokens, semer is defined as (s + i + d) / (l + 1). the irer of a single utterance is 1 if all the slots and intent are correctly recognized, and 0 otherwise. the irer of a dataset is the fraction of utterances whose irer is 1. ## conclusions we presented three different approaches to improving nlu performance for skills that have limited amounts of annotated data. the two data augmentation approaches, based on maximal fst-matching and tri-training ensembles, yield considerable relative improvements of 5.01 and 7.65%. a third