El Volumen Louder Por Favor: Code-switching in Task-oriented Semantic Parsing (2021)

Abstract: Being able to parse code-switched (CS) utterances, such as Spanish+English or Hindi+English, is essential to democratize task-oriented semantic parsing systems for certain locales. In this work, we focus on Spanglish (Spanish+English) and release a dataset, CSTOP, containing 5800 CS utterances alongside their semantic parses. We examine the CS generalizability of various Cross-lingual (XL) models and exhibit the advantage of pre-trained XL language models when data for only one language is present. As such, we focus on improving the pre-trained models for the case when only English corpus alongside either zero or a few CS training instances are available. We propose two data augmentation methods for the zero-shot and the few-shot settings: fine-tune using translate-and-align and augment using a generation model followed by match-and-filter. Combining the few-shot setting with the above improvements decreases the initial 30-point accuracy gap between the zero-shot and the full-data settings by two thirds.

## introduction code-switching (cs) is the alternation of languages within an utterance or a conversation @xcite . it occurs under certain linguistic constraints but can vary from one locale to another @xcite . we envision two usages of cs for virtual assistants. first, cs is very common in locales where there is a heavy influence of a foreign language (usually english) in the native "substrate" language (e.g., hindi or latin-american spanish). second, for other native languages, the prevalence of english-related tech words (e.g., internet, screen) or media vocabulary (e.g., movie names) is very common. while in the second case, a model using contextual understanding should be able to parse the utterance, the first form of cs, which is our focus in this paper, needs cross-lingual(xl) capabilities in order to infer the meaning. there are various challenges for cs semantic parsing. first, collecting cs data is hard because it needs bilingual annotators. this gets even worse considering that the number of cs pairs grows quadratically. moreover, cs is very dynamic and changes significantly by occasion and in time @xcite . as such, we need extensible solutions that need little or no cs data while having the more commonly-accessible english data available. in this paper, we first focus on the zero-shot setup for which we only use en data for the same task domains (we call this in-domain en data). we show that by translating the utterances to es and aligning the slot values, we can achieve high accuracy on the cs data. moreover, we show that having a limited number of cs data alongside augmentation with synthetically generated data can significantly improve the performance. our contributions are as follows: 1) we release a code-switched task-oriented dialog data set, cstop foot_0 , containing 5800 spanglish utterances and a corresponding parsing task. to the best of our knowledge, this is the first code-switched parsing dataset of such size that contains utterances for both training and testing. 2) we evaluate strong baselines under various resource constraints. 3) we introduce two data augmentation techniques that improve the code-switching performance using monolingual data. ## task in task-oriented dialog, the language understanding task consists of classifying the intent of an utterance, i.e., sentence classification, alongside tagging the slots, i.e., sequence labeling. we use the taskoriented parsing dataset released by @xcite as our en monolingual dataset. we release a similar dataset, cstop, of around 5800 spanglish utterances over two domains, weather and device, which are collected and annotated by native spanglish speakers. an example from the cstop alongside its annotation is shown in fig. 1 . note that the intent and slot lables start with in : and sl :, respectively. our task is to classify the sentence intent, here in:get weather as well as the label and value of the slots, here sl:date time corresponding to the span para next friday. moreover, other words are classified as having no label, i.e., o class. we discuss the details of this dataset in the next section. one of the unique challenges of this task, compared with common ner and language identification cs tasks, is the constant evolution of cs data. since the task is concerned with spoken language, the nature of cs is very dynamic and keeps evolving from domain to domain and from one community to another. furthermore, cross-lingual data for this task is also very rare. most of the existing techniques, either combine monolingual representations @xcite or combine the datasets to synthesize code-switched data @xcite . lack of monolingual data for the substrate language (very realistic if you replace es with a less common language) would make those techniques inapplicable. in order to evaluate the model in a task-oriented dialog setting, we use the exact-match accuracy (from now on, accuracy) as the primary metric. this is simply defined as the percentage of utterances for which the full parse, i.e., the intent and all the slots, have been correctly predicted. ## cstop dataset in this section, we provide details of the cstop dataset. we originally collected around 5800 cs utterances over two domains; weather and device. we picked these two domains as they represent complementary behavior. while weather contains slot-heavy utterances (average 1.6 slots per utterance), device is an intent-heavy domain with only average 0.8 slots per utterance. we split the data into 4077, 1167, and 559 utterances for training, testing, and validation, respectively. cs data collection proceeded in the following steps: table. 1 shows the number of distinct intents and slots for each domain and the number of utterances in cstop for each domain. we have also shown the most 15 common intents in the training set and a representative spanglish example alongside its slot values for those intents in table . 2. the first value in a slot tuple is the slot label and the second is the slot value. we can see that while most of the verbs and stop words are in spanish, nouns and slot values are mostly in english. we further calculate the prevalence spanish and english words by using a vocabulary file of 20k for each language. each token in the cstop training set is assigned to the language for which that token has a lower rank. the ratio of the spangish to english tokens is around 1.34 which matches our previous anecdotal observation. this ratio was consistent when increasing the vocabular size to even 40k. . ## model our base model is a bidirectional lstm with separate projections for the intent and slot tagging @xcite . we use the aligned word embedding muse @xcite with a vocabulary size of 25k for both en and es. our experiments showed that for the best xl generalization, it's best to freeze the word embeddings when the training data contains only en or es utterances. we refer to this model as simply muse. we also use sota pre-trained xl models; xlm @xcite and xlm-r @xcite . these models are pre-trained via masked language modeling (mlm) @xcite on massive multilingual data. they share the word-piece token representation, bpe @xcite and senten-cepiece @xcite , as well as a common mlm transformer for different languages. moreover, while xlm is pre-trained on wikipedia, xlm-r is trained on crawled web data which contains more non-english and possibly cs data. in order to adapt these models for the joint intent classification and slot tagging task, we use the method described in @xcite . for classification, we add a linear classifier on top of the first hidden state of the transformer. a typical slot tagging model feeds the hidden states, corresponding to each token, to a crf layer @xcite . to make this compatible with xlm and xlm-r, we use the hidden states corresponding to the first sub-word of every token as the input to the crf layer. ## zero-shot performance bottom part of we observe that having monolingual data of both languages yields very high accuracy, only a few points shy of training directly on the cs data. moreover, in this setting, even simpler models such as muse can yield competitive results with xlm-r while being much faster. however, the advantage of xl pre-training becomes evident when only one of the languages is present. as such, having only the substrate language (i.e., es) is almost the same as having both languages for xlm-r. note that we do not use es data for other results in this paper. obtaining semantic parsing dataset in another language is expensive and often only en data is available. our experiments show a huge performance gap when only using the en data, and thus in this paper, we will be focusing on using the en data alongside zero or a few cs instances. ## data augmentation approaches in this section, we discuss two data augmentation approaches. the first one is in a zero-shot setting and only uses en data to improve the performance on the spanglish test set. in the second approach, we assume having a limited number of spanglish data and use the en data to augment the few-shot setting.