CST 5: Data Augmentation for Code-Switched Semantic Parsing (2023)

Abstract: Extending semantic parsers to code-switched input has been a challenging problem, primarily due to a lack of supervised training data. In this work, we introduce CST5, a new data augmentation technique that fine-tunes a T5 model using a small seed set (≈100 utterances) to generate code-switched utterances from English utterances. We show that CST5 generates high quality code-switched data, both intrinsically (per human evaluation) and extrinsically by comparing baseline models which are trained without data augmentation to models which are trained with augmented data. Empirically we observe that using CST5, one can achieve the same semantic parsing performance by using up to 20x less labeled data. To aid further research in this area, we are also releasing (a) Hinglish-TOP, the largest human annotated code-switched semantic parsing dataset to date, containing 10k human annotated Hindi-English (Hinglish) code-switched utterances, and (b) Over 170K CST5 generated code-switched utterances from the TOPv2 dataset. Human evaluation shows that both the human annotated data as well as the CST5 generated data is of good quality.

## introduction code-switching (cs) occurs when a speaker alternates between two or more languages within a single conversation. with the increasing ubiquity of voice-based assistants (e.g., google assistant, alexa), the ability to parse code-switched utterances has become one of the key challenges towards building multilingual conversational agents. unfortunately, majority of semantic parsing datasets are in english, with very few datasets exhibiting code-switching. while data collection efforts @xcite 2021) have been made to bridge this gap, collecting such datasets requires time-consuming and expensive human annotations from raters who are proficient in multiple languages, making it difficult to obtain large datasets. another line of work generates artificial code-switched data, either using parallel sentences in the two languages as supervision @xcite or learning a generative language model from large code-switched corpora @xcite . however, these approaches assume the availability of large corpora (>50k sentences), either parallel, monolingual or code-switched in the languages of interest, which may not be available for the domains of interest. in this paper we propose cst5, a strategy to obtain code-switched labeled data by generating code switched utterances from a english semantic parsing dataset, using only a small (≈100 examples) seed set of (english, code-switched) example pairs to learn the transformation. this allows us to effectively leverage existing semantic parsing datasets in english to derive supervision for code-switched semantic parsing. using cst5, we achieve an average 25% improvement on the exact match (em) accuracy, equivalent to having a order of magnitude more training data. a few examples of the cs utterances produced by cst5 are shown in our work is inspired by the growing popularity 1 figure 1 : an overview of our approach. we first use human annotators to annotate utterances. we use these to fine tune a mt5 model for code switching and then use the rest of the monolingual corpus to generate more code switched data. figure 2 : we finetune mt5 on parallel humanannotated data and do inference using the finetuned mt5 model using the monolingual data of using large pre-trained language models (lm) (such as bert @xcite , t5 @xcite etc.) as a means of data augmentation @xcite @xcite . particularly, cst5 relies on mt5 @xcite , a large pre-trained multilingual lm, which is fine-tuned to perform code-switching using the small seed set, and then applied on a large monolingual semantic parsing dataset in english to generate code switched utterances. by mapping the labeled semantic parse for the english utterance, one can recover the semantic parse to the code-switched utterance, thus obtaining a labeled code-switched semantic parsing dataset. we observe that the mt5 model is well suited to the task of code-switching and the generated code-switched utterances were deemed of high quality upon manual inspection by human raters, judged to be over 89% semantically equivalent to the original english utterance, and 98% natural. we use cst5 to generate synthetic code switched data and measure the effect of data augmentation on parser performance with varying seed sizes and across various semantic parsing domains. we observe a 20x data reduction for code switching when using the xxl model. even with a 100 seed examples, our approach matches the performance of a model trained with 2000 examples for the xxl mt5 parser. to summarize, in this work: our work is at the intersection of two areas in nlp: data augmentation for code switching and conversational semantic parsing. ## experimental setup additionally for extrinsic evaluation, we answer the following research questions through our experiments: (a) what effect does adding augmented data to the training set have on the semantic parsing task? (b) what is the effect of varying the size of initial seed set for the data augmentation step on the overall performance?; (c) how does the performance vary across domains? to determine the quality of the augmented data, we trained semantic parsing models for codeswitched utterances. we fine tuned a second mt5 model for this task. we compared the performance of our semantic parsing models with and without augmented data while keeping everything else the same. to study the efficacy of our technique with regards to the seed set, we trained models by varying the seed set size 100, 500, 1000, 2000 and 3000 utterances. further, using an mt5 model finedtuned on this training data, we create augmentation data using the rest of un-annotated utterances from the topv2 dataset as described in section 3.2. although we only release the augmented (modelgenerated) data from the model trained on full 3000 training examples, we created an independent augmented data set for each batch of training data from the training split. for the mt5 models, we use the public checkpoints provided by @xcite . we only roughly tuned the hparams. for the xxl (13 billion parameters) mt5 model training, we use a learning rate of 0.001, batch size of 512 and fine tune for 20k steps. for the small mt5 model (300 million parameters), we fine tune for 200k steps keeping rest of the parameters exactly the same. we checked for overfitting by choosing earlier (10k, 50k steps) checkpoints but obtained best results with the parameters mentioned. models used for comparison for varying the size of the seed set, we trained three mt5 models. human annotated utterances en: will it be [ above ] [ 59 ] in [ celsius ] at [ menlo park ] [ tomorrow morning ] cs: kya [ kal subah ] [ menlo park ] me [ 59 ] [ celsius ] se [ upar ] hoga? en: remind [ me ] [ on september 1 ] that [ its my best friend's birthday ] ! cs: [ september 1 ko ] [ muje ] remind karaye ki [ mere best friend ka birthday he ] model generated utterances en: i need to change [ my ] [ car maintenance ] reminder to [ tuesday 9 am ] cs: mujhe [ mere ] [ car maintenance ] reminder ko [ tuesday subah 9 bajhe ke liye ] badalne ki zaroorat hai en: text [ my ] [ mom ] that [ i will um reach home in ten minutes ] cs: [ meri ] [ ma ] ko text karo ki [ mai um ten minutes me pahuch jaunga ] ## results as seen from figure 3 , we observe that increasing the size of the training data, improves the performance of both models(with and without augmented data set) significantly. while there is a consistent improvement on adding more training data, the improvement itself seems to diminish with more seed set examples. empirically, we observe that the models which make use of the augmented data reaches saturation, i.e. the value after which additional data points don't add to the overall performance of the model, quicker than the models which do not make use of the augmented data. this shows that the synthetic data generated by our technique is useful for the end task of semantic parsing. as seen from figure 3 , the performance of the mt5 model trained using the augmented data was consistently better than that trained without it. the xxl model trained using our technique with a seed small no augmentation small with cst5 augmentation xxl no augmentation xxl with cst5 augmentation 100 500 1,000 2,000 3,000 20 40 60 80 90 100 # training examples exact match acc. alarm event messaging music navigation reminder timer weather the xxl mt5 model was able to produce a significantly higher number of code-switched utterances compared to the small mt5 model, which ultimately led to the significant gain in its performance. the number of generated code-switched utterances post data-filtering can be seen in moreover, after an in-depth domain-wise analysis of the models performance (figure 4 ), we observed that the model struggled with the reminder domain. it is interesting to note that the reminder domain had the highest number of unique intents, 16. also, reminder domain had the highest number of average intents per utterance of 2.66. this collectively led to the low performance of the model on this domain. some of the domains saturated quicker and did not see much improvement upon adding more data points. we can empirically observe that for domains with more inherent difficulty (reminder for example) the value of added data is is more than simpler domains (weather for example). results on cstop additionally, we repeated a similar experiment for cstop @xcite , which contains spanish-english code switched utterances annotated in the top schema. unlike our approach cstop doesn't have parallel data with topv2. since only the weather domain was common between cstop and topv2, we work with the weather domain subset. as we need parallel english data to finetune the mt5 model, we used google translate to generate parallel english data for 100 spanish-english queries. we used this to finetune a mt5 model and generated new synthetic data using topv2 weather domain as the monolingual dataset. for semantic parsing using 100 queries from the training data we observe similar trends as hinglish. we observe an improvement in the parsing performance from 69.2% em to 77.8 % em on the testset using the synthetic code switched data . ## conclusion we proposed cst5, an approach to overcome the scarcity of labeled data for code-switched semantic parsing. cst5 generates code switched utterances from english utterances using a large pretrained lm mt5, and a small number of seed examples. we showed that the generated utterances were of high quality, as determined by human raters in a quality annotation task. by aligning the generated utterances with the semantic parse of the original english utterance, we derived a large supervised dataset for training a code-switched semantic parser using a labeled dataset in english. we did both intrinsic and extrinsic evaluation of the human annotated as well as the synthetic data. human raters found 98% of the synthetic data to be natural and 89% to be semantically equivalent. extrinsically, our experiments also demonstrated that data augmentation using cst5 effectively reduces the data requirements by 20x for the code switched semantic parsing task. to further research in this area, we will release the dataset of over 10k manually annotated hinglish utterances, along with over 170k examples generated using cst5 foot_1 while we applied cst5 to hindi-english code-switching, our approach is general and can be applied to any other language pair that exhibits code-switching. in the spirit of cst5, we believe that using large pre-trained language models to perform data augmentation for other code-switched nlp tasks is an attractive future direction to explore.