Data Augmentation and Learned Layer Aggregation for Improved Multilingual Language Understanding in Dialogue (2022)

Abstract: Scaling dialogue systems to a multitude of domains, tasks and languages relies on costly and time-consuming data annotation for different domain-task-language configurations. The annotation efforts might be substantially reduced by the methods that generalise well in zero- and few-shot scenarios, and also effectively leverage external unannotated data sources (e.g., Web-scale corpora). We propose two methods to this aim, offering improved dialogue natural language understanding (NLU) across multiple languages: 1) Multi-SentAugment, and 2) LayerAgg. Multi-SentAugment is a self-training method which augments available (typically few-shot) training data with similar (automatically labelled) in-domain sentences from large monolingual Web-scale corpora. LayerAgg learns to select and combine useful semantic information scattered across different layers of a Transformer model (e.g., mBERT); it is especially suited for zero-shot scenarios as semantically richer representations should strengthen the model’s cross-lingual capabilities. Applying the two methods with state-of-the-art NLU models obtains consistent improvements across two standard multilingual NLU datasets covering 16 diverse languages. The gains are observed in zero-shot, few-shot, and even in full-data scenarios. The results also suggest that the two methods achieve a synergistic effect: the best overall performance in few-shot setups is attained when the methods are used together.

## introduction the aim of natural language understanding (nlu) in task-oriented dialogue systems is to identify the user's need from their utterance @xcite . this comprises the following crucial information: 1) intents, what the user intends to do, and 2) (typically predefined) slots, associated arguments of the intent @xcite which need to be filled with specific values. intent detection is often framed as a standard sentence classification task, where every sentence maps to one or more intent classes; slot labelling is typically cast as a sequence labelling task, where each word is labelled with a bio-style slot tag @xcite , see figure 1 . the supervised models for nlu in english are plentiful and achieve extremely high accuracy @xcite . at the same time, porting an nlu system to any new domain and language requires collecting a large indomain dataset, and training a model for the target language @xcite . such in-domain annotations in multiple languages are extremely expensive and time-consuming @xcite , also reflected in the fact that large enough dialogue nlu datasets for other languages are still few and far between @xcite . this in turn creates the demand for strong multilingual and crosslingual methods which generalise well and learn effectively in zero-shot and few-shot scenarios. in this work, we propose two methods to this end: 1) multi-sentaugment, a weakly supervised data augmentation method which improves the capability of current state-of-the-art (sota) dialogue nlu in few-shot scenarios via self-training; 2) layer-agg learns to effectively leverage and combine the knowledge stored across different layers of a pretrained multilingual transformer (e.g., mbert). the main goal of multi-sentaugment is to reduce the required amount of labelled data and manual annotation labour by harvesting the large pool of unannotated data, and carefully selecting relevant in-domain examples which can then be automatically labelled @xcite . in a nutshell, domain-relevant unannotated sentences are first retrieved from a large multilingual sentence bank. the synthetic labels for the data are then generated by a teacher model, previously trained with available annotated data. a final student model is then trained on the combination of synthetically labeled and annotated data. to the best of our knowledge, our work is the first to mine large unannotated monolingual resources in multiple languages to augment data for multilingual dialogue nlu. the goal of layeragg is to leverage useful lexical and other semantic information scattered across layers @xcite of a pretrained multilingual transformer. moving away from the standard fine-tuning practice of using only the representations from the top layer, we hypothesise that the model's cross-lingual capabilities can be increased by forcing it (i) to propagate semantic information from lower layers, as well as (ii) to aggregate/combine semantic information from all its layers. in a nutshell, we propose to use a multilingual encoder with cross-layer transformer, which selects and combines the knowledge from all layers of a pretrained model during fine-tuning. our experiments show that multi-sentaugment gives consistent improvements in few-shot and fulldata scenarios on the two available multilingual dialogue nlu datasets: multiatis++ @xcite and xsid @xcite . the results further indicate that layeragg improves zero-shot performance on the same datasets. finally, since the two methods can be independently applied to sota nlu models, we demonstrate that they yield a synergistic effect: the highest scores on average are achieved with their combination. contributions. 1) multi-sentaugment is a simple yet effective data augmentation approach which leverages unannotated data from large web-scale corpora to boost multilingual dialogue nlu. 2) layeragg is a novel cross-layer attention method which learns to effectively combine useful semantic information from multiple layers of a multilingual transformer. transfer learning with large pretrained multilingual transformer-based language models (lms) such as mbert @xcite and xlm-r @xcite has demonstrated currently unmatched performance in many nlu tasks @xcite @xcite , including intent classification and slot labelling @xcite . fine-tuning a large multilingual lm has become a standard for multilingual nlu @xcite @xcite . however, the excessively high data annotation costs for multiple domains and languages still hinder progress in multilingual dialogue @xcite . in this paper, unlike prior work, we propose to use external unannotated data to mine and automatically label in-domain in-language examples which aid learning in low-data regimes across multiple languages. data augmentation in multilingual nlu, as well as data augmentation methods in nlp in general, aim to produce additional training data automatically, without the need to manually label it. in monolingual english-only settings, english nlu data has been augmented by generating additional data with a large monolingual language model @xcite ) such as bert @xcite or gpt-2 @xcite , or from atomic templates @xcite . in multilingual settings, data augmentation methods for nlu include simple text span substitution and syntactic structure manipulation @xcite . recently, code switching @xcite and generating translations through a pivot language @xcite have also been proposed as data augmentation methods. the previous work relies on (i) additional components such as syntactic parsers or pos taggers, or (ii) parallel and code-switched data. however, they might be unavailable or of low-quality for many (low-resource) languages. in contrast, multi-sentaugment relies on the cheapest and largest resource available: monolingual web-crawled data; it disposes of any dependency parsers and taggers, which makes it more widely applicable. mining knowledge from web-scale data was shown effective in various (non-dialogue) text classification tasks @xcite and in mt @xcite . 1 layer aggregation in pretrained lms. a standard practice is to use the output of the final/top layer of a pretrained lm as input into task-specific classifiers @xcite . at the same time, prior work shows that most of (decontextualised) lexical information @xcite and word-order information @xcite is localised in lower layers of bert. middle layers usually encode syntactic information @xcite while (contextual) semantic information is spread across all the layers of a pretrained lm @xcite , with higher layers capturing increasingly abstract language phenomena @xcite @xcite . @xcite showed that using a weighted combination of all layers works well in cross-lingual settings for a syntactic task of dependency parsing. in addition, they proposed to use layer dropout to redistribute how the information is localised in a fine-tuned bert model. in order to 'unlock' additional semantic knowledge from other layers, we propose an additional transformer encoder with cross-layer attention as a layer aggregation mechanism. we hypothesise that relying only on the representations from the top layer dilutes mbert's lexical and semantic information. moreover, we expect lexically and semantically richer representations to be especially useful for zero-shot settings: aggregated (contextualised) semantic information from lower layers could help correctly identify the intent of the sentence, while lexical information could help identify the slot tag for different languages. 2 ## methodology we assume a standard state-of-the-art approach to dialogue nlu in multiple languages @xcite , based on fine-tuning pretrained multilingual lms on the tasks of intent detection and slot labelling. following @xcite , we fine-tune the pretrained lm in a standard supervised fashion, with task-specific linear layers stacked on top. separate nlu models. the multilingual encoder for each nlu task is fine-tuned separately, and there is no knowledge exchange (but also no noise or destructive inference) between the two tasks. we adopt a standard task-specific fine-tuning setup @xcite . ## experimental setup evaluation datasets comprise two standard multilingual dialogue nlu datasets: multiatis++ @xcite and xsid (van der goot et al., 2021), created by translating monolingual labelled english data into target languages. multiatis++ is a single domain (airline) dataset while xsid covers 7 domains including alarm, weather, music, events and reminder. xsid is an evaluation only dataset, i.e., it contains training data only for english. the statistics of the datasets are presented in large (multilingual) sentence banks. we use the cc-100 dataset @xcite , which comprises monolingual commoncrawl data in 116 languages. for computational tractability with resources at our disposal, we rely on the smaller cc-100-100m dataset, a random sample from the full cc-100 foot_3 spanning 100m sentences in each language. cc-100 covers multiple domains, language styles and variations. multi-sentaugment: setup. unless noted otherwise, we use the laser multilingual sentence encoder (artetxe and schwenk, 2019), pretrained on 93 languages with a sentence similarity objective on parallel data. the similarity threshold θ is set to 0.8. besides the basic setup, (i) we also analyse the impact of the sentence encoder by running experiments with another sota multilingual encoder: labse @xcite ; fine-tuning setup. 1) in the zero-shot setup, we train the model on the english training data and evaluate on other (target) languages. 2) in the few-shot setup, unless stated otherwise, we add 10 target-language examples (i.e., shots) per intent to the english training data. 3) in the full-data setup, we use the entire training set of the target language (without any english data). for unsupervised sentence retrieval in few-shot and full-data setups, we only use the examples in the target language as our query set d l (see §3.1). in all experiments, we evaluate on the validation set after each epoch, and train for 20 epochs with a patience of 5 epochs, with adam (kingma and ba, 2015) as the optimiser, batches of 32; the learning rate is 5e -5, and the warm-up rate is 0.1. we experiment with mbert base and xlm-r base as multilingual encoders. the hyperparameters were set to the values corresponding to those in @xcite . ## results and discussion joint vs separate nlu. we first establish the performance of joint versus separate baseline nlu models. the main results, provided in tables 2 and 3 , indicate that joint nlu training performs better on intent classification while separate taskspecific nlu models are more beneficial on slot labelling. our results corroborate the findings from prior work @xcite @xcite . we suspect that joint training works better for intent classification as sentencelevel representations are enriched with lexical information through the additional slot-labelling loss. at the same time, separate training attains stronger performance in slot labelling as it retains more taskspecific representations for each token. ## analysis and further discussion target language analysis. while both multi-sentaugment and layeragg are language-agnostic techniques per se, the actual transfer results also depend on the linguistic properties of the source and the results are shown in ## conclusion and future work we presented 1) layeragg, a layer aggregation method which learns to effectively combine useful semantic information from multiple layers of a pretrained multilingual transformer, and 2) multi-sentaugment, a data augmentation approach that leverages unannotated web-scale monolingual corpora to reduce manual annotation efforts. our results suggest that both methods, applied with stateof-the-art multilingual dialogue nlu models, yield performance benefits both for intent classification and for slot labelling. the methods obtain consistent gains in zero-shot, few-shot and full-data setups on 2 multilingual nlu datasets spanning 16 languages. in future work, we will investigate further applications of multi-sentaugment in cross-lingual settings (e.g., by mining sentences in languages from the same language family). we will also extend the methods towards truly low-resource languages. the code is available online at: github.com/cambridgeltl/ multisentaugment_layeragg.