C ross A ligner & Co: Zero-Shot Transfer Methods for Task-Oriented Cross-lingual Natural Language Understanding (2022)

Abstract: Task-oriented personal assistants enable people to interact with a host of devices and services using natural language. One of the challenges of making neural dialogue systems available to more users is the lack of training data for all but a few languages. Zero-shot methods try to solve this issue by acquiring task knowledge in a high-resource language such as English with the aim of transferring it to the low-resource language(s). To this end, we introduce CrossAligner, the principal method of a variety of effective approaches for zero-shot cross-lingual transfer based on learning alignment from unlabelled parallel data. We present a quantitative analysis of individual methods as well as their weighted combinations, several of which exceed state-of-the-art (SOTA) scores as evaluated across nine languages, fifteen test sets and three benchmark multilingual datasets. A detailed qualitative error analysis of the best methods shows that our fine-tuned language models can zero-shot transfer the task knowledge better than anticipated.

## introduction natural language understanding (nlu) refers to the ability of a system to 'comprehend' the meaning (semantics) and the structure (syntax) of human language @xcite to enable the interaction with a system or device. cross-lingual natural language understanding (xnlu) alludes to a system that is able to handle multiple languages simultaneously @xcite . we focus on task-oriented xnlu that comprises two correlated objectives: i) intent classification, which identifies the type of user command, e.g. 'edit_reminder', 'send_message' or 'play_music' and ii) entity/slot recognition, which identifies relevant entities in the utterance including their types such as dates, messages, music tracks, locations, etc. in a modular dialogue system, this information is used by the dialogue manager to decide how to respond to the user @xcite . for neural xnlu systems, the limited availability of annotated data is a significant barrier to scaling dialogue systems to more users @xcite . therefore, we can use cross-lingual methods to zero-shot transfer the knowledge learnt in a high-resource language such as english to the target language of choice @xcite . to this end, we introduce a variety of alignment methods for zero-shot cross-lingual transfer, most notably crossaligner. our methods leverage unlabelled parallel data and can be easily integrated on top of a pretrained language model, referred to as xlm foot_0 , such as xlm-roberta @xcite . our methods help the xlm align its cross-lingual representations while optimising the primary xnlu tasks, which are learned only in the source language and transferred zero-shot to the target language. finally, we also investigate the effectiveness of simple and weighted combinations of multiple alignment losses, which leads to further model improvements and insights. our contributions are summarised as follows: ## related work several approaches to zero-shot cross-lingual transfer exist and can broadly be divided into: a) databased transfer, which focuses on training data transformation and b) model-based transfer that centres around modifying models' training routine. data-based transfer translating utterances for the intent classification task is relatively straightforward so previous works focused on projecting and/or aligning the entity labels between translated utterances. this is followed by standard supervised training with those pseudo-labels and is commonly known as the translate-train method. one of the earliest works still being used for this purpose is fastalign @xcite . it's an unsupervised word aligner trained on a parallel corpus to map each word (thus its entity label) in the source utterance to the word(s) in the target user utterance. projecting the entity labels can also be done with word-by-word translation and source label copying @xcite . a teacher model then weakly labels the target data, which is used to train the final student model. sometimes, this type of label projection is complemented with an additional entity alignment step @xcite . better performance can be achieved by using machine translation with entity matching and distributional statistics @xcite though this can be a costly process for each language. a category of 'word substitution' methods such as code-switching @xcite or dictionaryenhanced pretraining @xcite have also been shown to improve cross-lingual transfer. ## experimental setup datasets three multilingual datasets are used to compare our methods with their most relevant counterparts. the datasets, which are used as standard benchmarks for the xnlu tasks, comprise nine unique languages (de, pt, zh, ja, hi, tr, fr, es, th) from 15 test sets (20,000+ instances in total) featuring diverse examples of users interacting with taskoriented personal assistants designed to test the xnlu capabilities of multilingual models. multilingual task-oriented parsing (mtop) comprises 15k-22k utterances in each of 6 languages (en, de, fr, es, hi, th) spanning 11 domains @xcite . the multilingual task-oriented dialogue (mtod) consists of 43k english, 8k spanish and 5k thai utterances covering 3 domains @xcite . the multilingual atis++ (m-atis) contains up to 4.5k commands in each of 8 languages (en, es, pt, de, fr, zh, ja, hi, tr) featuring user interactions with a travel information system @xcite . xlm our pretrained language model of choice is xlm-roberta @xcite . we use the large (550m parameters) model from huggingface @xcite with a @xmath0,024. training setup we use a minimalist setup that features default settings and components to focus the results on the methods rather than hyperparameter tuning or custom architecture design. we implemented all models with pytorch using fixed hyperparameters between experiments except for mtod, where due to its size, we trained with fewer epochs and a lower learning rate (both 50% lower foot_1 ). terminology henceforth, we refer to models trained with labelled data in each language as target language, the models trained only on english data as zero-shot, our translate-intent method as translate-intent (ti), the scores reported by @xcite as previous sota, our io-only implementation of that model as xeroalign io (xa io ), our contrastive alignment method as contrastive (ctr) and our main method as crossaligner (ca). lastly, the simple sum of alignment losses is referred to as 1+1 and the weighted sum from 3.4 as cov. metrics we use accuracy for intent classification and f-score for entity/slot recognition. in addition, we use an overall score (the average of f-score and accuracy) for model ranking, similar to @xcite ; @xcite . results are shown as averages (mean) over all test sets and datasets, presented in tables 1 and 2 . intent classification is thus evaluated on ∼20,000 diverse user commands and entity recognition on ∼60,000 individual slots from 100+ slot types. for a full breakdown, see tables 4 , 5 and 6 in appendix a.2. ## error analysis in order to contextualise the numbers reported in tables 1 and 2 present the highlights in this section. readers interested in language-specific analysis (including many more examples) are encouraged to read appendix a.1. we focused on errors committed by crossaligner and xeroalign io , which achieved the best individual and combined scores. we sampled 100 random errors from each of the following settings: hindi, french and german from mtop, portuguese, chinese and spanish from m-atis and thai from mtod for a diverse pool of errors. the authors adjudicated with native speakers to categorise mistakes into the following types. error types we discovered two main sources of mistakes: a boundary error occurs when the model predicts more or fewer entity words/tokens than given in the gold annotation. a semantics error occurs when the wrong entity class/type is predicted. models can therefore commit: 1) both errors resulting in poor transfer, 2) a boundary error without a semantic error and vice versa giving us a partial transfer or 3) neither error (a false negative), which we deemed an acceptable transfer. we report individual and average error occurrences as well as transfer type percentages in poor transfer indicates that the prediction error is too serious and unusable (even misleading) in a real-world personal assistant. this is typically due to both a boundary and a semantics error, however, some mistakes can be serious enough alone to result in poor transfer. for example, a boundary error can cause the retrieved name of a dish, person or a location to be incomplete and therefore invalid. a semantics error that classifies '10 secondes' (french) as 'date_time' instead of 'mu-sic_rewind_time' would elicit the wrong agent response thus is unusable. on average, ∼28% of mistakes fall into the 'poor transfer' category. partial transfer is defined as either a boundary or a semantics error where neither is considered a serious problem. such entities could be made usable in a personal assistant application with simple post-processing rules. around 33 percent of errors were deemed to be partially correct. often, this was due to including some adjacent punctuation or an article/preposition as part of the entity or a slightly shorter/longer news headline even though a search engine query with that string would have returned the relevant article. entities such as '24 minat ka' (hindi) versus '24 minat' (24 minutes) exemplify the fact that a disputed entity boundary is the most frequent source of error in this category. on the semantics side, we considered a location partially correct if 'state_name' instead of 'city_name' (for washington d.c.) was predicted, a location was expected and the boundary was accurate. acceptable transfer these examples are 'errors' we considered correct and usable 'as is' because neither the entity boundary nor its semantics were thought to be wrong. on average, we deemed almost 39% of entities acceptable for a realworld personal assistant application with around half of those being down to annotation problems (labels missing or incorrect). in other cases, we accepted predictions that offered a valid alternative e.g. when both 'me' (french) and 'je' (i/me) are present in the user utterance and both refer to the same 'person_reminded'. valid alternatives were predicted but annotated somewhat differently. for example, when the entity boundary was slightly wider 'de ida e volta' (portuguese) instead of 'ida e volta' (round trip) where both entities are correct. similarly, classifying 'salmon' as an ingredient rather than a dish (when 'salmon' is an object of 'prepare') was considered an acceptable transfer. ## conclusions we have introduced a variety of cross-lingual methods for task-oriented xnlu to enable effective zero-shot transfer by learning alignment with unlabelled parallel data. the principal method, crossaligner, transforms english train data into a new language-agnostic task used to align model predictions across languages, achieving sota on entity recognition. we then presented a contrastive alignment that optimises for a small cosine distance between translated sentences while increasing it between unrelated sentences, using orders of magnitude less data than previous works. we proposed translate-intent, a fast and simple baseline that beats previous translate-train sota approaches without error-prone data transformations such as slot label projection. the best overall performance across nine languages, fifteen tests sets and three task-oriented multilingual datasets was achieved by a coefficient of variation weighted combination of crossaligner and xeroalign io . our quantitative analysis investigated which types of auxiliary losses yield the most effective combinations. this resulted in several proposed configurations also exceeding previous sota scores. our detailed qualitative error analysis revealed that the best methods have the potential to approach target language performance as most errors were deemed to be of low to medium severity. we hope our contributions and resources will inspire exciting future work in this fascinating nlp research area. spanish once again, articles, some prepositions and occasionally conjunctions e.g. 'de', 'las', 'y', 'la', 'por' (of, the, and, a, in/for) have caused the majority of boundary errors, most of which are partially acceptable. examples include time 'las 10 a.m.' (10 a.m.) and 'no mas tarde que las' (no later than), journey specifications 'de ida' and 'ida y vuelta' (round trip), periods of day 'la mañana' (morning) and 'la noche' (night), dates 'seis de junio' (june 6th) as well as skipping 'en punto' (o'clock) in 'antes de las 4 p.m. en punto' (before 4 p.m.). these minor errors show that the current sota in cross-lingual zero-shot transfer is close to solving these cases. other errors such as 'conexión' instead of 'con conexión' (with connection) and 'la mañana' rather than 'por la mañana' (in the morning) are more examples of disputed annotations. portuguese predictions closely follow spanish error patterns and reflect the wider issues with articles and prepositions, e.g. 'terça-feira de manhã' (tuesday morning), 'de ida e volta' (round trip), 'cinco de abril' (april 5th) or '5 horas da tarde' (5 p.m.) with 'de' and 'da' (both of ) being the unannotated parts that did not transfer optimally. other boundary mistakes were caused by 'das' (of) and 'as' (the), for example, 'antes das 6 horas da tarde' and 'após as 6 horas da tarde' (before and after 6 p.m). annotations that needlessly punished cross-lingual transfer included 'somente de ida' (one way) where 'somente' (only) was not annotated in the english dataset and 'econômica' (economy), which was annotated as 'class_type' in english, correctly transferred but flagged as wrong. hindi errors have a relatively high number (26) of problematic annotations although most mistakes are caused by the now familiar improper handling of prepositions, articles and/or possessives e.g. 'ke', 'tak', 'ka' ('of', 'by', ''s') in phrases such as '30 minat ka' (30 minutes), 'kitanee der tak' (how long), 'kal ke' (yesterday's), 'aaj ke' (today's) or '1 baje ka' (1 p.m.). transliterated entities i.e. english pronunciation written in devanagari, is the second largest category of transfer problems in hindi, e.g. 'pakrino romaano' (pecorino romano), 'goda cheez' (goda cheese), 'braun aaid garl' (brown eyed girl), 'paindora' (pandora), 'pool leeg' (pool league), 'daayanaasor jooniyar' (dinosaur junior) or 'da most byooteephul moment' (the most beat-iful moment). these are problematic because such entities are neither native to hindi nor are they written in latin alphabet hence may not have been observed in this form during xlm pretraining. thai errors were analysed with a translation service as we were unable to secure a native speaker. even so, we observed boundary errors previously seen in other languages. words such as 'nai' ('of' or 'in', the most frequent cause) and 'bpai' ('in', 'off' or 'to', no direct english translation) were the typical sources of boundary issues, e.g. 'nai sùt sàp-daa née' (this weekend), 'nai wan pút' (wednesday), 'bpai séu xyz' (go buy xyz) and 'nai wan née' (today or on this day). such patterns accounted for more than half of all mistakes. machine translation can also be a source of errors. for example, the word 'reminder' is an entity in english (tag: reminder/noun). it was translated as 'kam dteuuan', however, 'reminder' appears in thai data as 'dteuuan kwaam jam', which the model repeatedly missed, leading to 18 errors for what should be an easy case of zero-shot transfer.