Transfer-Free Data-Efficient Multilingual Slot Labeling (2023)

Abstract: Slot labeling (SL) is a core component of task-oriented dialogue (TOD) systems, where slots and corresponding values are usually language-, task- and domain-specific. Therefore, extending the system to any new language-domain-task configuration requires (re)running an expensive and resource-intensive data annotation process. To mitigate the inherent data scarcity issue, current research on multilingual ToD assumes that sufficient English-language annotated data are always available for particular tasks and domains, and thus operates in a standard cross-lingual transfer setup. In this work, we depart from this often unrealistic assumption. We examine challenging scenarios where such transfer-enabling English annotated data cannot be guaranteed, and focus on bootstrapping multilingual data-efficient slot labelers in transfer-free scenarios directly in the target languages without any English-ready data. We propose a two-stage slot labeling approach (termed TWOSL) which transforms standard multilingual sentence encoders into effective slot labelers. In Stage 1, relying on SL-adapted contrastive learning with only a handful of SL-annotated examples, we turn sentence encoders into task-specific span encoders. In Stage 2, we recast SL from a token classification into a simpler, less data-intensive span classification task. Our results on two standard multilingual TOD datasets and across diverse languages confirm the effectiveness and robustness of TWOSL. It is especially effective for the most challenging transfer-free few-shot setups, paving the way for quick and data-efficient bootstrapping of multilingual slot labelers for TOD.

## introduction and motivation slot labeling (sl) is a crucial natural language understanding (nlu) component for task-oriented dialogue (tod) systems @xcite . it aims to identify slot values in a user utterance and fill the slots with the identified values. for instance, given the user utterance "tickets from chicago to milan for tomorrow", the airline booking system should match the values "chicago'', "milan", and "tomorrow" with the slots departure_city, arrival_city, and date, respectively. building tod systems which support new domains, tasks, and also languages is challenging, expensive and time-consuming: it requires large annotated datasets for model training and development, where such data are scarce for many domains, tasks, and most importantly -languages @xcite . the current approach to mitigate the issue is the standard cross-lingual transfer. the main 'transfer' assumption is that a suitable large english annotated dataset is always available for a particular task and domain: (i) the systems are then trained on the english data and then directly deployed to the target language (i.e., zero-shot transfer), or (ii) further adapted to the target language relying on a small set of target language examples @xcite which are combined with the large english dataset (i.e., few-shot transfer). however, this assumption might often be unrealistic in the context of tod due to a large number of potential tasks and domains that should be supported by tod systems @xcite . furthermore, the standard assumption implicitly grounds any progress of tod in other languages to the english language, hindering any system construction initiatives focused directly on the target languages @xcite . therefore, in this work we depart from this often unrealistic assumption, and propose to focus on transfer-free scenarios for sl instead. here, the system should learn the task in a particular domain directly from limited resources in the target language, assuming that any english data cannot be guaranteed. this setup naturally calls for constructing a versatile multilingual data-efficient method that leverages scarce annotated data as effectively as possible and should thus be especially applicable to low-resource languages @xcite . putting this challenging setup into focus, we thus propose a novel two-stage slot-labeling approach, dubbed twosl. twosl recasts the sl task into a span classification task within its two respective stages. in stage 1, a multilingual general-purpose sentence encoder is fine-tuned via contrastive learning (cl), tailoring the cl objective towards slbased span classification; the main assumption is that representations of phrases with the same slot type should obtain similar representations in the specialised encoder space. cl allows for a more efficient use of scarce training resources @xcite @xcite . foreshadowing, it manages to separate the now-specialised sl-based encoder space into slottype specialised subspaces, as illustrated later in figure 2 . these sl-aware encodings are more interpretable and allow for easier classification into slot types in stage 2, using simple mlp classifiers. we evaluate twosl in transfer-free scenarios on two standard multilingual sl benchmarks: multi-atis++ @xcite and xsid @xcite , which in combination cover 13 typologically diverse target languages. our results indicate that twosl yields large and consistent improvements 1) across different languages, 2) in different training set size setups, and also 3) with different input multilingual encoders. the gains are especially large in extremely low-resource setups. for instance, on multiatis++, with only 200 training examples in the target languages, we observe an improvement in average f 1 scores from 49.1 without the use of twosl to 66.8 with twosl, relying on the same multilingual sentence encoder. similar gains were observed on xsid, and also with other training set sizes. we also report large gains over fine-tuning xlm-r for sl framed as the standard token classification task (e.g., from 50.6 to 66.8 on multiatis++ and from 43.0 to 52.6 on xsid with 200 examples), validating our decision to recast the task in twosl as a span classification task. in summary, the results suggest the benefits of twosl for transfer-free multilingual slot labeling, especially in the low-resource setups when only several dozen examples are available in the target language: this holds promise to quicken sl development cycles in future work. the results also demonstrate that multilingual sentence encoders can be transformed into effective span encoders using contrastive learning with a handful of examples. the cl procedure in twosl exposes their phrase-level semantic 'knowledge' @xcite . in general, we hope that this work will inspire and pave the way for further research in the challenging transfer-free few-shot setups for multilingual sl as well as for other nlp tasks. the code for twosl will be available online. ## related work multilingual slot labeling. recently, the sl task in multilingual contexts has largely benefited from the development of multilingually pretrained language models (plms) such as mbert @xcite and xlm-r @xcite . these models are typically used for zero-shot or few-shot multilingual transfer @xcite @xcite . further, the representational power of the large multilingual plms for cross-lingual transfer has been further refined through adversarial training with latent variables @xcite and multitask training @xcite . other effective methods for cross-lingual transfer are translation-based, where either the training data in the source language is translated into the target language or the evaluation data is translated into the source (translate-train and translate-test, respectively; @xcite ; @xcite ). the issues with these methods for sl are twofold. first, the translations might be of lower quality for low-resource languages or any language pair where large parallel datasets are lacking. second, they involve the crucial label-projection step, which aligns the words in the translated utterances with the words in the source language. therefore, (i) applying translation-based methods to sequence labeling tasks such as sl is not straightforward @xcite , (ii) it increases the number of potential accumulated errors @xcite , and (iii) requires powerful word alignment tools @xcite . several methods were proposed to mitigate the issues arising from the label-projection step. @xcite propose to jointly train slot tagging and alignment algorithms. @xcite and @xcite fine-tune the models for post-alignment, i.e., explicitly aligning the source and translated data for better cross-lingual dialogue nlu. these approaches still rely on the availability of parallel corpora which are not guaranteed for low-resource languages. thus, alternative approaches using code-switching @xcite , 2020; @xcite were proposed. all of the above methods assume the availability of an 'aid' for cross-lingual transfer such as a translation model or a bilingual lexicon; more importantly, they assume the existence of readily available taskannotated data in the source language. data-efficient methods for slot labeling. one approach to improve few-shot generalisation in tod systems is to pretrain the models in a way that is specifically tailored to conversational tasks. for instance, convex @xcite fine-tunes only a subset of decoding layers on conversational data. qanlu @xcite and qasl @xcite use questionanswering for data-efficient slot labeling in monolingual english-only setups by answering questions based on reduced training data. in addition, methods for zero-shot cross-lingual contrastive learning have been developed @xcite which allow for efficient use of available annotated data. further, qin et al. (2022) and liang et al. (2022) use code-switched examples to improve performance on intent classification and slot labeling in a zero-shot setting by training on code-switched examples on slot-value, sentence and word levels. unlike prior work, twosl focuses on adapting multilingual sentence encoders to the sl task in transfer-free low-data setups. ## methodology preliminaries. we assume the set of n s slot types @xmath0, . . . , sl ns } associated to an sl task. each word token in the input sentence/sequence @xmath1, w 2 , ..., w n should be assigned a slot label y i , where we assume a standard bio tagging scheme for sequence labeling (e.g., the labels are o, b-sl 1 , i-sl 1 ,. . ., i-sl ns ). 1 we also assume that m sl-annotated sentences are available in the target language as the only supervision signal. the full two-stage twosl framework is illustrated in figure 1 , and we describe its two stages in what follows. ## experimental setup training setup and data. the standard few-shot setup in multilingual contexts @xcite assumes availability of a large annotated task-specific dataset in english, and a handful of labeled examples in the target language. however, as discussed in §1, this assumption might not always hold. that is, the english data might not be available for many targetlanguage specific domains, especially since the annotation for the sl task is also considered more complex than for the intent detection task (van der @xcite @xcite . we thus focus on training and evaluation in these challenging transfer-free setups. we run experiments on two standard multilingual sl datasets, simulating the transfer-free setups: multiatis++ @xcite and xsid @xcite . their data statistics are provided in a current limitation of twosl is that it leans on whitespace-based word token boundaries in the sentences: therefore, in this work we focus on a subset of languages with that property, leaving further adaptation to other languages for future work. input sentence encoders. we experiment both with multilingual sentence encoders as well as general multilingual plms in order to (i) demonstrate the effectiveness of twosl irrespective of the underlying encoder, and to (ii) study the effect of pretraining task on the final performance. 1) xlm-r @xcite ) is a multilingual plm, pretrained with a large multilingual dataset in 100 languages via masked language modeling. 2) multilingual mpnet @xcite is pretrained for paraphrase identification in over 50 languages; the model was specifically pretrained in a contrastive fashion to effectively encode sentences. 3) we also run a subset of experiments with another stateof-the-art multilingual sentence encoder, labse @xcite , to further verify that twosl can be disentangled from the actual encoder. 3 all models are used in their 'base' variants, with 12 hidden-layers and encoding the sequences into 768dimensional vectors. this means that the actual encodings of (s mask , sp) pairs, which are fed to mlps in stage 2, are 1,536-dimensional; see §3. ## results and discussion impact of contrastive learning in twosl on slot representations. further, before delving deep into quantitative analyses, we investigate what 1xph[dpsohv 6orw) /dqjxdjh '( )5 +, 37 75 $9* 0rgho ;/05 psqhwzr&/ psqhwzlwk&/ main results. the results on xsid and multi-atis++ are summarised in a finer-grained inspection of the scores further reveals that for low-data setups, even when exactly the same model is used as the underlying encoder (i.e., xlm-r), twosl offers large benefits over token classification with full xlm-r fine-tuning, see ## conclusion and future work we proposed twosl, a two-stage slot labeling approach which turns multilingual sentence encoders into slot labelers for task-oriened dialogue (tod), which was proven especially effective for slot labeling in low-resource setups and languages. twosl was developed with the focus on transfer-free fewshot multilingual setups, where sufficient englishlanguage annotated data are not readily available to enable standard cross-lingual transfer approaches. in other words, the method has been created for bootstrapping a slot labeling system in a new language and/or domain when only a small set of annotated examples is available. twosl first converts multilingual sentence encoders into task-specific span encoders via contrastive learning. it then casts slot labeling into the span classification task supported by the fine-tuned encoders from the previous stage. the method was evaluated on two standard multilingual tod datasets, where we validated its strong performance across diverse languages and different training data setups. due to its multi-component nature, a spectrum of extensions focused on its constituent components is possible in future work, which includes other formulations of contrastive learning, tuning the models multilingually, mining (non-random) negative pairs and extending the method to crossdomain transfer learning for tod @xcite , especially for rare domains not covered by standard datasets. in this work, we have focused on the sample-efficient nature of twosl. we expect it to be complementary to modular and parameterefficient techniques @xcite . in the long run, we plan to use the method for large-scale fine-tuning of sentence encoders to turn them into universal span encoders which can then be used on sequence labelling tasks across languages and domains. twosl can be further extended to slot labelling with nested slots as well as to other 'non-tod' sequence labelling tasks (e.g., ner) for which evaluation data exists for truly low-resource languages: e.g., on african languages @xcite .