Model and Data Transfer for Cross-Lingual Sequence Labelling in Zero-Resource Settings (2022)

Abstract: Zero-resource cross-lingual transfer approaches aim to apply supervised modelsfrom a source language to unlabelled target languages. In this paper we performan in-depth study of the two main techniques employed so far for cross-lingualzero-resource sequence labelling, based either on data or model transfer.Although previous research has proposed translation and annotation projection(data-based cross-lingual transfer) as an effective technique for cross-lingualsequence labelling, in this paper we experimentally demonstrate that highcapacity multilingual language models applied in a zero-shot (model-basedcross-lingual transfer) setting consistently outperform data-basedcross-lingual transfer approaches. A detailed analysis of our results suggeststhat this might be due to important differences in language use. Morespecifically, machine translation often generates a textual signal which isdifferent to what the models are exposed to when using gold standard data,which affects both the fine-tuning and evaluation processes. Our results alsoindicate that data-based cross-lingual transfer approaches remain a competitiveoption when high-capacity multilingual language models are not available.

## introduction sequence labelling is the task of assigning a label to each token in a given input sequence. sequence labelling is a fundamental process in many downstream nlp tasks. currently, most successful approaches for this task apply supervised deep-neural networks @xcite @xcite . however, as it was the case for supervised statistical approaches @xcite , their performance still depends on the amount of manually annotated training data. additionally, deep-neural models still show a significant loss of performance figure 1 : in the data-based transfer approach we translate and project the labels of the gold data into the target language, and use the resulting silver data to train a model for the target language. in the model-based transfer approach we train a model with gold data in english and use it in a zero-shot setting in the target language. when evaluated in out-of-domain data @xcite . this means that to improvie their performance, it would therefore be necessary to develop very costly manually annotated data for each language and domain of application. thus, considering that for most of the languages in the world the amount of manually annotated corpora is simply nonexistent @xcite , then the task of developing sequence labelling models for languages and domain-specific tasks, for which supervised data is not available, remains a challenge of great interest. this task is known as zero-resource crosslingual sequence labelling. data-based cross-lingual transfer methods aim to automatically generate labelled data for a target language. previous works on data-based transfer have proposed translation and annotation projection as an effective technique for zero-resource cross-lingual sequence labelling @xcite . in this setting, as illustrated in figure 1 , the idea is to translate gold-labelled text into the target language and then, using automatic word alignments, project the labels from the source into the target language. the result is an automatically generated dataset in the target language that can be used for training a sequence labelling model. the emergence of multilingual language models @xcite allows for model-based cross-lingual transfer. as figure 1 illustrates, using labelled data in one source language (usually english), it is possible to fine-tune a pre-trained multilingual model that is directly used to make predictions in any of the languages included in the model. this is also known as zeroshot cross-lingual sequence labelling. in this work we present an in-depth study of both approaches using the latest advancements in machine translation, word aligners and multilingual language models. we focus on two sequence labelling tasks, namely, named entity recognition (ner) and opinion target extraction (ote). in order to do so, we present a data-based crosslingual transfer approach consisting of translating gold labeled data between english and 7 other languages using state-of-the-art machine translation systems. sequence labelling annotations are then automatically projected for every language pair. additionally, we also produced manual alignments for those 4 languages for which we had expert annotators. after translation and projection, for the data-transfer approach we fine-tune multilingual language models using the automatically generated datasets. we then compare the performance obtained for each of the target languages against the performance of the zero-shot cross-lingual method, consisting of fine-tuning the multilingual language models in the english gold data and generating the predictions in the required target languages. the main contributions of our work are the following: first, we empirically establish the required conditions for each of these two approaches, data-transfer and zero-shot model-based, to outperform the other. in this sense, our experiments show that, contrary to what previous research sug-gested @xcite , the zeroshot model-based approach obtains the best results when high-capacity multilingual models including the target language and domain are available. second, when the performance of the multilingual language model is not optimal for the specific target language or domain (for example when working on a text genre and domain for which available language models have not been trained), or when the required hardware to work with high-capacity language models is not easily accessible, then datatransfer based on translate and project constitutes a competitive option. third, we observe that machine translation data often generates training and test data which is, due to important differences in language use, markedly different to the signal received when using gold standard data in the target language. these discrepancies seem to explain the larger error rate of the translate and project method with respect to the zero-shot technique. finally, we create manually projected datasets for four languages and automatically projected datasets for seven languages. we use them to train and evaluate cross-lingual sequence labelling models. additionally, they are also used to extrinsically evaluate machine translation and word alignment systems. these new datasets, together with the code to generate them are publicly available to facilitate the reproducibility of results and its use in future research. foot_0 2 related work ## translation and projection method our data-based cross-lingual transfer method to perform cross-lingual sequence labelling is the following: we assume our source language to be english, for which we have train and development data. furthermore, we also assume that the only goldlabelled data available for the target language is the evaluation set. in this setting, we automatically generate data for the target language by translating the gold-labelled english data. then we project the gold labels from the source sentences to the translated sentences by leveraging automatic word alignments. given a sentence @xmath0, ..., x n ⟩ with length n in the source language and a translated sentence @xmath1, ..., y m ⟩ with length m in the target language, we use a word aligner to find a set of pairs @xmath2, y j ⟩ : x @xmath3, y @xmath4, y j ⟩ y i is the lexical translation of x j . next, given a sequence @xmath5, ..., x b ⟩ ∈ x labeled with a category c we will label the sequence @xmath6, ..., y d ⟩ ∈ y with category c if {∀y @xmath7: (⟨x i , y j ⟩ ∈ a)}. in other words, if a word labelled with a category in the source sentence is aligned to a word in the target sentence, we label the target word with the category from the word in the source sentence. figure 2 illustrates our method. when projecting annotations we find two main problems: split annotations and annotation collision. in the first case, a labeled sequence in the source sentence is split into multiple sequences in the target sentence. this happens when the alignment for a word is missing. in this case, we merge the sequences in the target sentence if the gap between them is just one word. if we still end up with multiple sequences, we choose the longest one. in the annotation collision case, a word in the target sentence is aligned to two different labelled sequences in the source language. if the two sequences are of the same category, we merge them and we label the two sequences as a single one in the target sequence. if they are of different category we just consider the one with the longest length. finally, if a punctuation symbol in the target sequence is aligned to a labeled word in the source sentence we remove this alignment. ## datasets we conducted experiments in two sequence labelling tasks, namely, opinion target extraction (ote) and named entity recognition (ner). figure 3 illustrates both tasks. opinion target expression (ote): given a review, the task is to detect the linguistic expression used to refer to the reviewed entity. we use the semeval-2016 task 5 aspect based sentiment analysis (absa) datasets @xcite . we experiment with the english, spanish, dutch, french, russian and turkish datasets from the restaurant domain. ## experimental setup we perform 1-to-1 annotation projection in two directions: translate-train: we translate the english train and development data to the target language. we project the gold labels from the english data to the translated dataset. we then train a sequence labelling model using only the automatically generated dataset for the target language. translate-test: we translate the target language test set to english. we then use a model trained in the english gold-labelled data to label the translated test set. finally, we project the labelled sequences back to the target language. these two data-based cross-lingual transfer approaches are compared with the zero-shot method in which a fine-tuned model using english goldlabelled data is evaluated by generating predictions in the target language. finally, we also fine-tuned language models on the gold-labelled data, which would constitute the upper-bound in our experimental setting. ## error analysis the experiments described in section 6 showed that translate-train and translate-test perform worse than the zero-shot approach when using xlm-r large. in this section we will assess the performance of the machine translation and word alignment models. furthermore, we will undertake an error analysis to better understand the shortcomings of translatetrain and translate-test with respect to the zero-shot cross-lingual transfer. ## concluding remarks in this paper we described an in-depth and comprehensive evaluation of model-based and data-based zero-resource cross-lingual sequence labelling on two different tasks. contrary to what previous research suggests, zero-shot transfer approach is the best performing method when using high capacity multilingual language models such as xlm-r large. however, data-based cross-lingual transfer approaches are still useful when having a model with poor downstream cross-lingual performance. for example, when using a pretrained language model not trained for a specific domain, or when the required hardware for working with such larger language models is not readily available. a detailed error analysis demonstrates that databased cross-lingual transfer is hindered by machine translations which, although linguistically sound, do not align with the cultural behaviour of the target language use. moreover, the results also show that the different word alignments methods (for annotation projection) are of high quality, obtaining comparable results with respect to manually generated alignments. in any case, our results establish that there is still room for improving the cross-lingual performance of zero-resource sequence labelling. zero-shot model transfer approach work for indo-european languages, while their performance for other language families remains unknown. finally, the error analysis was performed for the en-es language pair only. in any case, we believe that our main claim still holds. even though mt quality has substantially improved over the last few years, our results indicate the current optimal solution to perform crosslingual transfer is by using large multilingual language models such as xlm-roberta-large. thus, our error analysis suggests that this might be due to important differences in language use. more specifically, mt often generates a textual signal which is different to what the models are exposed to when using gold standard data, which affects both the fine-tuning and evaluation processes. this is confirmed by our error analysis which shows that mistranslations are not the main source of errors in the data-transfer method. use the train set as both, train and development data. for ner we use a batch size of 32, 2e -5 learning rate, we train the model for 4 epochs and 256 maximum sequence length. we use the default values (sequence labelling implementation of the huggingface library foot_5 ) for the remaining hyperparameters. for both tasks we use the bilou encoding scheme.