Unsupervised Cross-Lingual Transfer of Structured Predictors without Source Data (2022)

Abstract: Providing technologies to communities or domains where training data is scarce or protected e.g., for privacy reasons, is becoming increasingly important. To that end, we generalise methods for unsupervised transfer from multiple input models for structured prediction. We show that the means of aggregating over the input models is critical, and that multiplying marginal probabilities of substructures to obtain high-probability structures for distant supervision is substantially better than taking the union of such structures over the input models, as done in prior work. Testing on 18 languages, we demonstrate that the method works in a cross-lingual setting, considering both dependency parsing and part-of-speech structured prediction problems. Our analyses show that the proposed method produces less noisy labels for the distant supervision.

## introduction recent successes of nlp systems have been enabled by supervised learning algorithms requiring a large amount of labelled data. creating such data can be costly for structured prediction tasks such as dependency parsing @xcite . transfer learning (pan and yang, 2010) is a promising solution to this problem. in this work, we focus on a case of transfer learning, namely cross-lingual learning. we consider the setup where the target language is low-resource having only unlabelled data, commonly referred to as unsupervised cross-lingual transfer. this is an important problem because most world's languages are low-resource @xcite . successful transfer from high-resource languages enables language technologies development for these low-resource languages. one recent method for unsupervised crosslingual transfer is pptx @xcite . developed for dependency parsing, it transfers from multiple source languages, which has been shown to be superior to transferring from just a single language @xcite @xcite . pptx computes the union of high-probability trees from all source parsers and uses the result as supervision to train the target language parser. one advantage is that, in addition to not requiring labelled data in the target language, it does not require any data in the source languages either, which is useful if such data is private. all it needs is access to multiple, trained source parsers. despite its benefits, pptx has only been applied to dependency parsing, although in principle it should be extensible to other structured prediction problems. more concerningly, we show in this work that pptx generally underperforms compared to a majority voting baseline. in this paper, we generalise and improve pptx for structured prediction problems. as with pptx, this generalisation casts the unsupervised transfer problem as a supervised learning task with distant supervision, where the label of each sample in the target language is based on the structures predicted by an ensemble of source models. moreover, we propose the use of logarithmic opinion pooling @xcite to improve performance (see fig. 1 ). unlike pptx that performs simple union, the pooling considers the output probabilities in aggregating the source model outputs to obtain the structures used for distant supervision. we test our method on 18 languages from 5 language families and on two structured prediction tasks in nlp: dependency parsing and pos tagging. we find that our method generally outperforms both pptx and the majority voting baseline, with absolute accuracy gains of up to 7 % on parsing and 20 % on tagging. our analysis shows that the use of logarithmic opinion pooling results in fewer predicted figure 1 : illustration of our method for an input sentence saya makan nasi ("i eat rice"). 1 a set of structured prediction models as inputs. @xcite the models compute marginal probability distributions over substructures for each token x j . 3 logarithmic opinion pool of the distributions is computed. 4 substructures are filtered based on some threshold. 5 high-probability substructures are obtained. 6 high-probability structures are obtained from the substructures as distant supervision. in summary, our contributions in this paper are: 1. developing a generic unsupervised multisource transfer method for structured prediction problems; 2. leveraging logarithmic opinion pooling to take into account source model probabilities in the aggregation to produce the labels for distant supervision; and 3. outperforming previous work in dependency parsing and part-of-speech tagging, especially in the context of a stronger, multi-source transfer baseline. ## experimental setup data and evaluation we evaluate on dependency parsing and part-of-speech (pos) tagging. we use universal dependencies v2.2 @xcite and test on 18 languages spanning 5 language families (see appendix a). we divide the languages into distant and nearby groups based on their distance to english @xcite . we use the universal pos tags (upos) as labels for tagging. we exclude punctuation from parsing evaluation following the standard practice and report average performance across five random seeds unless stated otherwise. we also include a pptx baseline applied to tagging. our evaluation metric is accuracy for both tasks, which is equivalent to las for parsing. model architecture for parsing, we use the same architecture as was used by @xcite , consisting of embedding layers, a transformer encoder layer, and a biaffine output layer @xcite . at test time, we run the mst algorithm @xcite to find the highest scoring tree. for tagging, we replace the output layer with a linear crf layer. at test time, the viterbi algorithm is used to obtain the tag sequence with the highest score. source selection we adopt a "pragmatic" approach where we include 5 high-resource languages as sources: english, arabic, spanish, french, and german @xcite ), foot_2 which have been categorised as "quintessential richresource languages" due to the availability of massive language datasets @xcite . when a source language is also the target language, we exclude the language from the sources. for example, if arabic is the target language, then we use only the other 4 languages as sources, thus the target language is always unseen. see appendix b for more details. baselines our main baseline for both tasks is a majority voting ensemble (mv). for parsing, we score each possible arc by the number of source parsers that have the arc in the predicted tree and then run the mst algorithm. for tagging, we use the most commonly predicted tag for each input token. this baseline is not only more appropriate for multi-source transfer but also stronger than the direct transfer baseline used by @xcite which uses only a single source language (english), with accuracy gains of up to 15 points on both tasks. we also include knowledge distillation (kd) which has been used for parsing as a baseline @xcite . for tagging, we include bea @xcite which explicitly fa 43.7 *ar 37.3 id 59.0 ko 14.7 tr 20.1 hi 21.1 hr 57.4 he 56.2 distant languages / mv acc (%) 2 0 2 4 6 acc (%) bg 69.3 it 81.7 pt 76.6 fr 76.5 es 71.3 no 69.1 da 67.3 sv 70.1 nl 65.8 de 55.3 nearby languages / mv acc (%) kd pptx ours (a) dependency parsing fa 69.4 *ar 57.8 id 77.9 ko 45.0 tr 62.8 hi 59.7 hr 66.7 he 55.5 distant languages / mv acc (%) 10 5 0 5 10 acc (%) bg 75.6 it 74.5 pt 72.5 fr 65.4 es 67.1 no 63.2 da 73.9 sv 69.8 nl 70.8 de 50.2 nearby languages / mv acc (%) bea pptx ours (b) pos tagging figure 3: performance difference of pptx and our method against majority voting (mv) on parsing and tagging. numbers on the x-axis are the mv performance corresponding to the zero value on the y-axis. bea is run only once as it always gives the same result. ## results and analysis fig. 3a shows the accuracy difference of kd, pptx, and our method against mv on parsing. we see that pptx does not consistently outperform mv, substantially underperforming on 6 languages. 5 on the other hand, our method outperforms not only pptx but also both kd and mv on most languages. fig. 3b shows the corresponding results on pos tagging which is particularly convincing. we see that pptx often underperforms, with up to 10 % drop in accuracy compared to mv. in contrast, our method consistently outperforms mv with up to 10 % accuracy improvement. these results suggest that pptx may not improve over a method distant nearby pptx -1.2 ± 5.0 0.5 ± 2.4 ours 2.1 ± 2.7 2.8 ± 2.0 from the figure, we also see that on portuguese and italian, our method slightly underperforms compared to mv on parsing, but outperforms mv considerably on tagging. we hypothesise that this disparity is caused by the variability of the source models quality. on tagging, the direct transfer performance of 3 out of 5 source taggers is relatively poor on portuguese and italian, making it more likely for mv to predict wrongly as the good taggers are outvoted. in contrast, on parsing, arabic is the only source parser that has very poor transfer. the other source parsers achieve comparably good direct transfer performance so mv already performs well. parsing results using predicted pos tags since low-resource languages often don't have gold pos tags, we also evaluate our method for parsing using predicted pos tags. we use stanza @xcite to predict the pos tags of all target languages, and replace the gold upos with the predicted tags as the input. ## related work a straightforward method of multi-source transfer is training a model on the concatenation of datasets from the source languages. this approach was used by @xcite for dependency parsing and yields a substantial gain compared with singlesource transfer. more recent work by @xcite proposed to learn multilingual representations from the concatenation of source language data and use them to train a neural dependency parser. another method is language adversarial training, used by @xcite for various nlp tasks including named-entity recognition, which is often cast as structured prediction. despite their success, multi-source unsupervised cross-lingual transfer methods typically require the source language data, which is not always feasible. there are recent methods suitable in this sourcefree setup. @xcite proposed a method based on truth inference to model label confusion in multi-source transfer of named-entity recognisers. @xcite used teacher-student learning for named-entity recognition. a closely related work is by @xcite who argued that a small amount of labelled data in the target language is cheap to obtain and proposed an attention-based method to weight the source models leveraging 50 labelled target sentences. our work is different as we do not require any labelled data and evaluate on 3 times more languages than they did. in addition, their model is based on mbert @xcite , which benefits from larger data from multiple languages during pretraining. in many application scenarios, bert-based models are too costly, especially when criteria other than accuracy matter @xcite . our work builds upon the work of @xcite who proposed a method based on selftraining for unsupervised cross-lingual dependency parsing. in this work, we generalise their method to structured prediction problems and propose a modification to improve the quality of the distant supervision. ## conclusions in this paper, we (1) generalise previous methods for cross-lingual unsupervised transfer without source data to structured prediction problems and