Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction (2021)

Abstract: Zero-shot cross-lingual information extraction (IE) describes the construction of an IE model for some target language, given existing annotations exclusively in some other language, typically English. While the advance of pretrained multilingual encoders suggests an easy optimism of “train on English, run on any language”, we find through a thorough exploration and extension of techniques that a combination of approaches, both new and old, leads to better performance than any one cross-lingual strategy in particular. We explore techniques including data projection and self-training, and how different pretrained encoders impact them. We use English-to-Arabic IE as our initial example, demonstrating strong performance in this setting for event extraction, named entity recognition, part-of-speech tagging, and dependency parsing. We then apply data projection and self-training to three tasks across eight target languages. Because no single set of techniques performs the best across all tasks, we encourage practitioners to explore various configurations of the techniques described in this work when seeking to improve on zero-shot training.

## introduction we consider zero-shot cross-lingual information extraction (ie), in which training data exists in a source language but not in a target language. massively multilingual encoders like multilingual bert (mbert; @xcite and xlm-roberta (xlm-r; @xcite allow for a strategy of training only on the source language data, trusting entirely in a shared underlying feature representation across languages @xcite . however, in meta-benchmarks like xtreme @xcite , such cross-lingual performance on structured prediction tasks is far behind that on sentence- * equal contribution level or retrieval tasks @xcite : performance in the target language is often far below that of the source language. before multilingual encoders, cross-lingual ie was approached largely as a data projection problem: one either translated and aligned the source training data to the target language, or at test time one translated target language inputs to the source language for prediction @xcite . we show that by augmenting the source language training data with data in the target language-either via projection of the source data to the target language (so-called "silver" data) or via self-training with translated text-zero-shot performance can be improved. further improvements might come from using better pretrained encoders or improving on a projection strategy through better automatic translation models or better alignment models. in this paper, we explore all the options above, finding that everything is all it takes to achieve our best experimental results, suggesting that a silver bullet strategy does not currently exist. specifically, we evaluate: cross-lingual data projection techniques with different machine translation and word alignment components, the impact of bilingual and multilingual contextualized encoders on each data projection component, and the use of different encoders in task-specific models. we also offer suggestions for practitioners operating under different computation budgets on four tasks: event extraction, named entity recognition, part-ofspeech tagging, and dependency parsing, following recent work that uses english-to-arabic tasks as a test bed @xcite . we then apply data projection and self-training to three structured prediction tasks-named entity recognition, partof-speech tagging, and dependency parsing-in multiple target languages. additionally, we use self-training as a control against data projection to determine in which situations data projection improves performance. our contributions include the following: we also make available models and tools that enabled our analysis. ## universal encoders while massively multilingual encoders like mbert and xlm-r enable strong zero-shot crosslingual performance @xcite , they suffer from the curse of multilinguality @xcite : crosslingual effectiveness suffers as the number of supported languages increases for a fixed model size. we would therefore expect that when restricted to only the source and target languages, a bilingual model should perform better than (or at least on par with) a multilingual model of the same size, assuming both languages have sufficient corpora @xcite . if a practitioner is interested in only a small subset of the supported languages, is the multilingual model still the best option? to answer this question, we use english and arabic as a test bed. in ## data projection we create silver versions of the data by automatically projecting annotations from source english gold data to their corresponding machine translations in the target language. 3 data projection transfers word-level annotations in a source language to a target language via word-to-word alignments @xcite . the technique has been used to create cross-lingual datasets for a variety of structured natural language processing tasks, including named entity recognition @xcite and semantic role labeling @xcite @xcite . to create silver data, as shown in figure 1 , we: (1) translate the source text to the target language using the mt system described in section 5.2, (2) obtain word alignments between the original and translated parallel text using a word alignment tool, and (3) project the annotations along the word alignments. we then combine silver target data with gold source data to augment the training set for the structured prediction task. for step (1), we rely on a variety of source-totarget mt systems. to potentially leverage monolingual data, as well as contextualized cross-lingual information from pretrained encoders, we feed the outputs of the final layer of frozen pretrained encoders as the inputs to the mt encoders. we consider machine translation systems: (i) whose parameters are randomly initialized, (ii) that incorporate information from massively multilingual encoders, and (iii) that incorporate information from bilingual encoders that have been trained on only the source and target languages. after translating source sentences to the target language, in step (2) we obtain a mapping of the source words to the target words using publicly available automatic word alignment tools. similarly to our mt systems, we incorporate contextual encoders in the word aligner. we hypothesize that better word alignment yields better silver data, and better information extraction consequently. for step (3), we apply direct projection to transfer labels from source sentences to target sentences according to the word alignments. each target token receives the label of the source token aligned to it (token-based projection). for multi-token spans, the target span is a contiguous span containing all aligned tokens from the same source span (spanbased projection), potentially including tokens not aligned to the source span in the middle. three of the ie tasks we consider-ace, named entity recognition, and better-use span-based projection, and we filter out projected target spans that are five times longer than the source spans. two syntactic tasks-pos tagging and dependency parsinguse token-based projection. for dependency parsing, following @xcite , we adapt the disambiguation of many-to-one mappings by choosing as the head the node that is highest up in the dependency tree. in the case of a non-aligned dependency head, we choose the closest aligned ancestor as the head. to address issues like translation shift, filtered projection @xcite has been proposed to obtain higher precision but lower recall projected data. to maintain the same amount of silver data as gold data, in this study we do not use any task-specific filtered projection methods to remove any sentence. ## tasks we employ our silver dataset creation approach on a variety of tasks. 4 for english-arabic experiments, we consider ace, better, ner, pos tagging, and dependency parsing. for multilingual experiments, we consider ner, pos tagging, and dependency parsing. we use english as the source language and 8 typologically diverse target languages: arabic, german, spanish, french, hindi, russian, vietnamese, and chinese. because of the high variance of cross-lingual transfer @xcite , we report the average test performance of three runs with different predefined random seeds (except for ace). 5 for model selection and development, we use the english dev set in the zero-shot scenario and the combined english dev and silver dev sets in the silver data scenario. ## cross-lingual transfer results one might optimistically consider that the latest multilingual encoder (in this case xlm-r) in the zero-shot setting would achieve the best possible performance. however, in our extensive experiments in ## related work although projected data may be of lower quality than the original source data due to errors in translation or alignment, it is useful for tasks such as semantic role labeling @xcite , information extraction @xcite , pos tagging @xcite , and dependency parsing @xcite . the intuition is that although the projected data may be noisy, training on it gives a model useful information about the statistics of the target language. @xcite and @xcite use bootstrapping algorithms to iteratively construct projected datasets for semantic role labeling. @xcite additionally use manually defined filters to maintain high data quality, which results in a projected dataset that has low recall with respect to the source corpus. @xcite and @xcite find that a non-bootstrapped approach works well for cross-lingual srl. advances in translation and alignment quality allow us to avoid bootstrapping while still constructing projected data that is useful for downstream tasks. fei et al. (2020) and daza and frank (2020) also find improvements when training on a mixture of gold source language data and projected silver target language data. ideas from domain adaptation can be used to make more effective use of gold and silver data to mitigate the effects of language shift @xcite . improvements to task-specific models for zeroshot transfer are orthogonal to our work. for example, language-specific information can be incorporated using language indicators or embeddings @xcite , contextual parameter genera-tors @xcite , or language-specific semantic spaces @xcite . conversely, adversarial training @xcite has been used to discourage models from learning language-specific information @xcite @xcite . ## conclusion in this paper, we explore data projection and the use of silver data in zero-shot cross-lingual ie, facilitated by neural machine translation and word alignment. recent advances in pretrained encoders have improved machine translation systems and word aligners in terms of intrinsic evaluation. we conduct an extensive extrinsic evaluation and study how the encoders themselves-and components containing them-impact performance on a range of downstream tasks and languages. with a test bed of english-arabic ie tasks, we find that adding projected silver training data overall yields improvements over zero-shot learning. comparisons of how each factor in the data projection process impacts performance show that while one might hope for the existence of a silver bullet strategy, the best setup is usually task dependent. in multilingual experiments, we find that silver data tends to help languages with the weakest zeroshot performance, and that it is best used separately for each desired language pair instead of in joint multilingual training. we also examine self-training with translated text to assess when data projection helps crosslingual transfer, and find it to be another viable option for obtaining labels for some tasks. in future work, we will explore how to improve alignment quality and how to combine data projection and self-training techniques. notwithstanding any copyright annotation therein. this research made use of the following opensource software: allennlp @xcite , numpy @xcite , pytorch @xcite , pytorch lightning (falcon, 2019), scikit-learn @xcite , and transformers @xcite . franz josef och and hermann ney. 2003. a systematic comparison of various statistical alignment models. computational linguistics, 29(1):19-51. pedro javier ortiz suárez, laurent romary, and benoît sagot. 2020. a monolingual approach to contextualized word embeddings for mid-resource languages. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 1703-1714, online. association for computational linguistics. myle ott, sergey edunov, alexei baevski, angela fan, sam gross, nathan ng, david grangier, and michael auli. 2019. fairseq: a fast, extensible toolkit for sequence modeling. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics (demonstrations), pages 48-53, minneapolis, minnesota. association for computational linguistics. hiroaki ozaki, gaku morio, terufumi morishita, and toshinori miyoshi. 2021. project-then-transfer: effective two-stage cross-lingual transfer for semantic dependency parsing. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 2586-2594, online. association for computational linguistics. xiaoman pan, boliang zhang, jonathan may, joel nothman, kevin knight, and heng ji. 2017. crosslingual name tagging and linking for 282 languages. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1946-1958, vancouver, canada. association for computational linguistics. rrubaa panchendrarajan and aravindh amaresan. 2018. bidirectional lstm-crf for named entity recognition. in proceedings of the 32nd pacific asia conference on language, information and computation, hong kong. association for computational linguistics.