T -Projection: High Quality Annotation Projection for Sequence Labeling Tasks (2023)

Abstract: In the absence of readily available labeled data for a given sequence labeling task and language, annotation projection has been proposed as one of the possible strategies to automatically generate annotated data. Annotation projection has often been formulated as the task of transporting, on parallel corpora, the labels pertaining to a given span in the source language into its corresponding span in the target language. In this paper we present T-Projection, a novel approach for annotation projection that leverages large pretrained text2text language models and state-of-the-art machine translation technology. T-Projection decomposes the label projection task into two subtasks: (i) A candidate generation step, in which a set of projection candidates using a multilingual T5 model is generated and, (ii) a candidate selection step, in which the generated candidates are ranked based on translation probabilities. We conducted experiments on intrinsic and extrinsic tasks in 5 Indo-European and 8 low-resource African languages. We demostrate that T-projection outperforms previous annotation projection methods by a wide margin. We believe that T-Projection can help to automatically alleviate the lack of high-quality training data for sequence labeling tasks. Code and data are publicly available.

## introduction the performance of supervised machine-learning methods for natural language processing, including advanced deep-neural models @xcite @xcite , heavily depends on the availability of manually annotated training data. in addition, supervised models show a significant decrease in performance when evaluated in out-of-domain settings @xcite . this means that obtaining optimal results would require to manually generate annotated data for each application domain and language, an unfeasible task in terms of monetary cost and human effort. as a result, for the majority of languages in the world the amount of manually annotated data for many downstream tasks is simply nonexistent @xcite . the emergence of multilingual language models @xcite allows for zero-shot cross-lingual transfer. a model finetuned on one language, typically english, can be directly applied to other target languages. however, better results can be obtained by either machine translating the training data from english into the target languages or, conversely, translating the test data from the target language into english @xcite . sequence labeling tasks, which involve spanlevel annotations, require an additional step called annotation projection. this step involves identifying, in the translated sentence, the sequence of words that corresponds to the labeled spans in the source text @xcite . the majority of previous published work on this line of research explores the application of word-alignments @xcite . however, projection methods based on word-alignments have achieved mixed results as they often produce partial, incorrect or missing annotation projections @xcite . this is due to the fact that word alignments are computed on a word-byword basis leveraging word co-occurrences or similarity between word vector representations. that is, without taking into consideration the labeled spans or categories to be projected. other techniques have also been proposed, such as fine-tuning language models in the span projection task @xcite , translating the labeled spans independently from the sentence @xcite or including markers during the machine translation step @xcite . however, automatic annotation projection remains an open and difficult challenge. in this paper we present t-projection, a novel approach to automatically project sequence labeling annotations across languages. our method is illustrated by figure 1 . we split the annotation projection task into two steps. first, we use mt5 @xcite text-to-text model to generate a set of projection candidates in the target sentence for each labeled category in the source sentence. this step exploits the labeled categories as well as the cross-lingual capabilities of large pretrained multilingual language models. second, we rank the candidates based on the probability of being generated as a translation of the source spans. we use the m2m100 @xcite and nllb200 @xcite state-of-the-art mt models to compute the translation probabilities @xcite . we conduct an intrinsic evaluation on three different tasks, opinion target extraction (ote), named entity recognition (ner) and argument mining (am), and five different target languages (french, german, italian, russian and spanish). in this evaluation we compare the label projections generated by various systems with manually projected annotations. on average, t-projection improves the current state-of-the-art annotation projection methods by more than 8 points in f1 score, which constitutes a significant leap in quality over previous label projection approaches. additionally, we performed a real-world ner task evaluation involving eight low-resource african languages. in this downstream evaluation, t-projection outperformed other annotation projection methods by 3.6 points in f1 score. ## background while most of the previous approaches for annotation projection are based on the application of word alignments, other techniques have also been explored. @xcite use the statistical alignment of phrases to project the english labels of a multiparallel corpus into the target languages. instead of using discrete labels, @xcite project model expectations with the aim of facilitating the transfer of model uncertainty across languages. @xcite aim to filter goodquality projection-labeled data from noisy data by proposing a set of heuristics. other works have proposed to use word alignments generated by giza++ @xcite to project parallel labeled data from multiple languages into a single target language @xcite . @xcite use the word alignment probabilities calculated with fastalign @xcite and the pos tag distributions of the source and target words to project from the source corpus into a target language machine-translated corpus. finally, @xcite propose an annotation projection method based on machine translation and awesome @xcite , transformerbased word alignments to automatically project datasets from english to other languages. ## t-projection given a source sentence in which we have sequences of words labeled with a class, and its parallel sentence in a target language, we want to project the labels from the source into the target. as illustrated in figure 1 , t-projection implements two main steps. first, a set of projection candidates in the target sentence are generated for each labeled sequence in the source sentence. second, each projection candidate is ranked using a machine translation model. more specifically, candidates are scored based on the probability of being generated as a translation of the source labeled sequences. while the candidate generation step exploits the labeled spans and their categories in the source sentence as well as the zero-shot cross-lingual capabilities of large pretrained multilingual language models, the candidate selection step applies stateof-the-art mt technology to find those projection candidates that constitute the best translation for each source labeled span. as demonstrated by our empirical evaluation in sections 5 and 6, we believe that these two techniques leverage both the available information and knowledge from the annotated text and language models thereby allowing us to obtain better annotation projections. these two steps are described in detail in the following two subsections. ## experimental setup in order to evaluate our method we perform both intrinsic and extrinsic evaluations. ## intrinsic evaluation in this section we present a set of experiments to evaluate t-projection with respect to current stateof-the-art approaches for annotation projection. we also analyze separately the performance of the candidate generation and candidate selection steps. for the ote task we train t-projection and xlm-roberta with the english absa 2016 training set. we also train the four word alignment systems (excluding simalign which is an unsupervised method) using the english training set together with the respective translations as parallel corpora. we augment the parallel data with 50,000 random parallel sentences from paracrawl v8 @xcite . models are evaluated with respect to the manually label projections generated by @xcite . as the europarlbased ner dataset @xcite provides only test data for each language, t-projection and xlm-roberta are trained using the full english conll 2003 dataset @xcite together with the labeled english europarl test data. the word alignment models are in turn trained with the the parallel sentences from the europarl-based ner data plus 50,000 parallel sentences extracted from europarl v8 @xcite . we evaluate the model with respect to the manual annotations provided by @xcite . with respect to argument mining, we use the neoplasm training set from the abstrct dataset to train t-projection and xlm-roberta, adding its spanish translation as parallel corpora for the word alignment systems. as this is a medical text corpus, the parallel corpora is complemented with 50,000 parallel sen-tences from the wmt19 biomedical translation task @xcite . we evaluate the models with respect to the manually projected labels by @xcite . ## extrinsic evaluation in this section we evaluate t-projection in a realworld low-resource scenario, namely, named entity recognition in african languages. we compare the results obtained by training on ner dataset automatically generated by t-projection with respect to those automatically projected using two state-of-the-art label projection systems: awe-some (the second-best ner system in the results show that t-projection achieves superior performance for seven out of the eight languages. our model demonstrates a more pronounced performance difference in agglutinative languages such as igbo and shona. as outlined in section 5, our model produces superior alignments compared to awesome. on the other side, we found that easyproject, which utilizes markers for simultaneous translation and projection, introduces translation artifacts that hinder the performance of the downstream model. these artifacts are particularly noticeable in agglutinative languages, as easyproject tends to separate words. for instance, in the case of shona, consider the english sentence "[germany]'s representative to the [european union]'s veterinary committee [werner zwingmann]". our system produces the shona sentence "mumiriri [wegermany] kukomiti yemhuka [yeeuropean union] [werner zwingmann]", while easyproject produces "mumiriri we @xcite ku [european union] komiti yezvokurapa mhuka [werner zwingmann]". when training mdeber-tav3 with t-projection generated data, which preserves the agglutinated words, we achieve better results compared to easyproject that introduce artifacts by separating agglutinated words during translation and projection. our system is only inferior in the zulu language; however, on average, we improve the results by 3.6 f1 points. in contrast with previous work, our experiments revealed that concatenating english and translated data did not yield better results, potentially due to the superior quality of the data generated by t-projection. to the best of our knowledge, these are the best zero-shot results achieved for masakhaner2.0, underscoring the significant benefits of t-projection for nlp tasks in low-resource languages. ## concluding remarks in this paper we introduce t-projection, a new annotation projection method that leverages large multilingual pretrained text-to-text language models and state-of-the-art machine translation systems. we conducted experiments on intrinsic and extrinsic tasks in 5 indo-european languages and 8 african languages. t-projection clearly outperforms previous annotation projection approaches, obtaining a new state-of-the-art result for this task. a comprehensive analysis shows that both the generation candidate and the candidate selection steps crucially contribute to the final performance of t-projection. future work includes adding more tasks and languages, especially those with different segmentation such as japanese or chinese. unlike word alignment systems, t-projection does not need to segment the words to do the projection which is why we think that our model can also be competitive to project annotations for many language pairs.