The Effect of Alignment Correction on Cross-Lingual Annotation Projection (2023)

Abstract: Cross-lingual annotation projection is a practical method for improving performance on low resource structured prediction tasks. An important step in annotation projection is obtaining alignments between the source and target texts, which enables the mapping of annotations across the texts. By manually correcting automatically generated alignments, we examine the impact of alignment quality—automatic, manual, and mixed—on downstream performance for two information extraction tasks and quantify the trade-off between annotation effort and model performance.

## introduction cross-lingual annotation projection @xcite involves mapping source annotations to target text through alignments. recent studies such as @xcite and @xcite suggest that word alignment quality substantially impacts downstream performance. automatic word alignments are inexpensive to obtain but may be of low quality. on the other side of the alignment quality spectrum (figure 1 ) are manual (human-labeled) alignments, which are expensive but accurate. our goals are to quantify the impact of automatic vs. manual alignments on downstream task performance and to explore the quality spectrum to quantify the trade-off between automatic and manual alignments in terms of downstream performance and cost. we investigate the alignment quality spectrum on two structured prediction tasks: shallow semantic parsing (better basic ie 1 ) and named entity recognition (ner). in the better ie scenario, we start with a typical fully automatic translate, align, and project pipeline, so-called "silver" data creation, and compare with manually labeled data (annotated by two of the authors). for ner, the dataset we use already has annotations in two languages (with gold bitext and gold word alignments). rather than manually annotate alignments for every example, which is expensive as shown on the right end of the spectrum in figure 1 , we collect manual annotations on the data for which two alignment methods-silver data creation and an unsupervised embedding-based span alignment tool-point to different target spans ( §5). our contributions are: 1) evidence that manually correcting alignments improves downstream performance, 2) evidence that downstream performance correlates with the amount of manual alignment effort, and 3) analysis on the types of spans that are manually corrected. ## related work cross-lingual projection is a method of transferring annotations from a source language to a target language that has often been used to increase performance on the target language @xcite @xcite , but its utility depends on obtaining reliable alignments between the source and target text. there has been extensive research on supervised and unsupervised alignment at the word, phrase, and sentence levels @xcite @xcite @xcite . however, these studies report primarily intrinsic evaluation of alignment quality and leave extrinsic better (%) ner (%) unique source spans 4896 3180 identical automatic spans 2432 (50%) 1942 (61%) candidates for correction 2464 (50%) 1238 (39%) our work responds to these lines of prior work by examining the extrinsic downstream impact of our approaches and how the incorporation of different amounts of gold alignment data affects performance. in contrast to prior work on projection, such as @xcite , our work focuses on the impact of the alignment component of the projection pipeline on the overall effectiveness of cross-lingual projection and explicitly adjusts automatic alignments with manual corrections. ## tasks we investigate the impact of alignment quality using established models, as modeling improvements are outside the scope of this work. ## experiments and results we compare the performance of models trained on various combinations of gold, silver, and semiautomatic training data. we evaluate the models on the target language test sets: the chinese gold test set for ner and farsi semi-automatic analysis set for better. we also use english training data in two different ways: combined with the target language data ('en +' in ## analysis 6.1 annotation budget constraints annotation budgets constrain how much of the data can be projected along gold alignments. furthermore, it is cheaper to annotate alignments than to train annotators on a complex task. note that our annotation task took only ∼ 16 hours for all data. we analyze downstream performance as a function of the amount of data correction. for both ner and better, we subsample 10%, 25%, and 50% of the tasks (sentences) in which the two automatic alignment methods have a disagreement, and then replace their silver alignments with the correct alignments: gold (for ner) and manually corrected (for better). the remaining portion of the data comes from the silver data aligned with awesome-align (i.e., semi-10% consists of 10% corrected data and 90% automatic silver data). we do the sampling process 5 times and report the average performance. the results in figure 2a show that the performance consistently improves as subsampling percentage increases. in all of the cases where projected zh is combined with en, either added or fine-tuned, any percentage of manual correction outperforms training on silver zh only (see appendix e for the plot for the fine-tuning setting). figure 2b shows a similar trend for better. the plots suggest, however, that to outperform training on silver data, more than 50% of the alignments need to be corrected. we hypothesize this is in part due to the evaluation set being non-gold, so correcting the training data shifts away from the distribution of the evaluation data. ## conclusion and future work in this paper, we investigated how much the quality of alignments impacts downstream task performance when annotations are projected from english to another language. our experimental results show that the utilization of different alignment methodologies, followed by corrections of the disagreements arising from such approaches, can reduce human effort while improving results. there are several promising avenues for future research in cross-lingual annotation projection. although this study did not investigate the influence of translation quality on downstream performance (which is the first step of the data projection pipeline), we believe this could yield important findings. another direction involves the investigation of active learning techniques for prioritizing which alignments to correct.