Contextual Label Projection for Cross-Lingual Structured Prediction (2024)

Abstract: Label projection, which involves obtaining translated labels and texts jointly, is essential for leveraging machine translation to facilitate cross-lingual transfer in structured prediction tasks. Prior research exploring label projection often compromise translation accuracy by favoring simplified label translation or relying solely on word-level alignments. In this paper, we introduce a novel label projection approach, CLaP, which translates text to the target language and performs *contextual translation* on the labels using the translated text as the context, ensuring better accuracy for the translated labels. We leverage instruction-tuned language models with multilingual capabilities as our contextual translator, imposing the constraint of the presence of translated labels in the translated text via instructions. We benchmark CLaP with other label projection techniques on zero-shot cross-lingual transfer across 39 languages on two representative structured prediction tasks - event argument extraction (EAE) and named entity recognition (NER), showing over 2.4 F1 improvement for EAE and 1.4 F1 improvement for NER. We further explore the applicability of CLaP on ten extremely low-resource languages to showcase its potential for cross-lingual structured prediction.

## introduction cross-lingual transfer for structured prediction tasks such as named entity recognition, relation extraction, and event extraction, has gained considerable attention recently @xcite @xcite @xcite @xcite . it generalizes models trained in source languages to applications on other target languages @xcite @xcite . in south florida, the average number of suits against a neurosurgeon is five. source language: english (en) jointly translate sentence + obtain translated labels in translated sentence label projection , ( , trigger), ( , defendant) sentence : labels -roles : target language: chinese (zh) , ( , trigger), ( , defendant) sentence : labels -roles : , ( , trigger), ( , defendant) sentence : labels -roles : ## methodology in this section, we first formally define the previous attempts at label projection and later introduce clap, which provides a new perspective of using contextual machine translation for label projection. ## experiments and results this section outlines our experimental settings, which includes the datasets, baselines, and implementation details. subsequently, we provide an in-depth analysis of clap through both intrinsic and extrinsic evaluations. ## clap for low-resource languages to cater our model to a wide range of languages, we study the applicability of clap for low-resource languages. specifically, we consider the task of ner for 10 low-resource languages from africa and south america. for the test datasets, we utilize masakhaner @xcite for 9 african languages: hausa (ha), igbo (ig), chichewa (ny), kinyarwanda (rw), chshona (sn), kiswahili (sw), isixhosa (xh), yorùbá (yo), isizulu (zu), and refer to @xcite for the south american our model clap improves for 6 languages and performs the best for 3 languages. this result is particularly encouraging as our model uses a small and english-centric 13b llama-2 model and utilizing larger multilingual llms will amplify these improvements further (as shown in § 5.2). foot_7 label projection techniques several works have attempted to solve label projection for various structure extraction tasks such as semantic role labeling @xcite , slot filling @xcite , semantic parsing @xcite , ner @xcite , and question-answering @xcite @xcite . the earliest works @xcite utilized statistical word-alignment techniques like giza++ @xcite or fast-align @xcite for locating the labels in the translated sentence. recent works @xcite have also explored the usage of neural word aligners like qa-align @xcite and awesome-align @xcite . another set of works has explored the paradigm of mark-then-translate using special markers like quote characters ("") @xcite , xml tags (<@xmath0, and square braces ([0]) @xcite to locate the translated labels. overall, both these techniques can be error-prone and have poorer translation quality @xcite , as shown in § 4.4 and 5.1. a recent concurrent work codec @xcite improves the translation quality of text with markers by constrained decoding and data augmentation. ## conclusion and future work in our work, we propose a novel approach clap for label projection, which utilizes contextual machine translation using instruction-tuned language models. experiments on two structure prediction tasks of eae and ner across 39 languages demonstrate the effectiveness of clap compared to other label projection techniques. intrinsic evaluation provides deeper insights that justify our model improvements. additional experiments using larger llms, various translation models, translate-test paradigm, and 10 extremely low-resource languages demonstrate the generalizability and future potential of clap for cross-lingual structured prediction.