M ulti TACRED : A Multilingual Version of the TAC Relation Extraction Dataset (2023)

Abstract: Relation extraction (RE) is a fundamental task in information extraction, whose extension to multilingual settings has been hindered by the lack of supervised resources comparable in size to large English datasets such as TACRED (Zhang et al., 2017). To address this gap, we introduce the MultiTACRED dataset, covering 12 typologically diverse languages from 9 language families, which is created by machine-translating TACRED instances and automatically projecting their entity annotations. We analyze translation and annotation projection quality, identify error categories, and experimentally evaluate fine-tuned pretrained mono- and multilingual language models in common transfer learning scenarios. Our analyses show that machine translation is a viable strategy to transfer RE instances, with native speakers judging more than 83% of the translated instances to be linguistically and semantically acceptable. We find monolingual RE model performance to be comparable to the English original for many of the target languages, and that multilingual models trained on a combination of English and target language data can outperform their monolingual counterparts. However, we also observe a variety of translation and annotation projection errors, both due to the MT systems and linguistic features of the target languages, such as pronoun-dropping, compounding and inflection, that degrade dataset quality and RE model performance.

## introduction relation extraction (re), defined as the task of identifying and classifying semantic relationships between entities from text (cf. figure 1 ), is a fundamental task in information extraction @xcite . extending re to multilingual settings has recently received increased interest @xcite @xcite , both to address the urgent need for more inclusive nlp systems that cover more languages than just english @xcite , as well as to investigate language-specific phenomena and challenges relevant to this task. the main bottleneck for multilingual re is the lack of supervised resources, comparable in size to large english datasets @xcite , as annotation for new languages is very costly. most of the few existing multilingual re datasets are distantly supervised @xcite @xcite , and hence suffer from noisy labels that may reduce the prediction quality of models @xcite . available fully-supervised datasets are small, and cover either very few domain-specific relation types @xcite , or only a small set of languages @xcite . to address this gap, and to incentivize research on supervised multilingual re, we introduce a multilingual version of one of the most prominent supervised re datasets, tacred @xcite . multitacred is created by machinetranslating tacred instances and automatically projecting their entity annotations. machine translation is a popular approach for generating data in cross-lingual learning @xcite . although the quality of machine-translated data may be lower due to translation and alignment errors @xcite , it has been shown to be beneficial for classification and structured prediction tasks @xcite @xcite . the multitacred dataset we present in this work covers 12 languages from 9 language families. 1 we select typologically diverse languages which span a large set of linguistic phenomena such as compounding, inflection and pronoun-drop, and for which a monolingual pretrained language model is available. we automatically and manually analyze translation and annotation projection quality in all target languages, both in general terms and with respect to the re task, and identify typical error categories for alignment and translation that may affect model performance. we find that overall translation quality is judged to be quite good with respect to the re task, but that e.g. pronoun-dropping, coordination and compounding may cause alignment and semantic errors that result in erroneous instances. in addition, we experimentally evaluate fine-tuned pretrained monoand multilingual language models (plm) in common training scenarios, using source language (english), target language, or a mixture of both as training data. we also evaluate an english data fine-tuned model on back-translated test instances to estimate the effect of noise introduced by the mt system on model performance. our results show that in-language training works well, given a suitable plm. cross-lingual zero-shot transfer is acceptable for languages well-represented in the multilingual plm, and combining english and target language data for training considerably improves performance across the board. to summarize, our work aims to answer the following research questions: can we reaffirm the usefulness of mt and cross-lingual annotation projection, in our study for creating large-scale, high quality multilingual datasets for re? how do pretrained mono-and multilingual encoders compare to each other, in within-language as well as crosslingual evaluation scenarios? answers to these questions can provide insights for understanding language-specific challenges in re, and further research in cross-lingual representation and transfer learning. the contributions of this paper are: ## translating tacred we first briefly introduce the original tacred dataset, and then describe the language selection and automatic translation process. we wrap up with a description of the analyses we conduct to verify the translation quality. ## experiments in this section, we describe the experiments we conduct to answer the research questions "how does the performance of language-specific models compare to the english original?" and "how does the performance of language-specific models compare to multilingual models such as mbert trained on the english source data? how does the performance change when including target-language data for training". we first introduce the training scenarios, and then give details on choice of models and hyperparameters, as well as the training process. ## results and discussion we first present some insights into translation quality, and then discuss the performance of models for the different training scenarios. ## related work multilingual re datasets prior work has primarily focused on the creation of distantly supervised datasets. dis-rex @xcite and relx-distant @xcite are large, wikipedia-based datasets, but cover only 4 resp. 5 european languages. smiler @xcite covers 14 european languages, but is very imbalanced, both in terms of relation coverage in the different languages and training data per language @xcite . manually supervised datasets include bizrel @xcite hindi, telugu and english @xcite . the indore dataset uses mt to transfer manually labeled examples from english to the three other languages, but implements a heuristic to project entity annotations, without any verification step. other datasets are very small: the relx dataset contains a manually translated parallel test set of 502 sentences @xcite . @xcite create a small parallel re dataset of 533 sentences by sampling from tacred and translating into russian and korean. for the related task of event extraction, datasets worth mentioning are the multilingual ace 2005 dataset @xcite , the tac multilingual event extraction dataset @xcite , and the work of @xcite . machine translation for cross-lingual learning mt is a popular approach to address the lack of data in cross-lingual learning @xcite . there are two basic optionstranslating target language data to a well-resourced source language at inference time and applying a model trained in the source language @xcite @xcite , or translating source language training data to the target language, while also projecting any annotations required for training, and then training a model in the target language @xcite @xcite . both approaches depend on the quality of the mt system, with translated data potentially suffering from translation or alignment errors @xcite @xcite . with very few exceptions, using mt for multilingual re remains underexplored @xcite @xcite . re previous work in cross-and multilingual re has explored a variety of approaches. kim et al. (2014) proposed cross-lingual annotation projection, while faruqui and kumar (2015) machine-translate non-english sentences to english, and then project the relation phrase back to the source language for the task of open re. verga et al. (2016) use multilingual word embed-dings to extract relations from spanish text without using spanish training data. in a related approach, ni and florian (2019) describe an approach for cross-lingual re that is based on bilingual word embedding mapping. lin et al. (2017) employ convolutional networks to extract relation embeddings from texts, and propose cross-lingual attention between relation embeddings to model cross-lingual information consistency. chen et al. (2022c) introduce a prompt-based model, which requires only the translation of prompt verbalizers. their approach thus is especially useful in few-and zeroshot scenarios. ## conclusion we introduced a multilingual version of the largescale tacred relation extraction dataset, obtained via machine translation and automatic annotation projection. baseline experiments with inlanguage as well as cross-lingual transfer learning models showed that mt is a viable strategy to transfer sentence-level re instances and span-level entity annotations to typologically diverse target languages, with target language re performance comparable to the english original for many languages. however, we observe that a variety of errors may affect the translations and annotation alignments, both due to the mt system and the linguistic features of the target languages (e.g., compounding, high level of synthesis). multitacred can thus serve as a starting point for deeper analyses of annotation projection and re challenges in these languages. for example, we would like to improve our understanding of re annotation projection for highly inflectional/synthetic languages, where token-level annotations are an inadequate solution. in addition, constructing original-language test sets to measure the effects of translationese remains an open challenge. we plan to publish the translated dataset for the research community, depending on ldc requirements for the original tacred and the underlying tac corpus. we will also make publicly available the code for the automatic translation, annotation projection, and our experiments.