Multi- C ross RE A Multi-Lingual Multi-Domain Dataset for Relation Extraction (2023)

Abstract: Most research in Relation Extraction (RE) involves the English language, mainly due to the lack of multi-lingual resources. We propose Multi-CrossRE, the broadest multi-lingual dataset for RE, including 26 languages in addition to English, and covering six text domains. Multi-CrossRE is a machine translated version of CrossRE (Bassignana and Plank, 2022), with a sub-portion including more than 200 sentences in seven diverse languages checked by native speakers. We run a baseline model over the 26 new datasets and–as sanity check–over the 26 back-translations to English. Results on the back-translated data are consistent with the ones on the original English CrossRE, indicating high quality of the translation and the resulting dataset.

## introduction binary relation extraction (re) is a sub-field of information extraction specifically aiming at the extraction of triplets from text describing the semantic connection between two entities. the task gained a lot of attention in recent years, and different directions started to be explored. for example, learning new relation types from just a few instances (few-shot re; @xcite @xcite , or evaluating the models over multiple source domains @xcite . however, a major issue of re is that most research so far involves the english language only. after the very first multi-lingual work from the previous decade-the ace dataset @xcite including english, arabic and chinese-recent work has started again exploring multi-lingual re. @xcite published a multi-lingual dataset, built from entity translations and wikipedia alignments from the original english version. the latter was collected from automatic alignment between dbpedia and wikipedia. the result includes 14 languages, but with very diverse relation type distributions: only english contains instances of all the 36 types, while the most low-resource ukrainian contains only 7 of them (including the 'no relation'). this setup makes it hard to directly compare the performance on different languages. @xcite and googlere, foot_0 both consisting of triplets in the form (object, relation, subject) with the aim of investigating the knowledge present in pre-trained language models by querying them via fixed templates. in the field of distantly supervised re, @xcite @xcite introduce new datasets including respectively four and three languages in addition to english. in this paper, we propose multi-crossre, to the best of our knowledge the most diverse re dataset to date, including 27 languages and six diverse text domains for each of them. we automatically translated crossre @xcite , a fully manually-annotated multi-domain re corpus, annotated at sentence level. we release the baseline results on the proposed dataset and, as quality check, on the 26 back-translations to english. additionally, we report an analysis where native speakers in seven diverse languages manually check more than 200 translated sentences and the respective entities, on which the semantic relations are based. multi-crossre allows for the investigation of sentence-level re in the 27 languages included in it, and for direct performance comparison between them. our contributions are: 1 we propose a practical approach to machine-figure 1 : example sentence with color-coded entity markup. from top to bottom: the original english text, its translation to german, and translation back to english. in the first translation step the entity classification is not transferred to german. in the second translation step the entity machine learning is (wrongly) expanded by a comma-later corrected in our post-processing. ## multi-crossre crossre as english base, we use crossre @xcite ), foot_2 a recently published multi-domain dataset. crossre is entirely manually-annotated, and includes 17 relation types spanning over six diverse text domains: artificial intelligence ( ), literature ( ), music ( ), news ( ), politics ( ), natural science ( ). the dataset was annotated on top of crossner @xcite , a named entity recognition (ner) dataset. translation process with the recent progress in the quality of machine translation (mt), utilizing machine-translated datasets in training and evaluation of nlp methods has become a stan-dard practice @xcite . as long as the annotation is not spanbound, producing a machine-translated dataset is rather straightforward. the task however becomes more involved for datasets with annotated spans, such as the named entities in our case of the crossre dataset, or e.g. the answer spans in a typical question answering (qa) dataset. numerous methods have been developed for transferring span information between the source and target texts @xcite . these methods are often tedious and in many cases rely on languagespecific resources to obtain the necessary mapping. some methods also require access to the inner state of the mt system, e.g. its attention activations, which is generally not available when commercial mt systems are used. in this work, we demonstrate a practical and simple approach to the task of machine translating a span-based dataset. we capitalize on the fact that deepl, foot_3 a commercial machine translation service very popular among users thanks to its excellent translation output quality, is capable of translating document markup. this feature is crucial for professional translators-the intended users of the service-who need to translate not only the text of the source documents, but also preserve their formatting. in practice, this means that the input of deepl can be a textual document with formatting (a word document) and the service produces its translated version with the formatting preserved. for the crossre dataset, we only need to transfer the named entities, which can be trivially encoded as colored text spans in the input documents, where the color differentiates the individual entities. this is further facilitated by the fact that the entities do not overlap in the dataset, allowing for a simple one-to-one id-color mapping. observing that oftentimes the entities are over- ## manual analysis we performed a manual analysis for further inspecting the quality and usability of multi-crossre for studying multi-lingual re. we manually checked 210 sentences from a diverse set of seven languages, including one north germanic (danish), one uralic (finnish), one west slavic (czech), two germanic (german and dutch), one latin (italian), and one japonic (japanese). for each of them, native speakers annotated the following: 1 in how many sentences is the overall meaning preserved? 2 how many entities are transferred to language x? 3 how many entities are correctly translated? 4 how many entities are marked with the correct entity boundaries? we annotated 30 sentences for each language. ## conclusion we introduce multi-crossre, the most diverse re dataset to date, including 26 languages in addition to the original english, and six text domains. the proposed span-based mt approach could be easily applied to similar cases. we report baseline results on the proposed resource and, as quality check, we back-translate multi-crossre to english and run the baseline model again over it. our manual analysis reveals that the higher challenge during the translation is transferring the correct entity boundaries. however, given the model architecture, this does not influence the scores.