RED ^{\textrm{FM}} : a Filtered and Multilingual Relation Extraction Dataset (2023)

Abstract: Relation Extraction (RE) is a task that identifies relationships between entities in a text, enabling the acquisition of relational facts and bridging the gap between natural language and structured knowledge. However, current RE models often rely on small datasets with low coverage of relation types, particularly when working with languages other than English.In this paper, we address the above issue and provide two new resources that enable the training and evaluation of multilingual RE systems. First, we present SRED ^{\textrm{FM}} , an automatically annotated dataset covering 18 languages, 400 relation types, 13 entity types, totaling more than 40 million triplet instances. Second, we propose RED ^{\textrm{FM}} , a smaller, human-revised dataset for seven languages that allows for the evaluation of multilingual RE systems. To demonstrate the utility of these novel datasets, we experiment with the first end-to-end multilingual RE model, mREBEL, that extracts triplets, including entity types, in multiple languages. We release our resources and model checkpoints at [ https://www.github.com/babelscape/rebel ]( https://www.github.com/babelscape/rebel ).

## introduction the vast majority of online and offline content consists of raw, natural language text containing factual information. current large language models (llms) are pretrained on such text, allowing reasoning over it through tasks such as question answering @xcite or text summarization @xcite . on the other hand, structured resources such as knowledge graphs enable knowledge-based, explainable, machine-ready reasoning over their content. both approaches are important and are widely used within natural language processing systems, with recent trends looking at combining them @xcite . information extraction tackles the need for systems that extract structured information from raw text. specifically, end-to-end relation extraction extracts the relational information between entities in a given text, providing a structured prediction. however, although some highly capable systems have been released @xcite @xcite , there are few high-quality, contemporary resources. current re datasets are outdated, behind paywalls, have design flaws, or only consider english. while multilingual datasets exist, such as ace05 foot_0 or smiler @xcite , the former covers only six relation types, and requires a paid license for its use. the latter is more recent, bigger, and has a higher coverage of relation types, but it does not contain human-annotated samples that permit reliable evaluation and is not conducive to train end-to-end relation extraction systems. instead, the availability of large high-quality resources is fundamental in order to allow llms to be trained and evaluated on trustworthy multilingual re benchmarks. in this paper, we introduce large amounts of high-coverage re annotated data in a multilingual fashion. our new resources will enable the training of multilingual re systems and their evaluation. in particular, we provide three main contributions: ## red fm in this section, we present red fm , our supervised and multilingual dataset for relation extraction, and a larger sred fm , a silver-annotated dataset covering more languages and relation types. the creation of the dataset consists of several steps: data collection and processing (section 3.1), manual annotation (section 3.2), a triplet filtering system (section 3.3) and entity typing (section 3.4). figure 1 shows an overview of this process. ## mrebel for (1) and ( 2 ) we also train their untyped versions, mrebel 400 and mrebel 32 . ## experimental setup we evaluate mrebel and its variants on our own datasets, i.e. red fm and sred fm , and on smiler @xcite . unless stated otherwise, we train on the training sets of all languages simultaneously and apply early stopping based on the micro-f1 obtained on the overall validation set. we use the adafactor optimizer and the same cross-entropy loss with teacher forcing from huguet cabot and @xcite . the full list of hyperparameters is detailed in appendix d. ## results multilingual relation extraction first, in training on 400 relation types does lead to lower results, since there is a mismatch between the two stages of training. however, mrebel t 400 showed decent performance on sred fm as shown in multilingual relation classification from the bottom half of ## conclusions in this paper, we have addressed some of the key issues facing current multilingual relation extraction datasets by presenting two new resources: sred fm and red fm . sred fm is an automatically annotated dataset covering 18 languages, 400 relation types, 13 entity types, and more than 40 million triplet instances, while red fm is a smaller, humanly-revised dataset for seven languages. we improved the quality of the entity type annotations in these datasets by using a transformer-based ner classifier. we also introduced the triplet critic, a cross-encoder that is trained on annotated data to predict whether a given context entails a triplet. we demonstrated the utility of these new resources by training new, capable multilingual relation extraction models and evaluating them using our supervised data. we also presented mrebel, the first multilingual end-to-end relation extraction system that extracts triplets, including entity types.