REBEL : Relation Extraction By End-to-end Language generation (2021)

Abstract: Extracting relation triplets from raw text is a crucial task in Information Extraction, enabling multiple applications such as populating or validating knowledge bases, factchecking, and other downstream tasks. However, it usually involves multiple-step pipelines that propagate errors or are limited to a small number of relation types. To overcome these issues, we propose the use of autoregressive seq2seq models. Such models have previously been shown to perform well not only in language generation, but also in NLU tasks such as Entity Linking, thanks to their framing as seq2seq tasks. In this paper, we show how Relation Extraction can be simplified by expressing triplets as a sequence of text and we present REBEL, a seq2seq model based on BART that performs end-to-end relation extraction for more than 200 different relation types. We show our model’s flexibility by fine-tuning it on an array of Relation Extraction and Relation Classification benchmarks, with it attaining state-of-the-art performance in most of them.

## introduction extracting relational facts from text has been an ongoing part of natural language processing. the ability to extract semantic relationships between entities from text can be used to go from unstructured raw text to structured data that can be leveraged in an array of downstream tasks and applications, such as the construction of knowledge bases. traditionally this task has been approached as a two-step problem. first, the entities are extracted from text as in named entity recognition (ner). second, relation classification (rc) checks whether there exists any pairwise relation between the extracted entities @xcite . however, identifying which entities truly share a relation can become a bottleneck, requiring additional steps such as negative sampling and expensive annotation procedures. more recently, end-to-end approaches have been used to tackle both tasks simultaneously @xcite @xcite . this task is usually referred to as relation extraction or endto-end relation extraction (re). in this scenario, a model is trained simultaneously on both objectives. specific parts of the model can be assigned different tasks of the pipeline, such as ner, on the one hand, and classifying the relations between the predicted entities (rc), on the other. by training both tasks simultaneously, the model benefits from the information bias between the tasks as in multi-task setups @xcite , improving performance on the end-to-end re task. although successful, these models are often complex, with task-focused elements that need to be adapted to the number of relation or entity types, or they are not flexible enough to work for texts of different nature (sentence vs. document level) or domains. moreover, they usually require long training times in order to be fine-tuned on new data. in this paper, we present rebel (relation extraction by end-to-end language generation), an autoregressive approach that frames relation extraction as a seq2seq task, together with the rebel dataset, a large-scale distantly supervised dataset, obtained by leveraging a natural language inference model. our approach provides some upsides over previous end-to-end approaches thanks to our adoption of a simple triplet decomposition into a text sequence. by pre-training an encoder-decoder transformer (bart) using our new dataset, rebel achieves state-of-the-art performance on an array of re baselines within a few epochs of fine-tuning. its simplicity makes it highly flexible to adapt to new domains or longer documents. as the same model weights are still utilized after the pre-training phase, there is no need to train model-specific components from scratch, making training more efficient. moreover, although it is devised for relation extraction, the same approach can be generalized to relation classification, achieving competitive results. we make rebel available foot_0 both as a standalone model that can extract more than 200 different relation types, and as a pre-trained re model that can be easily fine-tuned on new re and rc datasets. we also provide the rebel dataset and the pipeline to extract high-quality re datasets from any wikipedia dump. ## rebel we tackle relation extraction and classification as a generation task: we use an autoregressive model that outputs each triplet present in the input text. to this end, we employ bart-large @xcite as the base model. in a translation task, teacher forcing leverages pairs of text in two languages by conditioning the decoded text on the input. at training time the encoder receives the text in one language, and the decoder receives the text in the other language, outputting the prediction for the next token at each position. in our approach, we translate a raw input sentence containing entities, together with implicit relations between them, into a set of triplets that explicitly refer to those relations. therefore, we need to express the triplets as a sequence of tokens to be decoded by the model. we design a reversible linearization using special tokens that enable the model to output the relations in the text in the form of triplets while minimizing the number of tokens that need to be decoded. for rebel, we have as input the text from the dataset and, as output, the linearized triplets. if x is our input sentence and y the result of linearizing the relations in x as explained in section 3.1, the task for rebel is to autoregressively generate y given x: by fine-tuning bart on such a task, using the cross-entropy loss as in summarization or machine translation, we maximize the log-likelihood of generating the linearized triplets given the input text. ## experimental setup in this section, we describe the setup to train and evaluate rebel for four different widely used re datasets and one rc dataset. statistics for all the datasets, including our pre-training dataset, can be found in while the training objective is on the autoregressive task, we evaluate the model on re, extracting all the triplets from the generated output, and evaluating using recall, precision, and micro-f1 based on the labeled triplets. for a triplet to be considered correct, the entities and the relation, as well as their types, have to be the same as the labeled ones (this is known as "strict" evaluation in re) using the evaluation code from @xcite . ## conclusion we have presented rebel, alongside a new distantly supervised dataset for pre-training. rebel frames re into a seq2seq task and, by leveraging bart, achieves state-of-the-art performances in an array of re benchmarks. we have also shown its flexibility in adapting to new domains, by training on just a few epochs to attain results that are comparable to the previous state of the art, as well as the possibility of using it to perform relation classification. we make rebel pre-training available as a standalone re for more than 200 relation types together with a pre-trained re model to serve as a baseline when fine-tuning on new re datasets. nonetheless, rebel is based on bart-large, which has a big parameter footprint. therefore, we also plan to release a pre-trained rebel-base using bart-base. this will enable quick and efficient re. moreover, our dataset creation pipeline enables a quick and effortless way of obtaining large high-quality re datasets in multiple languages from a wikipedia dump. since both wikipedia and wikidata are in constant change, our method provides a way to keep up with those changes and to have up-to-date re datasets. we leave to future work the possibility of using a multi-dataset approach as in @xcite , including both re and rc datasets, and seeing if it retains or improves performance. furthermore, using our silver dataset as pre-training could lead to improved performance for other systems, especially those which have shown better performance than rebel without pre-training, such as wang and lu (2020) for conll04.