An Improved Baseline for Sentence-level Relation Extraction (2022)

Abstract: Sentence-level relation extraction (RE) aims at identifying the relationship between two entities in a sentence. Many efforts have been devoted to this problem, while the best performing methods are still far from perfect. In this paper, we revisit two problems that affect the performance of existing RE models, namely entity representation and noisy or ill-defined labels. Our improved RE baseline, incorporated with entity representations with typed markers, achieves an F1 of 74.6% on TACRED, significantly outperforms previous SOTA methods. Furthermore, the presented new baseline achieves an F1 of 91.1% on the refined Re-TACRED dataset, demonstrating that the pretrained language models (PLMs) achieve high performance on this task. We release our code to the community for future research.

## introduction as one of the fundamental information extraction (ie) tasks, relation extraction (re) aims at identifying the relationship(s) between two entities in a given piece of text from a pre-defined set of relationships of interest. for example, given the sentence "bill gates founded microsoft together with his friend paul allen in 1975" and an entity pair ("bill gates", "microsoft"), the re model is expected to predict the relation org:founded by. on this task, sota models based on plms @xcite have gained significant success. recent work on sentence-level re can be divided into two lines. one focuses on injecting external knowledge into plms. methods of such, including ernie @xcite and know-bert @xcite , take entity embedding pretrained from knowledge graphs as inputs to the transformer. similarly, k-adapter @xcite introduces a plug-in neural adaptor that injects factual and linguistic knowledge into the language model. luke @xcite further extends the pretraining objective of masked language modeling to entities and proposes an entity-aware self-attention mechanism. the other line of work focuses on continually pretraining plms on text with linked entities using relationoriented objectives. specifically, bert-mtb @xcite proposes a matching-theblanks objective that decides whether two relation instances share the same entities. despite extensively studied, existing re models still perform far from perfect. on the commonly-used benchmark tacred @xcite , the sota f 1 result only increases from 70.1% (bert large ) to 72.7% (luke) after applying plms to this task. it is unclear what building block is missing to constitute a promising re system. in this work, we discuss two obstacles that have hindered the performance of existing re models. first, the re task provides a structured input of both the raw texts and side information of the entities, such as entity names, spans, and types (typically provided by ner models), which are shown important to the performance of re models @xcite . however, existing methods fall short of representing the entity information comprehensively in the text, leading to limited characterization of the entities. second, human-labeled re datasets (e.g., tacred), may contain a large portion of noisy or ill-defined labels, causing the model performance to be misestimated. @xcite relabeled the development and test set of ta-cred and found that 6.62% of labels are incorrect. @xcite refined many ill-defined relation types and further re-annotated the tacred dataset using an improved annotation strategy to ensure high-quality labels. to this end, we propose an improved re baseline, where we introduce the typed entity marker to sentence-level re, which leads to promising improvement of performance over existing re models. we evaluate our model on tacred @xcite , tacrev @xcite , and re-tacred @xcite . using roberta @xcite as the backbone, our improved baseline model achieves an f 1 of 74.6% and 83.2% on tacred and tacrev, respectively, significantly outperforming various sota re models. particularly, our baseline model achieves an f 1 of 91.1% on re-tacred, demonstrating that plms can achieve much better results on re than shown in previous work. 2 ## method in this section, we first formally define the relation extraction task in sec. 2.1, and then present our model architecture and entity representation techniques in sec. 2.2 and sec. 2.3. ## experiments in this section, we evaluate the proposed techniques based on widely used re benchmarks. the evaluation starts by first identifying the best-performing entity representation technique (sec. 3.2), which is further incorporated into our improved re baseline to be compared against prior sota methods (sec. 3.3). due to space limits, we study in the appendix of how the incorporated techniques lead to varied generalizability on unseen entities (appx. b) and how they perform under annotation errors (appx. c). ## conclusion in this paper, we present a simple yet strong re baseline that offers new sota performance, along with a comprehensive study to understand its prediction generalizability and robustness. specifically, we revisit two technical problems in sentencelevel re, namely entity representation and noisy or ill-defined labels. we propose an improved entity the statistics of the datasets are shown in tab. 3.