Abstains from Prediction: Towards Robust Relation Extraction in Real World (2022)

Abstract: “Supervised learning is a classic paradigm of relation extraction (RE). However, a well-performing model can still confidently make arbitrarily wrong predictions when exposed to samples of unseen relations. In this work, we propose a relation extraction method with rejection option to improve robustness to unseen relations. To enable the classifier to reject unseen relations, we introduce contrastive learning techniques and carefully design a set of class-preserving transformations to improve the discriminability between known and unseen relations. Based on the learned representation, inputs of unseen relations are assigned a low confidence score and rejected. Off-the-shelf open relation extraction (OpenRE) methods can be adopted to discover the potential relations in these rejected inputs. In addition, we find that the rejection can be further improved via readily available distantly supervised data. Experiments on two public datasets prove the effectiveness of our method capturing discriminative representations for unseen relation rejection.”

## introduction relation extraction aims to predict the relation between entities based on their context. the extracted relational facts play a vital role in various natural language processing applications, such as knowledge base enrichment @xcite , web search @xcite , and question answering @xcite . to improve the quality of extracted relational facts and benefit downstream tasks, many efforts have been devoted to this task. supervised relation extraction is a representative paradigm built upon the closed world assumption @xcite . benefiting from artfully designed network architectures @xcite @xcite and valuable knowledge in pretrained language model @xcite @xcite , models effectively capture semantic-rich representations and achieves superior results. however, conventional supervised relation extraction suffer from the lack of large-scale labeled data. to tackle this issue, distantly supervised relation extraction has attracted much attention. the existing works mainly focus on how to alleviate the noise generated in the automatic annotation. common approaches include selecting informative instances @xcite , incorporating extra information @xcite , and designing sophisticated training @xcite . although a supervised relation classifier achieves excellent performance on known relations, realworld inputs are often mixed with samples of unseen relations. a well-performing model can still confidently make arbitrarily wrong predictions when dealing with these unseen relations @xcite . the unrobustness is rooted in the shortcut feature @xcite of neural networks. models optimized by a supervised objective does not actively learn features beyond the bare minimum necessary to discriminate between known relations. as shown in figure 1 , if there is only president relation in the training data between obama and the united states, the model tends president of the united states is obama obama is elected the president of the united states obama + the united @xmath0 ## approach in this paper, we propose a robust relation extraction method in real world settings. by integrating rejection option, the classifier can effectively detect whether inputs express unseen relations instead of making arbitrary bad predictions. off-the-shell openre methods can be used to discover potential relations in these rejected samples. the problem setting in this work is formally stated as follows. let @xmath1, ..., r k } be a set of known relations and @xmath2, ..., r n } be a set of unseen relations where k∩@xmath3. let x be an input space. given the training data d ℓ = {(x ℓ i , y ℓ i )} @xmath4,...,n where x ℓ @xmath5, y ℓ @xmath6, we target constructing a mapping rule f : @xmath7, ..., r k , r * } where r * denotes rejection option. let d @xmath8, y u i )} @xmath9,...,m be the testing dataset where y u @xmath10. an desirable mapping rule f should meet the following objective as much as possible: ## experimental setup in this section, we describe the datasets for training and evaluating the proposed method. we also detail the baseline models for comparison. finally, we clarify the implementation details. ## results and analysis in this section, we present the experimental results of our method on fewrel and tacred datasets to demonstrate the effectiveness of our method. ## conclusions in this work, we introduce a relation extraction method with rejection option to improve the robustness in real-world applications. the proposed method employs contrastive training techniques and a set of carefully designed transformations to learn sufficient features. the classification of known relations and rejection of unseen relations can be done with these features. unseen relations in the rejected samples can be discovered by incorporating off-the-shelf openre methods. experimental results show that our method outperforms sota methods for unseen relation rejection.