Open Set Relation Extraction via Unknown-Aware Training (2023)

Abstract: The existing supervised relation extraction methods have achieved impressive performance in a closed-set setting, in which the relations remain the same during both training and testing. In a more realistic open-set setting, unknown relations may appear in the test set. Due to the lack of supervision signals from unknown relations, a well-performing closed-set relation extractor can still confidently misclassify them into known relations. In this paper, we propose an unknown-aware training method, regularizing the model by dynamically synthesizing negative instances that can provide the missing supervision signals. Inspired by text adversarial attack, We adaptively apply small but critical perturbations to original training data,synthesizing difficult enough negative instances that are mistaken by the model as known relations, thus facilitating a compact decision boundary. Experimental results show that our method achieves SOTA unknown relation detection without compromising the classification of known relations.

## introduction relation extraction (re) is an important basic task in the field of natural language processing, aiming to extract the relation between entity pairs from unstructured text. the extracted relation facts have a great practical interest to various downstream applications, such as dialog system @xcite , knowledge graph @xcite , web search @xcite , among others. many efforts have been devoted to improving the quality of extracted relation facts @xcite . conventional supervised relation extraction is oriented to known relations with pre-specified schema. hence, the paradigm follows a closedset setting, meaning that during both training and testing the relations remain the same. nowadays, neural re methods have achieved remarkable success within this setting @xcite ; and in contrast, open relation extraction (openre) is focused on discovering constantly emerging unknown relations. common practices include directly tagging the relational phrases that link entity pairs @xcite , and clustering instances with the same relation @xcite . however, relation extraction in real applications follows an open-set setting, meaning that both known and unknown relations are mixed within testing data. * this requires that a model can not only distinguish among the known relations, but also filter the instances that express unknown relations. the ability to filter these instances is also called none-of-the-above (nota) detection @xcite . unfortunately, a well-performing closed-set model can still confidently make arbitrarily wrong predictions when exposed to unknown test data @xcite . as shown in fig. 1 (a ), the decision boundary is optimized only on the known relational data (white points), leading to a three-way partition of the whole space. consequently, the unknown relational data (black points), especially those far from the decision boundary, will be confidently classified into one of the known relations. by contrast, a more compact decision boundary (as shown in fig. 1 (b) ) is desirable for nota detection. however, the compact decision boundary requires "difficult" negative data (red points in fig. 1 (b) ) to be used, so strong supervision signals can be provided. it is important to note that synthesizing such negative data is a non-trivial task. in this work, we propose an unknown-aware training method, which simultaneously optimizes known relation classification and nota detection. to effectively regularize the classification, we iteratively generate negative instances and optimize a nota detection score. during the testing phase, instances with low scores are considered as nota and filtered out. the key of the method is to synthesize "difficult" negative instances. inspired by text adversarial attacks, we achieve the goal by substituting a small number of critical tokens in original training instances. this would erase the original relational semantics and the model is not aware of it. by using gradient-based token attribution and linguistic rules, key tokens that express the target relation are found. then, the tokens are substituted by misleading normal tokens that would cause the greatest increase of nota detection score, thus misleading negative instances, which are more likely to be mistaken by the model as known relations, are synthesized. human evaluation shows that almost all the synthesized negative instances do not express any known relations. experimental results show that the proposed method learns more compact decision boundary and achieve state-of-the-art nota detection performance. our codes are publicly available at github. the first group explores model regularization using unknown data. @xcite manually collect unknown data to train a (n + 1)-way classifier with one additional class, where (n+1) th class represents the unknown class. instead of manually collecting unknown data, @xcite generate feature vectors of unknown data using a generative adversarial network @xcite . @xcite use mixup technique @xcite to synthesize known data into unknown data. the second group approaches this problem by discriminative representation learning, which facilitates open-set classification by widening the margin between known and unknown classes. msp @xcite is a maximum posterior probability-based baseline and odin @xcite enlarges the difference between known and unknown classes by adding temperature scaling and perturbations to msp. more recently, different optimization objectives such as large margin loss @xcite and gaussian mixture loss @xcite ) are adopted to learn more discriminative representations. @xcite al. (2017); xu et al. (2020); zhang et al. (2021) also impose gaussian assumption to data distribution to facilitate distinct unknown data. open-set relation extraction: open-set re is a pressing but underexplored task. most of the existing re methods manually collect nota data and adopt a (n + 1) way classifier to deal with nota relations @xcite @xcite . however, the collected nota data with manual bias cannot cover all nota relations and thus these methods cannot effectively deal with open-set re @xcite . ## conclusions in this work, we propose an unknown-aware training method for open-set relation extraction, which is a pressing but underexplored task. we dynamically synthesize negative instances by the attribution technique and relational linguistic rules to complete the missing supervision signals. the negative instances are more difficult than that of other competitive methods and achieve effective model regularization. experimental results show that our method achieves state-of-the-art nota detection without compromising the classification of known relations. we hope our method and analysis can inspire future research on this task.