Element Intervention for Open Relation Extraction (2021)

Abstract: Open relation extraction aims to cluster relation instances referring to the same underlying relation, which is a critical step for general relation extraction. Current OpenRE models are commonly trained on the datasets generated from distant supervision, which often results in instability and makes the model easily collapsed. In this paper, we revisit the procedure of OpenRE from a causal view. By formulating OpenRE using a structural causal model, we identify that the above-mentioned problems stem from the spurious correlations from entities and context to the relation type. To address this issue, we conduct Element Intervention , which intervene on the context and entities respectively to obtain the underlying causal effects of them. We also provide two specific implementations of the interventions based on entity ranking and context contrasting. Experimental results on unsupervised relation extraction datasets show our method to outperform previous state-of-the-art methods and is robust across different datasets.

## introduction relation extraction (re) is the task to extract relation between entity pair in plain text. for example, when given the entity pair (obama, the united states) in the sentence "obama was sworn in as the 44th president of the united states", an re model should accurately predict the relationship "president of" and extract the corresponding triplet (obama, president of, the united states) for downstream tasks. despite the success of many re models @xcite , most previous re paradigms rely on the predefined relation types, which are always unavailable in open domain scenario and thereby limits their capability in real applications. open relation extraction (openre), on the other hand, has been proposed to extract relation facts without pre-defined relation types neither annotated data. given a relation instance consisting of two entities and their context, openre aims to identify other instances which mention the same relation. to achieve this, openre is commonly formulated as a clustering or pair-matching task. therefore the most critical challenge for openre is how to learn effective representations for relation instances and then cluster them. to this end, @xcite adopts topic model @xcite to generate latent relation type for unlabelled instances. later works start to utilize datasets collected using distant supervision for model training. along this line, @xcite utilizes an auto-encoder model and trains the model through self-supervised signals from entity link predictor. @xcite encodes each instance with pretrained language model @xcite and learn the representation by self-supervised signals from pseudo labels. unfortunately, current openre models are often unstable and easily collapsed @xcite . for example, openre models frequently cluster all relation instances with context "was born in" into the relation type born in place because they share similar context information. however, "was born in" can also refer to the relation born in time. furthermore, current models also tend to cluster two relation instances with the same entities (i.e., relation instances with the same head and tail entities) or the same entity types into one relation. this problem can be even more severe if the dataset is generated using distant supervision because it severely relies on prototypical context and entity information as supervision signals and therefore lacks of diversity. in this paper, we attempt to explain and resolve the above-mentioned problem in openre from a causal view. specifically, we formulate the process of openre using a structural causal model (scm) @xcite , as shown in figure 1 . the main assumption behind the scm is that distant supervision will generate highly correlated relation instances to the original prototypical instance, and there is a strong connection between the generated instance to the prototypical instance through either their entities or their context. for example, " @xcite was born in [california]" and " @xcite was born in @xcite " are highly correlated because they share similar context "was born in" and entity "jobs". such connection will result in spurious correlations, which appear in the form of the backdoor paths in the scm. then the spurious correlations will mislead openre models, which are trained to capture the connection between entities and context to the relation type. based on the above observations, we propose element intervention, which conducts backdoor adjustment on entities and context respectively to block the backdoor paths. however, due to the lack of supervision signals, we cannot directly optimize towards the underlying causal effects. to this end, we further propose two surrogate implementations on the adjustments on context and entities, respectively. specifically, we regard the instances in the original datasets as the relation prototypes. then we implement the adjustment on context through a hierarchy-based entity ranking (hyber), which fixes the context, samples related entities from an entity hierarchy tree and learns the causal relation through rank-based learning. besides, we implement the adjustment on entities through a generation-based context con-trasting (gcc), which fixes the entities, generates positive and negative contexts from a generationbased model and learns the causal effects through contrastive learning. we conduct experiments on different unsupervised relation extraction datasets. experimental results show that our method outperforms previous state-of-the-art methods with a large margin and suffers much less performance discrepancy between different datasets, which demonstrate the effectiveness and robustness of the proposed methods. ## openre from causal view in this section, we formulate openre from the perspective of structural causal model and give the theoretical proof for intervention methods that block the backdoor paths from relation elements (i.e., context and entity pair) to the latent relation types. ## element intervention implementation as we mentioned above, it is difficult to directly optimize the causal effects via equation (4). to tackle this issue, this section provides a detailed implementation to approximate the causal effects. specifically, we regard all relation instances in the original data as the prototypical relation instance p , and then generate highly correlated relation instances x from p via a hierarchy-based sampling and generation-based contrasting. then we regard structural signals from the entity hierarchy and confidence score from the generator as distant supervision signals, and learn the causal effects via ranking-based learning and contrastive learning. ## related work current success of supervised relation extraction methods @xcite @xcite @xcite depends heavily on large amount of annotated data. due to this data bottleneck, some weakly-supervised methods are proposed to learn relation extraction models from distantly labeled datasets @xcite @xcite or few-shot datasets @xcite @xcite . however, these paradigms still require pre-defined relation types and therefore restricts their application to open scenarios. open relation extraction, on the other hand, aims to cluster relation instances referring to the same underlying relation without pre-defined relation types. previous methods for openre can be roughly divided into two categories. the generative method @xcite formulates openre using a topic model, and the latent relations are generated based on the hand-crafted feature representations of entities and context. while the discriminative method is first proposed by @xcite , which learns the model through the self-supervised signal from entity link predictor. along this line, @xcite propose the self-ore that learns the model through pseudo label and bootstrapping technology. however, @xcite point out that previous openre methods severely suffer from the instability, and they also propose two regularizers to guide the learning procedure. but the fundamental cause of the instability is still undiscovered. in this paper, we revisit the procedure of openre from a causal view. by formulating openre using a structural causal model, we identify the cause of the above-mentioned problems, and alleviate the problems by element intervention. there are also some recent studies try to introduce causal theory to explain the spurious correlations in neural models @xcite @xcite @xcite @xcite . however, to the best of our knowledge, this is the first work to revisit openre from the perspective of causality. ## conclusions in this paper, we revisit openre from the perspective of causal theory. we find that the strong connections between the generated instance to the prototypical instance through either their entities or their context will result in spurious correlations, which appear in the form of the backdoor paths in the scm. then the spurious correlations will mislead openre models. based on the observations, we propose element intervention to block the backdoor paths, which intervenes on the context and entities respectively to obtain the underlying causal effects of them. we also provide two specific implementations of the interventions based on entity ranking and context contrasting. experimen-tal results on two openre datasets show that our methods outperform previous methods with a large margin, and suffer the least performance discrepancy between datasets, which indicates both the effectiveness and stability of our methods.