A Simple Model for Distantly Supervised Relation Extraction (2022)

Abstract: Distantly supervised relation extraction is challenging due to the noise within data. Recent methods focus on exploiting bag representations based on deep neural networks with complex de-noising scheme to achieve remarkable performance. In this paper, we propose a simple but effective B ERT-based G raph convolutional network M odel (i.e., BGM). Our BGM comprises of an instance embedding module and a bag representation module. The instance embedding module uses a BERT-based pretrained language model to extract key information from each instance. The bag representaion module constructs the corresponding bag graph then apply a convolutional operation to obtain the bag representation. Our BGM model achieves a considerable improvement on two benchmark datasets, i.e., NYT10 and GDS.

## introduction in the distant supervision relation extraction (ds-re) setting, handling the noisy training data is a major challenge for downstream applications. to alleviate the severe noise problem in ds-re, @xcite incorporate the multi-instance learning (mil) framework. under this framework, the instances (i.e., sentences) for an identical entity pair are regarded as a bag. however, learning effective bag representations from the noisy data is a challenge resulting in unsatisfactory performance. recently, to obtain effective bag representations, various neural models are incorporated. @xcite propose a selective attention mechanism to capture relatively informative instances forming the bag representation. @xcite use graph convolution network (gcn) @xcite to encode syntactic information obtained from a dependency parser. however, the gcn's capacity in capturing the correlation among instances is not sufficiently explored. meanwhile, a prevalent trend is to use pretrained language models (plms) for various nlp tasks @xcite @xcite . plms work without explicit linguistic features and side-information like pos tags and entity types. @xcite use a plm to incorporate more linguistic and semantic information. the correlations among instances are implicitly represented through a naïve attention mechanism. very recently, @xcite propose plms combined with the contrastive instance learning (cil) to build bag representations. cil captures the correlation among instances through data augmentation with positive and negative pairs. these plms-based methods achieve a fabulous performance. hence, a question arises, how about combining plms and gcns to learn the instance correlations for bag representations? towards this goal, we propose an impressively simple model, i.e, bgm. our bgm comprises of a plm (e.g., bert) and a concise gcn. the plm brings accurate contextual representations for instances. the gcn whose nodes are instances encodes the correlation of instances in a bag through mutually aggregation. thus, we finetune an off-theshelf bert with a gcn in an end-to-end fashion. with the obtained bag representation, the entity relation in a given sentence is then predicted. our contributions are twofold. 1) we propose bgm model for ds-re. our bgm only comprises a plm and a gcn but without any prior knowledge or data augmentation. 2) experimental results on two benchmark datasets show that bgm achieves a consistent improvement on performance. ## related work the noisy data is a major challenge in ds-re for downstream applications. previous works can be divided into two categories: pcnn-based methods and plms-based methods. the former methods use piecewise convolutional neural networks (i.e., pcnn) as the backbone to encode sentences. thus, various methods are proposed to acquire effective bag representations. @xcite propose the selective attention mechanism over the bag's instances to use more informative instances. this method implicitly models the instance correlations. subsequently, @xcite propose a model by providing a better supervision with soft labels as golden labels. @xcite exploit a hierarchical attention paradigm to better capture valid instances. combined with previous intra-bag attention, @xcite design an inter-bag attention to obtain bag-group representations. @xcite build a co-occurrence graph to learn embeddings to enhance bag representations. @xcite employ a pattern-aware self-attention network to automatically discover relational patterns for pre-trained transformers. recently, plms-based methods achieve remarkable performance in various nlp tasks, including ds-re. @xcite adopt plms to incorporate a great deal of commonsense knowledge. @xcite focus on relational tokens using the sub-tree parsing and capture informative instances with fine-tuning bert. @xcite combine plms with contrastive learning with data augmentation to improve the overall performance. note that the three plms-based methods follow the soft attention mechanism adopted in @xcite . the attention mechanism uses the bag's target relation to emphasize the instances which better express the bag relation. in this paper, we propose a simple but effective model bgm. the bgm uses a plm and a gcn for ds-re. to the best of our knowledge, we are the first to use gcn to learn the bag representation directly over the instances. note that @xcite also use gcns but for encoding the syntactic tree of instances. in addition, compared with @xcite , we do not follow their selective attention with bag's target relation. our bgm uses a self-attention mechanism to capture the correlation among instances. ## method suppose that an instance bag b (e h ,e t ) = {s 1 , s 2 , ..., s n s } contains n s instances which all include the entity-pair e h , e t . each instance @xmath0, x 2 , ..., x n w } contains n w words. the pretrained language model @xcite entity1 ## discussion our bgm performs its calculation on the entire graph. the gcn layer calculates weights adopting the method of self-attention. in other words, the node features of the entire graph are updated after one calculation, and the learned parameters are not heavily related to the graph structure. compared with other attention-based methods, we do not follow their selective attention with the bag's target relation. our bgm uses a self-attention mechanism to capture the correlation among instances which are taken as the nodes of the graph. therefore, gcn combined with self-attention is one of effective ways for the setting of ds-re. ## conclusion in this paper, we propose a simple but effective model, a.k.a. bgm based on plms and gcn for ds-re. each instance is represented using a bertbased pre-trained language model. to capture the instance correlations, gcn for multiple instances within a bag is used. with this type of bag representation, a cross-entropy loss is applied for predicting the relation between entities. extensive experiments on two benchmark datasets show the superior performance. in our future, we will investigate the hidden theory in-depth for better explainability of our bgm model. in addition, extending the bgm to dealing with the case of a single-instance bag is an interesting problem.