Explore Unsupervised Structures in Pretrained Models for Relation Extraction (2022)

Abstract: Syntactic trees have been widely applied in relation extraction (RE). However, since parsing qualities are not stable on different text domains and a pre-defined grammar may not well fit the target relation schema, the introduction of syntactic structures sometimes fails to improve RE performances consistently. In this work, we study RE models with various unsupervised structures mined from pre-trained language models (e.g., BERT). We show that, similar to syntactic trees, unsupervised structures are quite informative for RE task: they are able to obtain competitive (even the best) performance scores on benchmark RE datasets (ACE05, WebNLG, SciERC). We also conduct detailed analyses on their abilities of adapting new RE domains and influence of noise links in those structures. The results suggest that unsupervised structures are reasonable alternatives of commonly used syntactic structures in relation extraction models.

## introduction relation extraction (re) tasks aim to extract pairs of text spans (entities), each of which expresses certain relation semantics. as shown in figure 1 , w 1 w 2 and w 5 are two extracted entities with the type of person (per) and geography (gpe) respectively, and they carry a physical location relation @xcite . while end-to-end neural relation extractors have achieved great success @xcite @xcite , proper prior knowledge or external features are still crucial to make re models more flexible and robust. syntactic trees (especially, dependency trees) are such features which are widely applied in re systems @xcite @xcite @xcite et al., 2020; @xcite . the prior dependency relations among words are believed to be helpful for spotting most user-defined relations. however, to fulfil their advantages, there are still some challenges to tackle. for example (figure 1 ), a syntactic tree usually characterizes the full syntactic structure of a sentence, but some parts of the dense structure may be redundant (even misleading) when detecting sparse relations among entities. another problem is that since syntactic parsers are trained on external treebanks, parsing qualities are no longer guaranteed when the treebanks' text domain mismatch the target re text domain. here, we investigate another kind of external structural features, namely unsupervised structures mined from pre-trained language models (plms). like other nlp tasks, plms have been deployed in re models due to their strong abilities on unsupervisely aggregating semantic information from large corpora and contextual information from the whole input sentence. current re models use plms to encode words: the classifier queries word embeddings from plms as features. we ask that, apart from embedding features, whether re models could also query plms for structural features. on the one side, we would like those structures to be easy to access as syntactic structures, on the other side, we would expect them to be able to provide rich syntactic and semantic information as plm embeddings. in this paper, we obtain unsupervised structures using three types of interpretation tools (attentionbased, occlusion-based and gradient-based). they basically use different kinds of information to characterize connections among input words, and output weighted complete graphs with words as nodes and connection scores as edge weights. then, we adopt graph convolutional networks (gcns) as a plug-and-play encoding module to incorporate structures into relation extraction models. we systematically test re models with unsupervised structures and find that, therefore, considering the cost of getting structural features (i.e., with or without human annotations), we think that unsupervised structures are promising features for future re models. ## preliminary pre-trained language models are important feature extractors in modern nlp systems. in this work, we will take the bert @xcite family as our demonstrative plms. main components of bert are transformer blocks @xcite which consist of multiple self-attention heads. for a token x i in an input sentence x of n tokens {x 1 , x 2 , . . . , x n }, an attention head computes the token's attention scores over all input tokens and outputs a hidden representation h i . we denote the encoding procedure as h @xmath0. graph convolutional networks (gcn) are powerful feature extractors for structured data. @xcite . they learn hidden representations for graph nodes by gradually collecting information from their neighbors. in relation extraction, gcns are popular tools for encoding external syntactic features @xcite @xcite . we follow @xcite by using a bi-directional version of gcn. denote {h ## how to get structures we describe the structures included in our re models. one kind of structure is derived from dependency trees, and another kind is mined from plms. both of them are represented by a n×n importance matrix g (also called saliency map). an entry g ij of g indicates the strength of connection between word x j and word x i . ## relation extraction model re model we apply different structures in the setting of joint entity relation extraction: given an input, the re model extracts both entities and relations among the extracted entities. the encoder part (figure 1 ) of our model is a stack of bert and gcn layers (with external structures). after obtaining hidden vector representations of each token, we apply a simple joint extraction procedure from @xcite . specifically, the model first performs a token-level sequence labelling for extracting entities. then, for each possible pairs of entities, it builds a representation of the pair by aggregating representations of left tokens, right tokens and inbetween tokens regarding that pair. the relation extractor consumes the entity pair representation and performs a multi-class classification. datasets we run experiments on three widely used datasets: ace05 foot_1 , scierc foot_2 and webnlg 6 . regarding text domains, ace05 collects texts from news, scierc texts are from ai conference proceedings, and webnlg are obtained by crowdsourcing with dbpedia relation triples as guidance. dataset statistics are shown in the metrics following previous works, we evaluate re models with micro precision, recall, and f1 score. for ace05 and scierc, we judge a correct extraction with strict evaluation criteria @xcite @xcite . for webnlg, we use partial matching criteria @xcite @xcite . the partial matching criteria considers only the head words of the entity pair and ignores the full entity span and types. ## conclusion we propose to study the performance of re models with variety of unsupervised structures mined from plms. we conducted a series of experiments on re to verify the effectiveness of them, and analyze their internal compositions and domain adaptation respectively. the main conclusion we reached is unsupervised structures are beneficial to re, and perform competitive or even better than dependency trees. we also find that unsupervised structures can enhance domain adaptation ability by simply adding raw texts during finetuning plms.