Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking (2022)

Abstract: Relation extraction (RE) is an important natural language processing task that predicts the relation between two given entities, where a good understanding of the contextual information is essential to achieve an outstanding model performance. Among different types of contextual information, the auto-generated syntactic information (namely, word dependencies) has shown its effectiveness for the task. However, most existing studies require modifications to the existing baseline architectures (e.g., adding new components, such as GCN, on the top of an encoder) to leverage the syntactic information. To offer an alternative solution, we propose to leverage syntactic information to improve RE by training a syntax-induced encoder on auto-parsed data through dependency masking. Specifically, the syntax-induced encoder is trained by recovering the masked dependency connections and types in first, second, and third orders, which significantly differs from existing studies that train language models or word embeddings by predicting the context words along the dependency paths. Experimental results on two English benchmark datasets, namely, ACE2005EN and SemEval 2010 Task 8 datasets, demonstrate the effectiveness of our approach for RE, where our approach outperforms strong baselines and achieve state-of-the-art results on both datasets.

## introduction relation extraction (re) provides deep analyses of the input text by extracting the relation between two given entities in the input. therefore, it is an important task in natural language processing (nlp) and is widely used in many downstream nlp applications such as summarization @xcite , question answering systems @xcite and text mining @xcite . to correctly extract the relation between two entities, it normally requires a good modeling and analysis of the input text. recent models such as lstm, transformers @xcite , and pre-trained language models (e.g., bert @xcite and xlnet @xcite ) have significantly improved the performance of re models with an important reason of their encoding power on contextual information. however, such models still reach a bottleneck because it is hard for them to capture structural information of the running text (which is essential for re) by modeling the text as a linear sequence of words. to deal with this situation, extra knowledge and features (e.g., syntactic knowledge) are used in many studies, while of all choices, the dependency parses have been widely used and demonstrated to be effective @xcite @xcite @xcite @xcite , for the reason that the dependency trees are able to provide long-distance word-word relations which are important structural complement to existing models for re. to leverage dependency information, most existing approaches in nlp either treat it as extra input features @xcite @xcite , which requires heavy feature engineering, or use complicated architectures @xcite @xcite @xcite @xcite @xcite to encode it, which suffers from the difficulty of designing an effective model. in addition, these approaches normally require dependency trees as extra input when processing sentences, and thus potentially suffer from noises from the dependency trees because of errors from automatic parsing. therefore, an alternative is needed to leverage dependency information, especially auto-generated figure 1 : an overview of our approach to train a syntax-induced encoder (highlighted in the red box). the left part shows the process to extract and mask dependencies (connection and type masking, respectively) in first, second, and third orders, where the word subscript denotes its sentential index. the right part illustrates the process to compute the scores of dependency connections and types in different orders to recover the ones that being masked. in this paper, we propose to enhance re through learning a good encoder equipped with dependency information, where the learning is carried out by a dependency-guided process. in detail, a dependency masking approach is designed to introduce such information, where we firstly apply an offthe-shelf dependency parser to large raw data and extract the dependency connections and types from the auto-parsed dependency trees, and then mask these connections and types so as to pre-train a syntax-induced encoder by recovering (predicting) them, which significantly differs from that of training word embeddings @xcite by predicting the context words along the dependency relations. in doing so, the dependency information weakly supervises the encoder and the pre-training on dependency masking ensures a selective learning process on those frequent and important dependency relations, which is more flexible than taking dependency parses (with noises) as fixed knowledge. in addition, by noting that higher order dependency information is beneficial in many cases @xcite @xcite , we further enhance our approach by pre-training with masking second and third order word dependencies rather than just doing it on the first order. once pre-trained, the resulted encoder is applied with ordinary fine-tune procedure for re. experimental results on two english benchmark datasets, namely, english ace2005en foot_1 and se-meval 2010 task 8 @xcite , for re demonstrate the effectiveness of our approach, which outperforms strong baselines and achieves state-of-the-art results on both of the datasets. ## the approach to learn a text encoder with important structural information for re, we propose to pre-train it with masking and recovering word-word dependency connections and types that are auto-analyzed from existing parsers. the resulted syntax-induced encoder is thus weakly supervised by such information and provided with necessary syntax integration. in doing so, the dependency information is introduced during pre-training the encoder, thus no extra input is required in applying it to real applications, avoiding particular design of models to leverage such information during inference. figure 1 illustrates the architecture of our approach to learn from an input sentence with n words and its dependency tree t x , so that the masking and recovering can be formalized by ## related work relation extraction is an important task in nlp and it requires deep understanding of the input text to achieve model performance. therefore, in addition to leveraging advanced text encoders (e.g., bi-lstm, transformer @xcite , bert @xcite ) to capture contextual information, structural information, namely, the dependency information, of the running text has been widely used as an effective resource to improve re @xcite @xcite @xcite . in most recent studies in nlp, the dependency information is leveraged either as extra input features @xcite @xcite or modeled by complicated graph-based architectures, such as convolutions neural networks @xcite and tree lstms @xcite . previous studies also tried to use attention mechanism to weight different dependency features @xcite @xcite and lstm to en-code linearized dependency path @xcite . in addition to modeling dependency information, there is another track to leverage it by pre-training dependency-based word embeddings through predicting the context words in auto-parsed dependency trees @xcite or designing an auxiliary module to learn the dependency information by treating the dependencies as additional input during pre-training @xcite . this research follows the pre-training paradigm and offers an alternative way to do so. specifically, compared with existing studies, our approach leverages the dependency information by inducing it to the pre-training process through masked dependency prediction, whose object is to predict the masked dependencies rather than directly using it as extra fixed input along with the input sentence through an additional module. also, since the dependency information is learnt by the syntax-induced encoder and the encoder is further fine-tuned on the training data in the same way as general re model, our approach neither requires any additional input features nor needs complicated architectures to encode them, which allows our model to be efficient in inference. ## conclusion in this paper, we propose to use dependency masking and recovering to improve the text encoder and thus enhance re that requires deep understanding of the running text, where the encoder is trained on large scaled auto-parsed data. specifically, we try such masking on first, second, and third order word dependencies from the auto-parsed data, and train a base encoder that is able to recover all the masked dependencies. in doing so, the resulted syntaxinduced encoder is integrated with dependency information in a dynamic and flexible manner and it can be directly applied to different downstream tasks requiring no extra input or particular design to accommodate dependency information. experimental results and analyses on two english benchmark datasets (i.e., ace05 and semeval) for re show the effectiveness of our approach, where our approach outperforms strong baselines and achieve state-of-the-art on both datasets.