Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction (2020)

Abstract: Despite the recent progress, little is known about the features captured by state-of-the-art neural relation extraction (RE) models. Common methods encode the source sentence, conditioned on the entity mentions, before classifying the relation. However, the complexity of the task makes it difficult to understand how encoder architecture and supporting linguistic knowledge affect the features learned by the encoder. We introduce 14 probing tasks targeting linguistic properties relevant to RE, and we use them to study representations learned by more than 40 different encoder architecture and linguistic feature combinations trained on two datasets, TACRED and SemEval 2010 Task 8. We find that the bias induced by the architecture and the inclusion of linguistic features are clearly expressed in the probing task performance. For example, adding contextualized word representations greatly increases performance on probing tasks with a focus on named entity and part-of-speech information, and yields better results in RE. In contrast, entity masking improves RE, but considerably lowers performance on entity type related probing tasks.

## introduction relation extraction (re) is concerned with extracting relationships between entities mentioned in text, where relations correspond to semantic categories such as org:founded by, person:spouse, or org:subsidiaries (figure 1 ). neural models have shown impressive results on this task, achieving state-of-the-art performance on standard datasets like semeval2010 task 8 @xcite @xcite , ta-cred @xcite @xcite , and nyt @xcite @xcite . the majority of models implement an encoder architecture to learn a fixed size representation of the input, e.g. a sentence, which is passed to a classification layer to predict the target relation label. these good results suggest that the learned representations capture linguistic and semantic properties of the input that are relevant to the downstream re task, an intuition that was previously discussed for a variety of other nlp tasks by @xcite . however, it is often unknown which exact properties the various models have learned. our aim is to pinpoint the information a given re model is relying on, in order to improve model performance as well as to diagnose errors. a general approach to model introspection is the use of probing tasks. probing tasks @xcite , or diagnostic classifiers, are a well established method to analyze the presence of specific information in a model's latent representations, e.g. in machine-translation @xcite , language modeling @xcite , and sentence encoding @xcite . for each probing task, a classifier is trained on a set of representations, and its performance measures how well the information is encoded. the probing task itself is typically selected in accordance with the downstream task, e.g. an encoder trained on re may be probed for the entity type of a relation argument. if the classifier correctly predicts the type, it implies the encoder retains entity type information in the representations, which also directly inform the relation prediction. the simplicity of this ap-proach makes it easier to pinpoint the information a model is relying on, as opposed to probing the downstream task directly. our goal in this paper is to understand which features of the input a model conditioned on relation extraction has learned as useful for the task, in order to be able to better interpret and explain model predictions. relation extraction literature is rich with information about useful features for the task @xcite @xcite . consequently, our initial question is whether and how good the sentence representations learned by state-of-the-art neural re models encode these well-known features, such as e.g. argument entity types, dependency path or argument distance features. another question is how the prior imposed by different encoding architectures, e.g. cnn, rnn, graph convolutional network and self-attention, affects the features stored in the learned sentence representations. finally, we would like to understand the effect of additional input features on the learned sentence representations. these include explicit semantic and syntactic knowledge like entity information and grammatical role, and as recently proposed, contextualized word representations such as elmo @xcite and bert @xcite . we therefore significantly extend earlier work on probing tasks as follows: ## experiment setup this section first introduces the four sentence encoding architectures we consider for evaluation ( §3.1), followed by a description of the supporting linguistic knowledge we evaluate: entity masking and contextualized word representations ( §3.2). we also introduce the two datasets we use for training the relation extraction models and probing the sentence representations ( §3.3). ## results baseline performances are reported in the top section of encoder architecture for most probing tasks, except sentlen and argord, a proper encoder clearly outperforms bag-of-embeddings (boe), which is coherent with the findings of @xcite and @xcite . similarly, the results indicate that the prior imposed by the encoder architecture preconditions the information encoded in the learned embeddings. models with a local or recency bias (cnn, bilstm) perform well on probing tasks with local focus, such as poshead{l,r} and postail{l,r} and distance related tasks (argdist, argord). similarly, models with access to dependency information (gcn) perform well on tree related tasks (sdptreedepth). due to the graph pruning step @xcite , the gcn is left with a limited view of the dependency tree, which explains the low performance on treedepth. surprisingly, while self-attention exhibits superior performance on the re task, it consistently performs lower on the probing tasks compared to the other encoding architectures. this could indicate self-attention encodes "deeper" linguistic information into the sentence representation, not covered by the current set of probing tasks. probing tasks compared to the baselines, all proper encoders exhibit consistently high performance on typehead and typetail, clearly highlighting the importance of entity type information to re. in contrast, encoders trained on the downstream task perform worse on sentlen, which intuitively makes sense, since sentence length is mostly irrelevant for re. this is consistent with @xcite , who found sentlen performance to decrease for models trained on more complex downstream tasks, e.g. neural machine translation, strengthen the assumption that, as a model captures deeper linguistic properties it will tend to forget about this superficial feature. with the exception of the cnn, all encoders consistently show low performance on the argument distance (argdist) task. a similar performance pattern can be observed for argord, where models that are biased towards locality (cnn and bilstm) perform bet-ter, while models that are able to efficiently model long range dependencies, such as gcn and s-att., show lower performance. the superior re task performance of the latter indicates that their bias may allow them to learn "deeper" linguistic features. the balanced performance of cnn, bilstm and gcn encoders across part-of-speech related tasks (posheadl, posheadr, postaill, postailr) highlights the importance of part-of-speech-related features to re, again with the exception of s-att., which performs just slightly above baselines. on treedepth and sdptreedepth (with gcn as the exception), average performance in many cases ranges just slightly above baseline performance, suggesting that treedepth requires more nuanced syntactic information, which the models fail to acquire. the good performance on grammatical role tasks (grhead, grtail) once more emphasizes the relevance of this feature to re, with the gcn exhibiting the best performance on average. this is unsurprising, because the gcn focuses on token-level information along the dependency path connecting the arguments, and hence seems to be able to capture grammatical relations among tokens more readily than the other encoders (even though the gcn also does not have access to the dependency labels themselves). entity masking perhaps most interestingly, masking entity mentions with their respective named entity and grammatical role information considerably lowers the performance on entity type related tasks (typehead and typetail). this indicates that masking forces the encoder's focus away from the entity mentions, which is confirmed by the performance decrease in probing tasks with a focus on argument position and distance, e.g. argdist, argord, and sentlen. cnn and bilstm encoders exhibit the greatest decrease in performance, suggesting a severe overfitting to specific entity mentions when no masking is applied. in comparison, the gcn shows less tendency to overfit. surprisingly, with entity masking the self-attentive encoder (s-attn.) increases its focus on entity mentions and their surroundings as suggested by the performance increase on the distance and argument related probing tasks. word representations adding contextualized word representations computed by elmo or bert greatly increases performance on probing tasks with a focus on named entity and part-of-speech information. this indicates that contextualized word representations encode useful syntactic and semantic features relevant to re, which is coherent with the findings of @xcite and @xcite , who both highlight the effectiveness of linguistic features encoded in contextualized word representations (deep language representations) for downstream tasks. the improved performance on syntactic and semantic abilities is also reflected in an overall improvement in re task performance. compared to elmo, encoders with bert generally exhibit an overall better and more balanced performance on the probing tasks. this is also reflected in a superior re performance, suggesting that a bidirectional language model encodes linguistic properties of the input more effectively. somewhat surprisingly, bert without casing performs equally or better on the probing tasks focused on entity and part-of-speech information, compared to the cased version. while this intuitively makes sense for semeval, as the dataset focuses on semantic relations between concepts, it is surprising for tacred, which contains relations between proper entities, e.g. person and company names, with casing information more important to identify the entity type. probing vs. relation extraction one interesting observation is that encoders that perform better on probing tasks do not necessarily perform better on the downstream re task. for example, cnn+elmo scores highest for most of the probing tasks, but has an 8.1 lower f1 score than the best model on this dataset, s-att.+bert cased with masking. similarly, all variants of the self-attentive encoder (s-att.) show superior performance on re but consistently come up last on the probing tasks, occasionally performing just above the baselines. @xcite observed a similar phenomena for encoders trained on neural machine translation. ## conclusion we introduced a set of probing tasks to study the linguistic features captured in sentence encoder representations trained on relation extraction. we conducted a comprehensive evaluation of common re encoder architectures, and studied the effect of explicitly and implicitly provided semantic and syntactic knowledge, uncovering interesting properties about the architecture and input features. for example, we found self-attentive encoders to be well suited for the re on sentences of different complexity, though they consistently perform lower on probing tasks; hinting that these architectures capture "deeper" linguistic features. we also showed that the bias induced by different architectures clearly affects the learned properties, as suggested by probing task performance, e.g. for distance and dependency related probing tasks. in future work, we want to extend the probing tasks to also cover specific linguistic patterns such as appositions, and also investigate a model's ability of generalizing to specific entity types, e.g. company and person names.