Separating Retention from Extraction in the Evaluation of End-to-end R elation E xtraction (2021)

Abstract: State-of-the-art NLP models can adopt shallow heuristics that limit their generalization capability (McCoy et al., 2019). Such heuristics include lexical overlap with the training set in Named-Entity Recognition (Taille et al., 2020) and Event or Type heuristics in Relation Extraction (Rosenman et al., 2020). In the more realistic end-to-end RE setting, we can expect yet another heuristic: the mere retention of training relation triples. In this paper we propose two experiments confirming that retention of known facts is a key factor of performance on standard benchmarks. Furthermore, one experiment suggests that a pipeline model able to use intermediate type representations is less prone to over-rely on retention.

## introduction information extraction (ie) aims at converting the information expressed in a text into a predefined structured format of knowledge. this global goal has been divided into subtasks easier to perform automatically and evaluate. hence, named entity recognition (ner) and relation extraction (re) are two key ie tasks among others such as coreference resolution (cr), entity linking or event extraction. traditionally performed as a pipeline @xcite , these two tasks can be tackled jointly in order to model their interdependency, alleviate error propagation and obtain a more realistic evaluation setting @xcite . following the general trend in natural language processing (nlp), the recent quantitative improvements reported on entity and relation extraction benchmarks are at least partly explained by the use of larger and larger pretrained language models (lms) such as bert @xcite to obtain contextual word representations. concurrently, code for reproducing our evaluation settings is available at github.com/btaille/retex there is a realization that new evaluation protocols are necessary to better understand the strengths and shortcomings of the obtained neural network models, beyond a single holistic metric on an hold-out test set @xcite . in particular, generalisation to unseen data is a key factor in the evaluation of deep neural networks. it is all the more important in ie tasks that revolve around the extraction of mentions: small spans of words that are likely to occur in both the evaluation and training datasets. this lexical overlap has been shown to be correlated to neural networks performance in ner @xcite . for pipeline @xcite and @xcite expose shallow heuristics in neural models: relying too much on the type of the candidate arguments or on the presence of specific triggers in their contexts. in end-to-end relation extraction, we can expect that these ner and re heuristics are combined. in this work, we argue that current evaluation benchmarks measure both the desired ability to extract information contained in a text but also the capacity of the model to simply retain labeled (head, predicate, tail) triples during training. and when the model is evaluated on a sentence expressing a relation seen during training, it is hard to disentangle which of these two behaviours is predominant. however, we can hypothesize that the model can simply retrieve previously seen information acting like a mere compressed form of knowledge base probed with a relevant query. thus, testing on too much examples with seen triples can lead to overestimate the generalizability of a model. even without labeled data, lms are able to learn some relations between words that can be probed with cloze sentences where an argument is masked @xcite . this raises the additional question of lexical overlap with the orders of magnitude larger unlabeled lm pretraining corpora that will remain out of scope of this paper. ## datasets and models we study three recent end-to-end re models on conll04 @xcite , ace05 @xcite and scierc @xcite . they rely on various pretrained lms and for a fairer comparison, we use bert @xcite on ace05 and conll04 and scibert @xcite on scierc 1 . pure @xcite follows the pipeline approach. the ner model is a classical span-based model @xcite . special tokens corresponding to each predicted entity span are added and used as representation for relation classification. for a fairer comparison with other models, we study the approximation model that only requires one pass in each encoder and limits to sentence-level prediction. however, it still requires finetuning and storing two pretrained lms instead of a single one for the following models. spert (eberts and ulges, 2020) uses a similar span-based ner module. re is performed based on the filtered representations of candidate arguments as well as a max-pooled representation of their middle context. while entity filtering is close to the pipeline approach, the ner and re modules share a common entity representation and are trained jointly. we also study the ablation of the max-pooled context representation that we denote ent-spert. two are better than one (tabto) @xcite intertwines a sequence encoder and a table encoder in a table filling approach @xcite . contrary to previous models the pretrained lm is frozen and both the final hidden states and attention weights are used by the encoders. the prediction is finally performed by a multi-dimensional rnn (md-rnn). because it is not based on span-level predictions, this model cannot detect nested entities, e.g. on scierc. ## partitioning by lexical overlap following @xcite , we partition the entity mentions in the test set based on lexical overlap with the training set. we distinguish seen and unseen mentions and also extend this partition to relations. we denote a relation as an exact match if the same (head, predicate, tail) triple appears in the train set; as a partial ## more implementation details in appendix a match if one of its arguments appears in the same position in a training relation of same type; and as new otherwise. we implement a naive retention heuristic that tags an entity mention or a relation exactly present in the training set with its majority label. we report micro-averaged precision, recall and f1 scores for both ner and re in an entity mention is considered correct if both its boundaries and type have been correctly predicted. for re, we report scores in the boundaries and strict settings @xcite . in the boundaries setting, a relation is correct if its type is correct and the boundaries of its arguments are correct, without considering the detection of their types. the strict setting adds the requirement that the entity type of both argument is correct. ## swapping relation heads and tails a second experiment to validate that retention is used as a heuristic in models' predictions is to modify their input sentences in a controlled manner ## related work several works on generalization of ner models mention lexical overlap with the training as a key indicator of performance. @xcite separate mentions in the test set as seen and unseen during training and measure out-of-domain generalization in an extensive study of two crf based models and senna combining a convolutional neural network with a crf @xcite . @xcite compare the effect of introducing contextual embeddings in the classical bilstm-crf architecture in a similar setting and show that they help close the performance gap on unseen mentions and domains. @xcite ; @xcite study the influence of several properties such as lexical overlap, label consistency and entity length on state-of-the-art models performance. they model these properties as continuous scores associated to each mention and bucketized for evaluation. lexical overlap has also been mentioned in coreference resolution @xcite where coreferent mentions tend to co-occur in the test and train sets. in this line of works, the impact of lexical overlap is measured either by separating performance depending on the property of mentions (seen or unseen) or with outof-domain evaluation with a test set from a different dataset with lower lexical overlap with the train set. another recently proposed method for finegrained evaluation of nlp models beyond a single benchmark score is to modify the test sentences in a controlled manner. @xcite expose lexical overlap as a shallow heuristic adopted by state-of-the-art natural language inference models, especially by swapping subject and object of verbs in the hypothesis of some examples where the premise entails the hypothesis. while such a modification changes the label of these examples to non-entailment, all models tested show a spectacular drop of accuracy on these models. @xcite propose a broader set of test set modifications to individually test robustness of nlp models to several patterns such as the introduction of negation, swapping words with synonyms, changing tense and much more. in pipeline re where ground truth candidate arguments are given, models often use intermediate representations based on entity types that reduce lexical overlap issues. however, rosenman et al. (2020) show that they still tend to adopt shallow heuristics based on the type of the arguments and the presence of triggers indicative of the presence of a relation. they propose hard cases with several mentions of same types for which relation classifiers struggle connecting the correct pair. concurrently, @xcite confirm that re benchmarks present shallow cues such as the type of the candidate arguments that can be used alone to infer the relation. we propose to extend previous work on ner and re to the more realistic end-to-end re setting with two of the previously described approaches: 1) separating performance by lexical overlap of mentions or argument pairs and 2) modifying some conll04 test examples by swapping relations heads and tails. ## conclusion in this paper, we study three state-of-the-art endto-end relation extraction models in order to highlight their tendency to retain seen relations. we confirm that retention of seen mentions and relations play an important role in overall re performance and can explain the relatively higher scores on conll04 and ace05 compared to scierc. furthermore, our experiment on swapping relation heads and tails tends to show that the intermediate manipulation of type representations instead of lexical features enabled in the pipeline pure model makes it less prone to over-rely on retention. while the limited extend of our swapping experiment is an obvious limitation of this work, it shows limitations of both current benchmarks and models. it is an encouragement to propose new benchmarks that might be easily modified by design to probe such lexical overlap heuristics. contextual information could for example be contained in templates of that would be filled with different (head, tail) pairs either seen or unseen during training. furthermore, pretrained language models can already capture relational information between phrases @xcite and further experiments could help distinguish their role in the retention behaviour of re models.