E xposing S hallow H euristics of R elation E xtraction M odels with C hallenge D ata (2020)

Abstract: The process of collecting and annotating training data may introduce distribution artifacts which may limit the ability of models to learn correct generalization behavior. We identify failure modes of SOTA relation extraction (RE) models trained on TACRED, which we attribute to limitations in the data annotation process. We collect and annotate a challenge-set we call Challenging RE (CRE), based on naturally occurring corpus examples, to benchmark this behavior. Our experiments with four state-of-the-art RE models show that they have indeed adopted shallow heuristics that do not generalize to the challenge-set data. Further, we find that alternative question answering modeling performs significantly better than the SOTA models on the challenge-set, despite worse overall TACRED performance. By adding some of the challenge data as training examples, the performance of the model improves. Finally, we provide concrete suggestion on how to improve RE data collection to alleviate this behavior.

## introduction in the relation extraction (re) task, our goal is, given a set of sentences @xmath0, e 1 , e 2 , r) where a relation @xmath1, each represented as a span over s). re is often represented as relation classification (rc): given a triplet of (s, e 1 , e 2 ), determine which relation @xmath2, or indicate @xcite . this can be presented as a set of |r| binary decision problems, (s, e 1 , e 2 , r) → {0, 1}: return 1 for tuples for which the relation holds, and 0 otherwise. the reduction between re and rc is clear: given a sentence, extract all entity pair candidates (given a ner system), and run the rc problem on each of them. indeed, contemporary methods are all rc methods, and the popular tacred large-scale relation extraction dataset is annotated for rc: each instance in the dataset is a triplet of (s, e 1 , e 2 ) and is associated with a label @xmath3. importantly, the annotation is non exhaustive: not all e 1 , e 2 pairs in the dataset are annotated (only 17.2% of the entity pairs whose type match a tacred relation are). while this saves a lot of annotation effort, as we show this also leads to sub-optimal behavior of the trained models, and hinders our ability to properly assess their real-world utility. we show that state of the art models trained on tacred are often "right for the wrong reasons" @xcite : instead of learning to perform the intended task, they rely on shallow heuristics which are effective for solving many dataset instances, but which may fail on more challenging examples. in particular, we show two concrete heuristics: classifying based on entity types, and classifying based on the existence of an event without linking the event to its arguments. we show that while they are not well attested in the dev and test sets, these challenging examples do occur in practice. we introduce cre (challenging re), a challenge set for quantifying and demonstrating the problem, and show that four sota rc models significantly fail on the challenge set. we release the challenge set to encourage future research on better models. 1 while we demonstrate the problem on tacred, we stress that the model behaviors we expose are directly linked to the dataset construction procedure, and will likely occur in any dataset that is created in a similar fashion. we propose guidelines to help guide better datasets, and in particular better evaluation sets, in the future. heuristic examples prediction correct? event (1) edward was born in york in 1561 , the son of john, and his wife mary. birth date (2) edward was born in york in 1561 , the son of john , and his wife mary. birth date (3) edward was born in york in 1561 , the son of john, and his wife mary . birth date event (4) loomis is married to hilary mills , who wrote a biography about norman mailer. spouse (5) loomis is married to hilary mills, who wrote a biography about norman mailer . spouse (6) loomis is married to hilary mills , who wrote a biography about norman mailer. spouse (7) loomis is married to hilary mills , who wrote a biography about norman mailer . spouse (8) loomis is married to hilary mills , who wrote a biography about norman mailer . spouse (9) loomis is married to hilary mills , who wrote a biography about norman mailer . spouse type & event (10) ucf also has 400 beds at the rosen college apartments community, located on rosen college of hospitality management campus. # members (11) ucf also has 400 beds at the rosen college apartments community , located on the rosen college of hospitality management campus. # members may alleviate this problem: models trained for qa are better at linking events to their arguments. while performing worse on tacred overall, they perform significantly better on the challenge set. we demonstrate that the event+type heuristic is a particularly strong one for relation-classification datasets, and is widely used by trained state-of-theart relation classifiers. ## challenge set consider the date of birth relation, that holds between a person e 1 and a year e 2 , and the classification instance: a model making use of the event+type heuristic will correctly classify this relation. 3 the heuristic is challenged by sentences that include multiple entities of an applicable type. for example: while such sentences are frequent in practice, these cases are not represented in the dataset: in only 17.2% of the sentences in tacred more than a single pair is annotated (and only 3.46% of the sentences have more than one different annotated labels in the sentence) . additionally, due to the data collection method @xcite , if a sentence includes a positive pair for a relation @xmath4, it is significantly more likely that this pair will be chosen for annotation rather than a matching-type pair with a no-relation label. in other words, the data collection process leads to no-relation labels between a pair of entities being assigned with very high probability to sentences in which other pairs in the sentence also do not adhere to a relation of interest. as a result, for models trained on tacred, the model is incentivized to learn to identify the existence of the relation in the sentence, irrespective of the arguments. there is no signal in the data to incentivize the model to distinguish cases where the relation holds between the given arguments, from cases where the relation holds but between a different pair of arguments. we expect the same to hold for any large-scale rc dataset created in a similar manner. 4 challenge set construction. we seek a benchmark to highlight rc models' susceptibility to the event+type heuristic. the benchmark takes the form of a challenge/contrast set; a collection of related examples that specialize in a specific failure case, meant only for evaluation and not training @xcite . in contrast to the nli challenge set of @xcite , in which challenge instances are based on synthetic examples, our benchmark is based on naturally occurring examples from real-world corpora. methodology. coming up with a set of realworld sentences that demonstrates a failure mode is not easy. the main challenge is in identifying potential candidate sentences to pass to manual annotation. to identify such cases, we require an effective method for sampling a population that is likely to exhibit the behavior we are interested in. we propose the following challenge-set creation methodology: (1) use a strong seed-model to perform large-scale noisy annotation; (2) identify suspicious cases in the model's output; (3) manually verify (annotate) suspicious cases. in stage (2), we identify suspicious cases by looking for sentences in which: (a) there are at least two entity-pairs of a ne type which is compatible to a tacred relation (in most of the cases, the entitypairs share one of the items, i.e. (e 1 , e 2 ), (e 1 , e 3 )); and (b) these two pairs were assigned by the seed model to the same relation. note that cases that satisfy condition (b) have, with high probability, at least one incorrect model prediction. on the other hand, given a strong seed model, there is also a high probability that one of the predictions is correct. for our seed-model we use a sota relation classification model which is based on fine-tuned span-bert @xcite , which we run over a large corpus of english wikipedia sentences. 5 out of the sentences that passed stage (2), we randomly sampled 100 sentences for each of the 30 relation predicted by the model. all instance entity pairs were manually labeled by two of the authors of this work, while closely adhering to the tacred annotation guideline. for each instance, the annotators provide a binary decision: does the entity pair in question adhere to relation r (as predicted by the model) or not. 6 the resulting challenge set (cre) has 3,000 distinct sentences, and 10,844 classification instances. the dataset is arranged into 30 groups of 100 sentences, where each sentence is binary labeled for a given relation. example sentences from the challenge-set are given in figure 1 . in 57% of the sentences, there are at least two classification instances with conflicting labels, indicating the use of the event+type heuristic. on average there are 3.7 candidate entity-pairs per sentence. in 89.2% of the sentences in the set, the entity-pairs share an argument. further details are available in the supplementary material. ## evaluating re models by construction, the cre dataset includes many instances that fail the seed rc model, which is based on fine-tuned spanbert @xcite . to verify that the behavior is consistent across models, we evaluate also rc models fine-tuned over other state-of-the-art lms: bert @xcite , roberta @xcite and @xcite . when evaluated on tacred test set ( the tacred-trained models often classify 7 cre is binary labeled and relatively balanced between positive and negative examples, making accuracy a valid and natural metric. we chose to report acc+ and accinstead of the popular precision and recall because precision and recall emphasize the positive class, and do not tell the full story of the negative class (indeed, prec/rec do not involve the truenegative case), which is of interest to us. using the acc +/- metric allows us to focus on both the positive and negative classes. ## qa models perform better the cre dataset results indicate that re-trained models systematically fail to link the provided relation arguments to the relation mention. we demonstrate that qa-trained models perform better in this respect. the qa models differ in both in their training data (squad 2.0, @xcite ) and in their training objective (span-prediction, rather than classification). inspired by @xcite we reduce rc instances into qa instances. we follow the reduction from @xcite between qa and binary relation classification which works by forming two questions for each relation instance, one for each argument. for example, for the relation instance pair (mark, fb, founded) we ask "who founded fb?" and "what did mark found?". 8 if the qa model answers either one of the questions with the correct span, we return 1 (relation holds), otherwise we return 0 (relation does not hold). we use three pre-trained sota models, finetuned on squad 2.0: qa-spanbert, qa-bert, and qa-albert @xcite . results. while the scores on the tacred test set are unsurprisingly substantially worse (with f 1 of 59.1%, 52.0% and 61.4%) than the tacred trained models, they also perform better on the cre dataset ( qa-trained models pay more attention to the relation between an event and its arguments than rc-trained models. ## augmenting the training data with challenge set examples we test the extent by which we can "inoculate" @xcite the relation extraction models by enhancing their training data with some examples from the challenge-set. we re-train each model on the tacred dataset, which we augment with half of the challenge-set (5504 examples, 8% of the size of the original tacred training set). the other half of the challenge-set is used for evaluation. results. we begin by evaluating the inoculated models on the original tacred evaluation set, establishing that this result in roughly the same scores, with small increases for most models (rc-spanbert: 71.0 f 1 (original: 70.8), rc-bert: 69.9 f 1 (original:67.5), rc-knowbert: 72.1 f 1 (original:71.5), rc-roberta: 70.8 f 1 (original:71.25)). when evaluating on the cre dataset examples, we see a large increase in performance for the inoculated dataset, as can be seen in ## discussion and conclusion we created a challenge dataset demonstrating the tendency of tacred-trained models to classify using an event+type heuristic that fails to connect the relation and its arguments. qa-trained models are less susceptible to this behavior. continuing @xcite , we conclude that challenge sets are an effective tool of benchmarking against shallow heuristics, not only of models and systems, but also of data collection methodologies. we suggest the following recommendation for future re data collection: evaluation sets should be exhaustive, and contain all relevant entity pairs. ideally, the same should apply also to training sets. if impractical, the data should at least attempt to exhaustively annotate confusion-sets: if a certain entity-pair is annotated in a sentence, all other pairs of the same entity-types in the sentence should also be annotated.