Let’s S top I ncorrect C omparisons in E nd-to-end R elation E xtraction! (2020)

Abstract: Despite efforts to distinguish three different evaluation setups (Bekoulis et al., 2018), numerous end-to-end Relation Extraction (RE) articles present unreliable performance comparison to previous work. In this paper, we first identify several patterns of invalid comparisons in published papers and describe them to avoid their propagation. We then propose a small empirical study to quantify the most common mistake’s impact and evaluate it leads to overestimating the final RE performance by around 5% on ACE05. We also seize this opportunity to study the unexplored ablations of two recent developments: the use of language model pretraining (specifically BERT) and span-level NER. This meta-analysis emphasizes the need for rigor in the report of both the evaluation setting and the dataset statistics. We finally call for unifying the evaluation setting in end-to-end RE.

## introduction named entity recognition (ner) foot_1 and relation extraction (re) are key information extraction tasks, for example at the heart of knowledge graph construction along with coreference resolution and entity linking. in the traditional pipeline approach, these tasks are treated with two models trained separately and applied sequentially @xcite . nevertheless, combining information from both submodules is beneficial @xcite and end-to-end re models tackling both tasks jointly have been proposed to better model their interdependency and overcome cascading errors @xcite . this end-to-end setting has recently received more attention in the wake of improved language models (lm). however, in this prolific and competitive domain, authors have used several evaluation settings to compare their performance. and despite the attempt to clearly identify three main setups @xcite , this multiplication of settings makes the apprehension of the literature difficult and confusing, but more importantly, it has led to erroneous comparisons and conclusions. in this paper, we first present a quick literature review of the recent advances in end-to-end re. our main contribution is then the identification of invalid comparison patterns in recent publications. we list them with the hope of stopping the propagation of erroneous results and presenting a curated list of published results. to further this contribution, we propose a small empirical study to quantify the impact of switching the two main metrics and estimate it can lead to a relative overestimation of around 5% in the end-to-end re results on ace05. as a second contribution, we take advantage of this quantitative study to perform the omitted ablations of two recent developments in the literature: lm pretraining and span-level ner. it confirms that recent empirical gains are mainly due to lm pretraining, while there is no evidence for quantitative gains from span-level ner. finally, we argue that the main cause for previously identified mistakes is the lack of reproducibility and, consequently, of previous work reproductions. we call for a more rigorous report of both evaluation settings and dataset statistics in general and particularly in end-to-end re. and we also suggest unifying our evaluation setting to reduce the chance of future mistakes and enable more meaningful cross-dataset analyses. ## a quick literature review in order to have a global view of recent evolutions, we present a quick literature review of end-to-end re models. we focus on supervised extraction of representations enc. ner re reference criterion code lm word char hand pos dep tag dec. dec. (giorgi et al., 2019) s * b -b mlp biaff. (eberts and ulges, 2020) sb b -s mlp pmaxpool (wadden et al., 2019) b b -s mlp biaff. (li et al., 2019) s b --mt qa mt qa (dixit and al-onaizan, 2019) s e s c l s mlp biaff. (luan et al., 2019) b e g ns l s mlp biaff. (nguyen and verspoor, 2019) sr g l l b mlp mhs-biaff. (sanh et al., 2019) -e g c l b crf mhs-lin. (luan et al., 2018) b e g ns s mlp biaff. (sun et al., 2018) s ≈ ns c l b mlp pcnn (bekoulis et al., 2018a,b) sbr s/w l l b crf mhs-lin. (zhang et al., 2017) s ≈ g c l b i-lstm i-lstm (li et al., 2017) s ≈ ns c l b mlp sp ltsm (katiyar and cardie, 2017) s w l b i-mlp i-pointer (zheng et al., 2017) s ns l b mlp pcnn (adel and schütze, 2017) r w -b cnn pcnn+crf (gupta et al., 2016) r t -b i-rnn i-rnn (miwa and bansal, 2016) s ns l b mlp sp lstm (miwa and sasaki, 2014) s -b i-svm i-svm (li and ji, 2014) sb -b i-perc. i-perc. ## datasets and metrics datasets although a variety of datasets have been used, we limit our report to the five we identified as the most frequently studied for brevity. following @xcite , end-to-end re has traditionally been explored on english news articles, which is reflected in the domain of its historical benchmarks, conll04 and the ace datasets. conll04 @xcite is annotated for four entity types and five relation types and specifically only contains sentences with at least one relation. the ace04 dataset @xcite defines seven coarse entity types and seven relation types. ace05 resumes this setting but merges two relation types leading to six of them. more recently, @xcite propose the ade dataset in the biomedical domain, which focuses on one relation, the adverse drug event between a drug and one of its adverse effects. in the scientific domain, @xcite introduce scierc composed of 500 scientific article abstracts annotated with six types of scientific entities, coreference clusters, and seven relations between them. ## identified issues in published results this variety of evaluation settings, visible in ## a small empirical study given these previous inconsistencies, we can legitimately wonder what is the impact of different evaluation settings on quantitative performance. however, it is also unrealistic to reimplement and test each and every paper in a same setting to establish a benchmark. instead, we propose a small empirical study to quantify the impact of using the boundaries setting instead of the strict setting on the two main benchmarks: conll04 and ace05. we discard the relaxed setting because it cannot evaluate true end-to-end re without striclty taking argument detection into account. it is also limited to conll04 and we have no example of misuse. we will consider a limited set of models representative of the main entity filtering approach. and we seize this opportunity to perform two ablations that correspond to meaningful recent proposals and are missing in related work. first, when looking at second, to our knowledge, no article compares the recent use of span-level ner instead of classical sequence tagging in end-to-end re. ## conclusion the multiplication of settings in the evaluation of end-to-end relation extraction makes the comparison to previous work difficult. indeed, in this confusion, numerous articles present unfair comparisons, often overestimating the performance of their proposed model. furthermore, this fragmentation of the community complicates the emergence of new proposals. our critical literature review epitomizes the need for more rigorous reports of evaluation settings, including detailed datasets statistics. and we call for a unified end-to-end re evaluation setting to prevent future mistakes and enable more meaningful cross-domain comparisons.