What’s wrong with your model? A Quantitative Analysis of Relation Classification (2024)

Abstract: With the aim of improving the state-of-the-art (SOTA) on a target task, a standard strategy in Natural Language Processing (NLP) research is to design a new model, or modify the existing SOTA, and then benchmark its performance on the target task. We argue in favor of enriching this chain of actions by a preliminary error-guided analysis: First, explore weaknesses by analyzing the hard cases where the existing model fails, and then target the improvement based on those. Interpretable evaluation has received little attention for structured prediction tasks. Therefore we propose the first in-depth analysis suite for Relation Classification (RC), and show its effectiveness through a case study. We propose a set of potentially influential attributes to focus on (e.g., entity distance, sentence length). Then, we bucket our datasets based on these attributes, and weight the importance of them through correlations. This allows us to identify highly challenging scenarios for the RC model. By exploiting the findings of our analysis, with a carefully targeted adjustment to our architecture, we effectively improve the performance over the baseline by >3 Micro-F1.

## introduction a major trend in nlp research aims at designing more sophisticated setups and model architectures in order to improve the state-of-the-art (sota) on a target task. the improvements are usually based on intuitions that target limitations of the previous sota on the task. the most common procedure follows the steps of (1) intuition, (2) modeling, (3) experiments, (4) results, and (5) analysis of the results. the latter is occasionally enriched with ablation or case studies with the main aim of proving the validity of the initial intuition and the effectiveness of the proposed methodology. we claim that conducting a preliminary in-depth analysis can help find good intuitions, and therefore guide better modeling and reducing the probability of nega-tive experiments, usually not reported in the paper. following previous error-guided analysis @xcite @xcite , we argue in favor of changing the standard chain of actions listed above: first perform an exhaustive quantitative analysis of the previous sota to identify failure cases and challenging scenarios, and then effectively target the baseline improvement in order to tackle those. we introduce an in-depth performance analysis suite in the context of relation classification (rc). within the field of information extraction (ie), which broadly aims at extracting structured knowledge from unstructured text, the goal of rc aims at classifying the semantic relation between two named entities. we pick this task because, despite its popularity, the task is far from being solved or reaching high performance, especially when considering realistic challenging setups-e.g. cross-domain @xcite , or document-level @xcite . we inspect the research approach of some of the most cited papers in the field from recent years, on top of which current sota are based: baldini @xcite introducing the widely adopted entity markers, @xcite introducing the typed entity markers and proposing a pipeline approach for end-to-end relation extraction (re), and @xcite at the time of writing holding the sota on three of the most established datasets in the field. we also inspect the research approach of papers published in the last year at major nlp conferences (acl, naacl, emnlp, aacl, eacl) that propose new sota models for rc, or for the related tasks of end-to-end re and few-shot rc @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . we find that that the common procedure consists of the five steps earlier mentioned. specifically, we found that in most cases, the intuition (step 1) that is used as a starting point and as a motivation for the model improvement is based on generic observations of the model architecture, instead of on a quantitative analysis which could lead to more effective targeted improvements. in this work, we propose a systematic quantitative analysis of a sota rc model to detect sets of challenging instances sharing common characteristics (e.g., entity distance). the goal is to identify hard-to-handle setups for the sota architecture. importantly, our approach is easily reproducible in future setups with different models, and/or on different datasets. the relevance of performing an in-depth analysis is supported by a demonstration of how the acquired information can help to effectively address the weaknesses of the baseline and design a new sota. our contributions are: foot_0 the first target the comparison of a set of ner systems. the latter, instead, perform a deep evaluation of cws systems proving that despite the excellent performance achieved on some datasets, there is no perfect system for cws. this concept has also been applied by @xcite for interpreting the results over a set of sequence tagging setups (ner, cws, pos, chunking). within the field of information extraction, previous work explored error-driven analysis for the automatic categorization of model prediction errors @xcite . ## attribute guided analysis we propose a systematic quantitative analysis of the performance of the crossre baseline model's performance across the 36 in-and cross-domain setups derived from training and testing the model on the six domains included in crossre. the analysis is performed over the development sets of the dataset. inspired by the work of @xcite on named entity recognition, we introduce the first evaluation suite for rc, opening the way to other similar structured prediction tasks. the analysis evaluates the performance of the model over 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 instances grouped by common values of potentially influential attributes (e.g., entity distance, sentence length). in what follows, we will describe the attributes considered and the bucketing strategy employed for splitting the evaluation instances based on the attribute values. last, we go through the results of our correlation analysis. ## application: model improvement as mentioned in the introduction, our final aim is to guide better modeling by targeting quantitatively measured weaknesses of the model. here we present a case study which exploits the findings of our proposed analysis. from the overall results in within this case study, we aim to focus on improving the model architecture. therefore, here we target the entity distance attribute, which holds the second highest absolute correlation (0.4), for improving the model performance. ## conclusion we present a tool for systematic quantitative analysis of the performance of rc models, and conduct the first in-depth analysis of an rc system, across 36 in-and cross-domain setups. we identify potentially influential attributes, and correlate their value with model performance. our findings highlight the influence of data scarcity of relation types over the model performance. the second most correlated attribute is the distance between the two entities: the further away, the more challenging it is to classify the semantic relation between them. last, we provide a case study exploiting the findings of the analysis for improving the baseline architecture with a simple yet effective method. we target the entity distance weakness, and by repeating the entities closer to each other at the end of the sentence we achieve a new sota on crossre, with an average @xmath0. we provide code for reproducing the proposed analysis on other rc setups (or related tasks, e.g., end-to-end re). and we also release the code of the new sota architecture. our aim is to encourage preliminary quantitative analysis of models prior to designing new architectures. future work includes expanding the set of attributes proposed in this work for rc in order to comprise other tasks, with different challenges. a solved task? in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 5676-5686, online. association for computational linguistics. lisheng fu, thien huu nguyen, bonan min, and ralph grishman. 2017. domain adaptation for relation extraction with domain adversarial neural network. in proceedings of the eighth international joint conference on natural language processing (volume 2: short papers), pages 425-429, taipei, taiwan. asian federation of natural language processing. tianyu gao, xu han, hao zhu, zhiyuan liu, peng li, maosong sun, and jie zhou. 2019. fewrel 2.0: towards more challenging few-shot relation classification. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6250-6255, hong kong, china. association for computational linguistics. claire gardent, anastasia shimorina, shashi narayan, and laura perez-beltrachini. 2017. creating training corpora for nlg micro-planners. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 179-188, vancouver, canada. association for computational linguistics. jia guo, stanley kok, and lidong bing. 2023. towards integration of discriminability and robustness for document-level relation extraction. in proceedings of the 17th conference of the european chapter of the association for computational linguistics, pages 2606-2617, dubrovnik, croatia. association for computational linguistics. qiushi guo, xin wang, and dehong gao. 2022. dependency position encoding for relation extraction. in findings of the association for computational linguistics: naacl 2022, pages 1601-1606, seattle, united states. association for computational linguistics. xu han, hao zhu, pengfei yu, ziyun wang, yuan yao, zhiyuan liu, and maosong sun. 2018. fewrel: a large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4803-4809, brussels, belgium. association for computational linguistics. arzoo katiyar and claire cardie. 2016. investigating lstms for joint extraction of opinion entities and relations. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 919-929, berlin, germany. association for computational linguistics. arzoo katiyar and claire cardie. 2017. going out on a limb: joint extraction of entity mentions and relations without dependency trees. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 917-928, vancouver, canada. association for computational linguistics. juhyuk lee, min-joong lee, june yong yang, and eunho yang. 2022. does it really generalize well on unseen data? systematic evaluation of relational triple extraction methods. in proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 3849-3858, seattle, united states. association for computational linguistics. pengfei liu, jinlan fu, yang xiao, weizhe yuan, shuaichen chang, junqi dai, yixin liu, zihuiwen ye, and graham neubig. 2021a. explainaboard: an explainable leaderboard for nlp. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing: system demonstrations, pages 280-289, online. association for computational linguistics. yang liu, jinpeng hu, xiang wan, and tsung-hui chang. 2022. a simple yet effective relation information guided approach for few-shot relation extraction. in findings of the association for computational linguistics: acl 2022, pages 757-763, dublin, ireland. association for computational linguistics. zihan liu, yan xu, tiezheng yu, wenliang dai, ziwei ji, samuel cahyawijaya, andrea madotto, and pascale fung. 2021b. crossner: evaluating crossdomain named entity recognition. proceedings of the aaai conference on artificial intelligence, 35(15):13452-13460. makoto miwa and mohit bansal. 2016. end-to-end relation extraction using lstms on sequences and tree structures. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1105-1116, berlin, germany. association for computational linguistics. nicholas popovic and michael färber. 2022. few-shot document-level relation extraction. in proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5733-5746, seattle, united states. association for computational linguistics. amir pouran ben veyseh, thien nguyen, and dejing dou. 2019. improving cross-domain performance for relation extraction via dependency prediction and information flow control. in proceedings of the twenty-eighth international joint conference on artificial intelligence, ijcai-19, pages 5153-5159. international joint conferences on artificial intelligence organization.