ILDAE : Instance-Level Difficulty Analysis of Evaluation Data (2022)

Abstract: Knowledge of difficulty level of questions helps a teacher in several ways, such as estimating students’ potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions. Can we extract such benefits of instance difficulty in Natural Language Processing? To this end, we conduct Instance-Level Difficulty Analysis of Evaluation data (ILDAE) in a large-scale setup of 23 datasets and demonstrate its five novel applications: 1) conducting efficient-yet-accurate evaluations with fewer instances saving computational cost and time, 2) improving quality of existing evaluation datasets by repairing erroneous and trivial instances, 3) selecting the best model based on application requirements, 4) analyzing dataset characteristics for guiding future data creation, 5) estimating Out-of-Domain performance reliably. Comprehensive experiments for these applications lead to several interesting results, such as evaluation using just 5% instances (selected via ILDAE) achieves as high as 0.93 Kendall correlation with evaluation using complete dataset and computing weighted accuracy using difficulty scores leads to 5.2% higher correlation with Out-of-Domain performance. We release the difficulty scores and hope our work will encourage research in this important yet understudied field of leveraging instance difficulty in evaluations.

## introduction transformer-based language models @xcite @xcite have improved state-of-the-art performance on numerous natural language processing benchmarks @xcite @xcite ; however, recent studies @xcite have raised questions regarding whether these models are uniformly better across all instances. this has drawn attention towards instancelevel analysis of evaluation data @xcite @xcite which was previously limited to training data @xcite @xcite . furthermore, it is intuitive that not all instances in a dataset are equally difficult. however, instance-level difficulty analysis of evaluation data (ildae) has remained underexplored in many different ways: what are the potential applications and broad impact associated with ildae? in this work, we address the above question by first computing difficulty scores of evaluation instances (section 2) and then demonstrating five novel applications of ildae (figure 1 ). 1. efficient evaluations: we propose an approach of conducting efficient-yet-accurate evaluations. our approach uses as little as 5% evaluation instances (selected via ildae) to achieve up to 0.93 kendall correlation with evaluations conducted using the complete dataset. thus, without considerably impacting the effectiveness of evaluations, our approach saves computational cost and time. ## other applications of ildae we now briefly discuss other ildae applications. ## conclusion we conducted instance-level difficulty analysis of evaluation data (ildae) in a large-scale setup of 23 datasets and presented its five novel applications. with these applications, we demonstrated ildae's impact in several important areas, such as conducting efficient evaluations with fewer instances, improving dataset quality, and estimating out-of-domain performance reliably. we release our computed difficulty scores and hope that our encourage research in this important yet understudied field of leveraging instance difficulty in evaluations.