Item Response Theory for Natural Language Processing (2024)

Abstract: This tutorial will introduce the NLP community to Item Response Theory (IRT; Baker 2001). IRT is a method from the field of psychometrics for model and dataset assessment. IRT has been used for decades to build test sets for human subjects and estimate latent characteristics of dataset examples. Recently, there has been an uptick in work applying IRT to tasks in NLP. It is our goal to introduce the wider NLP community to IRT and show its benefits for a number of NLP tasks. From this tutorial, we hope to encourage wider adoption of IRT among NLP researchers.

## description this tutorial will introduce the nlp community to item response theory @xcite . irt is a method from the field of psychometrics for model and dataset assessment. irt has been used for decades to build test sets for human subjects and estimate latent characteristics of dataset examples. recently, there has been an uptick in work applying irt to tasks in nlp. it is our goal to introduce the wider nlp community to irt and show its benefits for a number of nlp tasks. from this tutorial, we hope to encourage wider adoption of irt among nlp researchers. as nlp models improve in performance and increase in complexity, new methods for evaluation are needed to appropriately evaluate performance improvements. in addition, data quality continues to be important. models exploitation of annotation artifacts, annotation errors, and a misalignment between models and dataset difficulty can hinder an appropriate assessment of model performance. as models reach and exceed human performance on certain tasks, it gets more difficult to distinguish between improvements and innovations and changes in scores due to chance. in this three-hour, introductory tutorial, we will review the current state of evaluation in nlp, then introduce irt as a tool for nlp researchers to use when evaluating their data and models. we will also introduce and demonstrate the py-irt python package for irt modelfitting to help encourage adoption and facilitate irt use. we believe that this should be a tutorial instead of a specialized workshop since the tutorial will aid in exposing a larger nlp audience to irt. while this methodology has been applied successfully to nlp applications, further community exposure specifically for graduate students may provide a new methodological perspective. we aim to make the tutorial interactive with hands-on jupyter note-books which will give concrete simple examples. tutorial materials are available online. foot_0 2 target audience/prerequisites the tutorial content will be self-contained so that a broad target audience of *cl conference attendees (researchers, phd students, industry professionals, etc.) can take away information on incorporating irt in their workflow. in terms of prerequisites, we expect the audience to have basic knowledge of probability and statistics. we also expect audience members to have experience with python is useful for py-irt. ## diversity considerations the presenters represent a mix of industry and academic researchers. we also span both europe and the us. the methods described can be applied to a variety of nlp tasks and languages. the tutorial content will be posted online for wide distribution beyond those able to attend the conference. ## ethics statement irt methods can provide fine-grained information about dataset examples and models. with regard to datasets, irt can potentially surface discrepancies in how groups of examples are handled by nlp models. for example, irt analyses may show that examples collected from a certain demographic group are systematically more difficult than those examples collected from another demographic group. ## pedagogy we hope that this tutorial can serve as a comprehensive introduction to irt for an nlp audience and that the content can be reused by others who are not able to attend. to that end, the tutorial will include a combination of presentation slides, demos via jupyter notebooks, and interactive sessions in jupyter notebooks. all content for the tutorial will be hosted online and made publicly available for future use and dissemination. jose hernandez-orallo is professor at the universitat politècnica de valència and senior research fellow at the leverhulme centre for the future of intelligence, university of cambridge, uk. his academic and research activities have spanned several areas of ai, machine learning, data science and intelligence measurement, with a focus on a more insightful analysis of the capabilities, generality, progress, impact and risks of ai. he has published five books and more than two hundred journal articles and conference papers on these topics. his research in the area of machine intelligence evaluation has been covered by several popular outlets, such as the economist, new scientist and nature. for a couple of decades, he has vindicated a more integrated view of the evaluation of natural and artificial intelligence, a position represented by his book "the measure of all minds" (cambridge university press, 2017, prose award 2018) and by multiple papers and events, using irt, extensions and techniques from some other disciplines to evaluate general-purpose ai such as llms. he is a member of aaai, claire and ellis, and a eurai fellow. website: https://josephorallo.webs.upv.es/ ## estimate audience size we expect between 50 to 150 attendees. this is based on previous experience at *cl tutorials as well as interest from others to learn about irt methods.