The INLG 2024 Tutorial on Human Evaluation of NLP System Quality: Background, Overall Aims, and Summaries of Taught Units (2024)

Abstract: Following numerous calls in the literature for improved practices and standardisation in human evaluation in Natural Language Processing over the past ten years, we held a tutorial on the topic at the 2024 INLG Conference. The tutorial addressed the structure, development, design, implementation, execution and analysis of human evaluations of NLP system quality. Hands-on practical sessions were run, designed to facilitate assimilation of the material presented. Slides, lecture recordings, code and data have been made available on GitHub (https://github.com/Human-Evaluation-Tutorial/INLG-2024-Tutorial). In this paper, we provide summaries of the content of the eight units of the tutorial, alongside its research context and aims.

## research context and aims human evaluation is widely considered the most reliable form of evaluation in natural language processing (nlp), but recent research has thrown up a number of concerning issues, including in the design @xcite and execution @xcite of human evaluation experiments, but also obstacles in adopting good practices @xcite . standardisation and comparability across different experiments is low, as is reproducibility in the sense that repeat runs of the same evaluation often do not support the same main conclusions, quite apart from not producing similar scores. the situation is likely to be in part due to how human evaluation is viewed in nlp: not as something that needs to be studied and learnt before venturing into conducting an evaluation experiment, but something that anyone can throw together without prior knowledge by pulling in a couple of students from the lab next door. our aim with this eight-unit tutorial is primarily to inform participants about the range of options available and choices that need to be made when creating human evaluation experiments, and what the implications of different decisions are. moreover, we present best practice principles and practical tools that help researchers design scientifically rigorous, informative and reliable experiments. the tutorial is structured into seven units each consisting of a lecture and (in the case of units 3, 4, 5, and 6) a brief hands-on exercise, followed by an extended practical session (unit 8), where participants create evaluation experiments and analyse results from them, using tools and other resources provided as part of the tutorial. we aim to address all aspects of human evaluation of system outputs in a research setting, equipping participants with the knowledge, tools, resources and hands-on experience needed to design and execute rigorous and reliable human evaluation experiments. publicly shared materials and online resources will continue to support participants in developing and conducting experiments after the tutorial. foot_0 2 tutorial unit summaries unit 1: introduction in unit 1 our aims are (i) to give a first idea of what human evaluation means in nlp; (ii) to summarise the current state of human evaluation in nlp; and (iii) to survey some of the challenges and issues that have been identified, and how current research is beginning to address them. unit 1 lays the groundwork for the tutorial, starting with an example from image caption generation, shown in figure 1 . among the possible captions for the labelled image are: different evaluation methods can yield different estimates of system quality and system rankings. in selecting evaluation methods, aspects to take into account include the application context (in a social media context, the last two captions above may be most suitable), and user-specific characteristics (deciding between the last two captions above depends on the user's taste in breakfast). the field of nlp has a 40+ year history of conducting human evaluation experiments to determine system quality, but very few established shared standards and methods for human evaluation. as early as the 1980s, spärck jones (1981) advocated the systematic testing of variations of data sets and systems in "a uniform framework for system characterisation and evaluation." for human evaluation at least, we are still far from such a framework, and this is likely a substantial contributing factor to a range of issues and challenges in human evaluation of nlp systems that have been identified recently: the tutorial aims to contribute to addressing these issues through providing structured, step-by-step information and guidance on how to put together scientifically rigorous experiments that produce reliable answers to questions like the above (which of the captions are good? better? more correct? on what grounds?). ## conclusion in this paper, we have provided abstract-style summaries of each of the units of the inlg'24 tutorial on human evaluation of nlp system quality, also incorporating some of the core content from each of the tutorial's eight taught units. the tutorial resources we provide via the tutorial's github page (footnote 1) of course provide more information and go into more detail on all of the aspects of human evaluation of nlp systems mentioned above, and this paper should not be considered complete or comprehensive in this respect. we will set up a feedback mechanism for participants and offline users of our content, via the tutorial github, to help us improve content and resources for future editions of the tutorial.