Human-Centered Evaluation of Language Technologies (2024)

Abstract: Evaluation is a cornerstone topic in NLP. However, many criticisms have been raised about the community’s evaluation practices, including a lack of human-centered considerations about people’s needs for language technologies and their actual impact on people. This “evaluation crisis” is exacerbated by the recent development of large generative models with diverse and uncertain capabilities. This tutorial aims to inspire more human-centered evaluation in NLP by introducing perspectives and methodologies from human-computer interaction (HCI), a field concerned primarily with the design and evaluation of technologies. The tutorial will start with an overview of current NLP evaluation practices and their limitations, then introduce the “toolbox of evaluation methods” from HCI with varying considerations such as what to evaluate for, how generalizable the results are to the real-world contexts, and pragmatic costs to conduct the evaluation. The tutorial will also encourage reflection on how these HCI perspectives and methodologies can complement NLP evaluation through Q&A discussions and a hands-on exercise.

## tutorial description designing effective evaluation methods for natural language processing (nlp) has long been challenging due to the complex nature of language, openendedness of tasks, and multifaceted and contextdependent definitions of language quality. this challenge is exacerbated as "general" capability models (e.g., large language models) become more capable and prevalent. not only must they be evaluated across a diverse range of tasks and domains, which can be difficult to define and validate, but their wide range of potential capabilities, including those potentially unanticipated by model develop-ers @xcite , may also render evaluation results ungeneralizable to and unreliable in real-world contexts where the model is to be used. researchers have pointed out shortcomings of popular nlp benchmarks, metrics, and human evaluation methods (e.g., human ratings), such as their inability to capture nuanced meanings, their lack of validity, their perpetuation of biases and potential harm, and a lack of standardization and reproducibility @xcite @xcite . ultimately, nlp models are to be incorporated into real-world applications, interacted with by people, and can have a profound impact on people's lives. evaluation methods must take on a human-centered perspective that centers around people's needs, values, and interaction behaviors in order to produce results that can realistically reflect real-world performance and possible impacts. these kinds of human-centered considerations are at the forefront of evaluation practices in social science where the validity of measurements is a key focus, as well as in human-computer interaction (hci), a field primarily focusing on how to design technologies and evaluate the designs. in the past half-decade, hci researchers have developed a "toolbox of methods" as different "ways of knowing" (olson and kellogg, 2014) people's needs, usage, and interaction outcomes with technologies. this tutorial aims to provide an introduction to these hci perspectives and evaluation methods to inspire more human-centered evaluation methods in nlp, and to facilitate collaboration between the hci and nlp communities. this 3-hour tutorial will include 110 minutes of instructors' presentations followed by q&a and a hands-on exercise. the presentations will start with a brief overview of current evaluation practices in nlp, including automatic evaluation and human evaluation. in this part, we will review common goals and assumptions that are built into existing evaluation practices. we also aim to highlight concerns and limitations-e.g., lack of reliability, realism, and standardization-which may lead to an overall lack of validity in the evaluation outcomes. with these concerns and limitations of nlp evaluation in mind, we will introduce complementary perspectives in social sciences and hci. we will introduce measurement modeling-a framework that disentangles what is measured (i.e., theoretical, frequently unobservable constructs) from how it is measured (operationalizations) and offers a rich vocabulary via validity and reliability to assess measurements @xcite . we will further illustrate how these concepts can be applied to better assess nlp evaluation approaches (e.g., @xcite . we will then provide an overview of common hci evaluation methods, from human-subjects studies and surveys to analytical and simulated evaluations, and discuss the benefits and drawbacks of each. by comparing these different methods, we will particularly highlight the consideration of realism @xcite @xcite -designing evaluations in a way that the conclusion can be generalized to the real-world contexts where the technology will be used, and pragmatic costs to conduct the evaluation. our goal is to inspire nlp researchers to explore diverse evaluation methods as alternatives to benchmarks and automated metrics, and develop human-centered evaluation methods with downstream human needs and lower adoption barriers (for people who should be doing evaluation, such as model developers) in mind. to further ground the introduction to hci evaluation, we will present examples of hci works conducting evaluations for language technologies such as chatbots @xcite and writing support @xcite . lastly, the hands-on exercise will ask participants to work in groups to choose an evaluation method and design the details for a given use case. the exercise is designed to encourage participants to explore and compare different evaluation methods they learn from the tutorial, and facilitate further reflections and discussions. ## expected outcome we plan to make the tutorial presentation materials public and the videos accessible to a wide population. with participants' consent, we may also share notes from the q&a session and discussions in the hands-on exercise. expected audience size: we expect to have more than 100 in-person attendees, based on the audience size of a naacl 2022 tutorial on humancentered evaluation focusing on explanation @xcite , and the recent popularity of the topic of model evaluation. target audience and prerequisite background: as an introductory tutorial, our presentation will not assume any prior familiarity with hci evaluation methods or the hci literature more generally. we expect the audience to have some familiarity with common nlp tasks but not necessarily expert knowledge of nlp evaluation. ## diversity considerations instructors: the instructors consist of researchers across nlp, hci, and psychology at varying career stages, spanning both industry and academia, with equal gender balance. diversifying audience participation: the tutorial format is designed to encourage broad participation from researchers and practitioners across industry and academia; no prior familiarity with hci methods is expected, and the presentation materials will be made publicly available. ## presenter biographies su lin blodgett is a researcher at microsoft research montréal. her work has examined measurement and evaluation in nlp, and she has coorganized three editions of the hci+nlp workshop, a chi panel on responsible language technologies, and a facct tutorial on measurement and nlp. jackie chi kit cheung is an associate professor at mcgill university and at the mila quebec ai institute. his work has involved developing new evaluation methods and datasets for a range of nlp tasks including common sense reasoning, automatic summarization, and authorship attribution. q. vera liao is a principal researcher at microsoft research. she is an hci researcher by training and recently works on human-ai interaction, explainable ai, and responsible ai. she taught tutorials at naacl 2022, neurips 2022, chi 2023, chi 2020, as well as various seminars internationally. she is frequently involved in organizing events (e.g. panels, workshops) that connect the ai and hci communities. ziang xiao is an assistant professor in the department of computer science. his work lies in the intersection of human-computer interaction, natural language processing, and social psychology. ziang is on the organizing committee and an associate chair for multiple hci venues (chi, cscw, iui). he co-organized the 3rd hci+nlp workshop at naacl 2024. he co-organized the first workshop on human-centered evaluation and auditing of language models at chi 2024. ## ethics statement we hope that our tutorial will inspire humancentered evaluation practices that may help alleviate potential harm and ethical concerns brought about by language technologies. as many of the evaluation methods we will present involve human participants, we will also address ethical considerations emerging from their application, e.g., risks and best practices surrounding human-subjects recruitment and study design.