Methods for the Design and Evaluation of HCI + NLP Systems (2021)

Abstract: HCI and NLP traditionally focus on different evaluation methods. While HCI involves a small number of people directly and deeply, NLP traditionally relies on standardized benchmark evaluations that involve a larger number of people indirectly. We present five methodological proposals at the intersection of HCI and NLP and situate them in the context of ML-based NLP models. Our goal is to foster interdisciplinary collaboration and progress in both fields by emphasizing what the fields can learn from each other.

## introduction nlp is the subset of ai that is focused on the scientific study of linguistic phenomena (association for computational linguistics, 2021). humancomputer interaction (hci) is "the study and practice of the design, implementation, use, and evaluation of interactive computing systems" @xcite . grudin described hci and ai as two fields divided by a common focus @xcite : while both are concerned with intelligent behavior, the two fields have different priorities, methods, and assessment approaches. in 2009, grudin argued that while ai research traditionally focused on longterm projects running on expensive systems, hci is focused on short-term projects running on commodity hardware. for successful hci+nlp applications, a synthesis of both approaches is necessary. as a first step towards this goal, this article, informed by our sensibility as hci researchers, provides five concrete methods from hci to study the design, implementation, use, and evaluation of hci+nlp systems. one promising pathway for fostering interdisciplinary collaboration and progress in both fields is to ask what each field can learn from the methods of the other. on the one hand, while hci directly and deeply involves the end-users of a system, nlp involves people as providers of training data or as judges of the output of the system. on the other hand, nlp has a rich history of standardised evaluation metrics with freely available datasets and comparable benchmarks. hci methods that enable deep involvement are needed to better understand the perspective of people using nlp, or being affected by it, their experiences, as well as related challenges and benefits. as a synthesis of this user focus and the standardized benchmarks, hci+nlp systems could combine more standardized evaluation procedures and material (data, tasks, metrics) with user involvement. this could lead to better comparability and clearer measures of progress. this may also spur systematic work towards "grand challenges", that is, uniting hci researchers under a common goal @xcite . to facilitate a productive collaboration between hci+nlp, clearly defined tasks that attract a large number of researchers would be helpful. these tasks could be accompanied with data to train models, as a methodological approach from nlp, and methodological recommendations on how to evaluate these systems, as a methodological approach from hci. one task could e.g. define which questions should be posed to experiment participants. if the questions regarding the evaluation of an experiment are fixed, the results of different experiments could be more comparable. this would not only unite a variety of research results, but it could also increase the visibility of the researchers who participate. complementary, nlp could benefit from asking further questions about use cases and usage contexts, and from subsequently evaluating contributions in situ, including use by the intended target group (or indirectly affected groups) of nlp. method description 1. user-centered nlp user studies ensure that users understand the output and the explanations of the nlp system 2. co-creating nlp deep involvement from the start enables users to actively shape a system and the problem that the system is solving 3. experience sampling richer data collected by (active) users enables a deeper understanding of the context and the process in which certain data was created 4. crowdsourcing an evaluation at scale with humans-in-the-loop ensures high system performance and could prevent biased results or discrimination 5. user models simulating real users computationally can automate routine evaluation tasks to speed up the development ## methods for hci+nlp this section presents and discusses a set of concrete ideas and directions for developing evaluation methods at the intersection of hci and nlp. ## discussion figure 1 situates the different methods in the context of hci+nlp systems. the figure illustrates that two approaches are focused on the model side and three methods are focused on the user side. methods 1 and 2 are focused on the nlp system itself. the 1. user-centered nlp is at the heart of the model and focuses on users' understanding of the output and the explanations of the nlp system. while method 2 is also strongly related to the user, we put it on the system side to highlight that when 2. co-creating an nlp system, the goal is not just to evaluate the experience with an nlp system, but to enable users to actively shape the system. this does not only include what the system looks like but means involving users in the problem formulation stage and allowing them to shape what problem is being solved. considering the input that an nlp system is trained on, method 3. experience sampling provides a simpler way of collecting metadata and more actively involving people in the collection of the dataset. regarding the output of an nlp system, we showed the utility of 4. crowdsourcing the evaluation of nlp systems, which puts users into the loop to evaluate existing nlp systems at scale. the advantage of this is that a large number of users can be involved in the evaluation of the system. finally, method 5 proposes simulating real users through other ml-based systems. these 5. user models can act as proxies for real users and allow a fast, automated evaluation of nlp systems at scale. we hope that this work informs novel approaches on how to standardize tools for large-scale interactive evaluations that will generate comparable and actionable benchmarks. ## conclusion the five methods presented in figure 1 cover the whole spectrum of hci+nlp systems including the input, the nlp system, and the output of the system. though each method has merits on its own, for successful future hci+nlp applications, we believe that the whole will be greater than the sum of its parts. the design of future hcp+nlp applications should be centered around users (1) and involve them not only in the evaluation but also in the development and the problem formulation of an nlp system (2). rich-meta data (3) that shapes the input of such a system are equally important as a thorough investigation of the output of the system, both by humans-in-the-loop (4) and by approaches based on computational methods that automate certain key aspects of such systems (5). we hope that this overview of hci and nlp methods is a useful starting point to engage interdisciplinary collaborations and to foster an exchange of what hci and nlp have to offer each other methodologically. with this work, we hope to stimulate a discussion that brings hci and nlp together and that advances the methodologies for technical and human-centered system design and evaluation in both fields.