Designing, Evaluating, and Learning from Humans Interacting with NLP Models (2023)

Abstract: The rapid advancement of natural language processing (NLP) research has led to various applications spanning a wide range of domains that require models to interact with humans – e.g., chatbots responding to human inquiries, machine translation systems assisting human translators, designers prompting Large Language Models for co-creation or prototyping AI-infused applications, etc. In these cases, humans interaction is key to the success of NLP applications; any potential misconceptions or differences might lead to error cascades at the subsequent stages. Such interaction involves a lot of design choices around models, e.g. the sensitivity of interfaces, the impact of design choice and evaluation questions, etc. This tutorial aims to provide a systematic and up-to-date overview of key considerations and effective approaches for studying human-NLP model interactions. Our tutorial will focus specifically on the scenario where end users – lay people and domain experts who have access to NLP models but are less familiar with NLP techniques – use or collaborate with deployed models. Throughout the tutorial, we will use five case studies (on classifier-assisted decision making, machine-aided translation, dialog systems, and prompting) to cover three major themes: (1) how to conduct human-in-the-loop usability evaluations to ensure that models are capable of interacting with humans; (2) how to design user interfaces (UIs) and interaction mechanisms that provide end users with easy access to NLP models; (3) how to learn and improve NLP models through the human interactions. We will use best practices from HCI to ground our discussion, and will highlight current challenges and future directions.

## introduction with the rapid advancement of natural language processing (nlp) research, there are numerous applications across a wide range of domains that require models to interact with humans -for example, chatbots responding to human inquiries @xcite , machine translation systems aiding human translators @xcite , designers prompting large language models for co-creation @xcite or prototyping aiinfused applications @xcite . in each of these cases, (timely) human interaction has been the key to the success; and any potential misconceptions or differences introduced to this interaction process might lead to error cascades at later stages @xcite . such interaction involves a lot of design choices around modelsthe sensitivity of interfaces @xcite and modalities @xcite , the impact of questions during human evaluation @xcite , or incorporating steer-ability in models @xcite . these choices are equally (if not more) important compared to the algorithms or datasets, but they are often undervalued and sometimes even considered a trivial part of the equation. in fact, while many of these topics have been extensively investigated in human-computer interaction (hci), they have only recently gained sufficient attention in nlp. nlp researchers entering the interaction world typically have to go through a steep learning curve before they can fully utilize the best practices from hci, resulting in some unintentional decisions that have adversely affected the reproducibility of earlier work @xcite . in this tutorial, we aim to provide a systematic and up-to-date overview of key considerations and effective approaches for studying human-nlp model interactions. interactions can take various forms depending on the stage of model develop-ment and the human involved; for example, nlp researchers and developers may interactively debug models during development, crowdworkers may participate in data annotation, etc.our tutorial will focus specifically on the scenario where end users -lay people and domain experts who try to use and benefit from nlp models -interact or collaborate with deployed models @xcite . throughout the tutorial, we will use four case studies (on model-assisted decision making, machine-aided translation, dialog systems, and prompting) to cover three major themes: (1) how to conduct usability evaluations to ensure that models are capable of interacting with humans; (2) how to design user interfaces (uis) and interaction mechanisms that allow end users to easily access nlp models; (3) how to learn from and improve nlp models through human interaction. we will ground our discussion in hci best practices, highlighting current challenges and future directions. ## tutorial outline this will be a three-hour tutorial devoted to the cutting-edge topic of designing, learning from, and evaluating human-ai interaction. each theme will take 35 mins, followed by 10 mins for q&a and 10 mins for a break. each part includes an overview of the corresponding topics, widely used methods, and a deep dive into a set of representative nlp and hci work. in the last 15 minutes, we will conclude our tutorial by highlighting challenges and research opportunities in the field. ## diversity considerations our chosen tutorial topic inherently touches on human user distribution. as mentioned before, we will discuss the importance of high coverage of user groups, and the impact of design biases on people from different demographics (e.g., ages, cultures, languages, and gender). as such, we believe our tutorial will be a strong advocate for diversity in the nlp model and interaction designs. besides diversity-related topics, our presenter team will also make our tutorial more accessible to different user groups. specifically, we will share our tutorial with a worldwide audience by promoting it on social media. we will also work with *cl d&i teams, and consult resources such as the big directory to diversify our audience participation. ## prerequisites & reading list the tutorial is targeted toward nlp researchers and practitioners working with humans. the prerequisite includes familiarity with basic knowledge of nlp and language systems. knowledge of system deployment is a plus. we will also provide a more paced introduction to some materials. the authors will also release an nlp+hci playbook as a resource for people interested in getting started in human-centered nlp research. here are a few papers that lay a foundation for this area: • putting humans in the natural language processing loop: a survey @xcite ; • all that's human is not gold: evaluating human evaluation of generated text @xcite ; • re-examining whether, why, and how human-ai interaction is uniquely difficult to design @xcite ); • does the whole exceed its parts? the effect of ai explanations on complementary team performance @xcite ; sherry tongshuang wu (she/her) is an assistant professor at the human-computer interaction institute, carnegie mellon university. her primary research investigates how humans (ai experts, lay users, domain experts) interact with (debug, audit, and collaborate) ai systems. sherry has organized two workshops at nlp and hci conferences: shared stories and lessons learned workshop at emnlp 2022 and trust and reliance in ai-human teams at chi 2022. she is currently developing a new course on human-centered nlp at cmu. diyi yang (she/her) is an assistant professor in the cs department at stanford university. her research focuses on learning with limited and noisy text data, user-centric language generation, and computational social science. diyi has organized four workshops at nlp conferences: widening nlp workshops at naacl 2018 and acl 2019, casual inference workshop at emnlp 2021, and nlg evaluation workshop at emnlp 2021. she also gave a tutorial at the acl 2022 on learning with limited data. she has taught courses on natural language processing at georgia tech since 2019 and is now developing a new course on human-centered nlp at stanford university. sebastin santy (he/him) is a second-year phd student at the paul g. allen school of cse, university of washington. he works on problems in the intersection of hci and nlp and specifically his research focuses on uncovering design biases in nlp systems. he previously worked on multilinguality and machine translation. ## ethics statement we do not anticipate any ethical issues related to the tutorial logistics, but we plan to cover ethical considerations in our content, especially when we discuss human-centered evaluation metrics like safety, and when we review the impact of different communication and initiation methods in interaction designs (e.g. leading to confirmation biases).