User or Labor: An Interaction Framework for Human-Machine Relationships in NLP (2022)

Abstract: The bridging research between Human-Computer Interaction and Natural Language Processing is developing quickly these years. However, there is still a lack of formative guidelines to understand the human-machine interaction in the NLP loop. When researchers crossing the two fields talk about humans, they may imply a user or labor. Regarding a human as a user, the human is in control, and the machine is used as a tool to achieve the human’s goals. Considering a human as a laborer, the machine is in control, and the human is used as a resource to achieve the machine’s goals. Through a systematic literature review and thematic analysis, we present an interaction framework for understanding human-machine relationships in NLP. In the framework, we propose four types of human-machine interactions: Human-Teacher and Machine-Learner, Machine-Leading, Human-Leading, and Human-Machine Collaborators. Our analysis shows that the type of interaction is not fixed but can change across tasks as the relationship between the human and the machine develops. We also discuss the implications of this framework for the future of NLP and human-machine relationships.

## introduction research at the intersection of natural language processing (nlp) and human-computer interaction (hci) is developing rapidly. humans and machines are now both engaged in each step of the end-to-end nlp pipeline @xcite . nlp systems are trained on data created and annotated by humans, such as news articles @xcite , wikipedia pages @xcite , product reviews @xcite , and social media posts @xcite . yet, humans are also empowered by nlp systems, such as writing assistants @xcite , collaborative text revision @xcite , and translators @xcite . nonetheless, humans and machines play distinct roles in these scenarios. from the hci perspective, researchers usually consider humans as the users of certain technology within the interaction, while many nlp researchers emphasize the labor responsibility of humans to improve the performance of nlp models in tasks. humans and machines naturally have different strengths, such as trustworthiness, automation, and assessment ability @xcite . humans stand out for trustworthiness, while machines are known for automation. both humans and machines have the competence to evaluate each other, given their own specialties. while more and more interdisciplinary research and systems are being built to bridge the hci and nlp fields, we still lack a normative understanding of how human and machine interaction works within the nlp context. more importantly, does the interaction work well? to fill this gap, we address two main research questions: rq1: how does human-machine interaction happen in nlp? rq2: how do humans and machines interact with each other in nlp? we address our research questions by conducting a systematic literature review on the interactive nlp research from leading hci and nlp venues. our goal was to define a generalizable human-machine interaction framework in nlp to explain current implementation, guide the design of human-machine interaction, and inspire future research in interactive nlp systems. based on our synthesis, we defined three properties of interaction: continuity, variety of interaction options, and medium (rq1). we also conceptualized four re-lationships summarizing the roles of human and machine interaction patterns in different scenarios: human-teacher and machine-learner, machine-leading, human-leading, and human-machine collaborators (rq2). we used these properties and conceptualizations to contribute a theoretical interaction framework for human-machine relationships in nlp. future research from hci and nlp can use this framework to design and evaluate humanmachine interaction for human and machine needs within their role and responsibility. ## related work traditionally, nlp models are trained, fine-tuned, and tested on existing datasets before being deployed to solve real-world problems. while hci research usually involves users designing, developing, implementing, and evaluating systems. to encourage collaboration between hci and nlp, combining both approaches is required for successful 'hci+nlp' applications @xcite . further, @xcite proposed five methods for designing and evaluating hci+nlp systems: user-centered nlp, cocreating nlp, crowdsourcing, and user models. the human-machine interaction in nlp focuses on how people interact with machines and how nlp systems can be designed to support humans. in recent years, many researchers and practitioners have made significant advances in interactive nlp systems, such as text classification @xcite ), text summarization @xcite , semantic parsing and entity linking @xcite @xcite ), dialogue systems @xcite , topic modeling @xcite . to humans what the model requires, provide userfriendly interfaces for collecting feedback, and effectively learn from it. for example, humans can provide various types of feedback, such as training data providers, annotations, and evaluators of the system's output to improve the model's performance, interpretability, or usability. also, it is vital to understand the constraints of collecting and interpreting human inputs cor-rectly. many design considerations influence the efficiency and effectiveness of interactive learning from human feedback @xcite . for example, noise caused by human error (such as when a human teacher fails to provide the conventional ground truth) could be challenging in designing human-machine interaction systems. in addition, data collected from humans may be poor quality @xcite . social bias @xcite can also be introduced automatically during data collection @xcite and language model training, which emerging the need of developing fair and responsible models @xcite ; @xcite . therefore, it is critical to appropriately interpret collected human data and analyze the effects of various interaction types on learning outcomes @xcite . ## survey and analysis methods we conducted a systematic literature review to understand the types of interactions humans and machines have within the nlp and hci context. to select the targeted papers, we searched from acl anthology (nlp database) and acm digital library (hci database) for articles that have been published over the last two years and included papers with keywords such as 'human-in-the-loop,' 'interactive,' 'collaborative,' 'active learning,' 'hci + nlp,' 'human-machine,' and 'human-ai.' we also searched for workshops held in acl or acm conferences, such as hci+nlp in eacl 2021 @xcite . further, we conducted a backward reference search on the articles we found to identify any additional missing articles. this resulted in a total of 73 papers at the intersection of nlp and hci. we narrowed down our search to only include articles that made algorithmic contributions, system contributions, or empirical contributions. we excluded papers that only contributed opinions, theories, surveys, or datasets. this resulted in a total of 54 papers. next, we included only papers that had simultaneous interactions between humans and machines. we discarded 21 articles that had no interaction, or the interaction between the human and machine was asynchronous. this resulted in a final set of 33 articles included in our analysis. further, we conducted a thematic analysis @xcite of our dataset. we began by reading through each article and taking notes on observations and insights regarding the inter ## discussion based on our findings, we developed an interaction framework for the four types of human-machine relationships in one interaction cycle (see figure 1 ). the "human-in-the-loop" concept has become popular in nlp. but it is too broad to describe the nuance in similar but different human-machine interactions. our interaction framework depicts human and machine actions under different roles. it is important for nlp and hci practitioners to define human-machine interactions when designing interactive nlp systems because it builds up a clearer understanding of the human and machine's strengths and weaknesses. we can use the identi-fied strengths to complement the weaknesses. for example, in human-teacher and machine-learner interactions, the strengths of automation in annotation tasks can be leveraged to complete a task more quickly, while the human's strength in assessment can validate the machine's output. for human-leading or machine-leading, the leader may have limited knowledge and capacity that could be complemented with external expertise. next, we share how nlp and hci practitioners can use our conceptual framework in practice. while human-leading is similar to human-teacher, machine learner (since both are driven by humans who take initial and final action with machines' intermediate response), the framework captures the different statuses of machines and the corresponding actions of humans and machines. as a learner, the machine is a novice at the beginning and is launched by mimicking humans finishing their remaining tasks. however, the machines in the human-leading relationship have their own expertise to offer suggestions. accordingly, this also leads to different humans' final actions, i.e., humans as teachers will validate machines' learning results, while humans would choose machines' suggestions based on personal interests during humanleading interaction. additionally, the framework can be used as a guideline to check whether the interaction design is appropriate. for instance, the machine directly overwrites a human's control will be problematic when human-leading is required. for instance, someone is writing in colloquial english, using 'ain't' as a general preverbal negator @xcite . a machine trained in standard english will detect this as an error and would like to correct it, but it violates the writer's intention. figure 2 shows some trends of interactions: human-leading relationship usually happens with sequential interactions; machine-leading relationship uses more gui and has iterative interaction; human-machine collaborators mainly use nui; human-teacher, machine-learner relationship usually happens iteratively as well. ## limitations from the papers we reviewed, we didn't find any interaction that machines validate humans' actions. this might be because humans overall are more trustworthy than machines and may be influenced by the range of literature we reviewed. additionally, 'collaboration' can sometimes be interchangeable with 'interaction' based on their semantic meanings. but we strive to name our precise definitions of each relationship with clear and straightforward enough abstract names. in addition, we initially coded 'what are the outcomes of the research paper' to understand how interaction can influence the research outcome, such as efficiency, creativity, etc. but it didn't synthesize sufficiently with other codes. more importantly, the research outcome is usually the goal of those research papers so we couldn't derive the causal relationship from interaction design to the research outcome. however, this guides us in the future to study how we can manipulate desired research improvement by designing human-machine interaction. ## conclusion humans and machines interact with each other in a variety of ways. for example, humans may be involved in providing input to a machine learning algorithm, labeling data for training purposes, or evaluating the output of a machine learning system. additionally, humans may interact with machine learning systems through natural language interfaces, such as chatbots or virtual assistants. we contribute an interaction framework for humanmachine interactions through a systematic literature review and thematic analysis, which conceptualizes four human-machine relationships based on three different interaction properties, to help re-searchers and practitioners better understand and manage human-machine interactions in nlp.