Putting Humans in the Natural Language Processing Loop: A Survey (2021)

Abstract: How can we design Natural Language Processing (NLP) systems that learn from human feedback? There is a growing research body of Human-in-the-loop (HITL) NLP frameworks that continuously integrate human feedback to improve the model itself. HITL NLP research is nascent but multifarious—solving various NLP problems, collecting diverse feedback from different people, and applying different methods to learn from human feedback. We present a survey of HITL NLP work from both Machine Learning (ML) and Human-computer Interaction (HCI) communities that highlights its short yet inspiring history, and thoroughly summarize recent frameworks focusing on their tasks, goals, human interactions, and feedback learning methods. Finally, we discuss future studies for integrating human feedback in the NLP development loop.

## introduction traditionally, natural language processing (nlp) models are trained, fine-tuned, and tested on existing dataset by machine learning experts, and then deployed to solve real-life problems of their users. model users can often give invaluable feedback that reveals design details overlooked by model developers, and provide data instances that are not represented in the training dataset @xcite . however, the traditional linear nlp development pipeline is not designed to take advantage of human feedback. advancing on conventional workflow, there is a growing research body of human-in-theloop (hitl) nlp frameworks, or sometimes called mixed-initiative nlp, where model developers continuously integrates human feedback into different steps of the model deployment workflow (figure 1 ). this continuous feedback loop cultivates a humanai partnership that enhance model performance and build users' trust in the nlp system. just like traditional nlp frameworks, there is a high-dimensional design space for hitl nlp systems. for example, human feedback can come from end users @xcite or crowd workers @xcite , and human can intervene models during training @xcite or deployment @xcite . this is the first survey on the hitl nlp topic. we make two main contributions: (1) we summarize recent studies of hitl nlp and position each work with respect to its task, goal, human interaction, and feedback learning method ( ## human-in-the-loop nlp tasks in this section, we categorize surveyed hitl paradigms based on their corresponding tasks. ## human-machine interaction this section discusses the mediums that users use to interact with hitl systems and different types of feedback that the system collect. ## how to use user feedback this section summarizes how existing hitl nlp systems utilize different types of feedback. ## conclusion and future directions in this paper, we summarize recent literature on hitl nlp from both nlp and hci communities, and position each work with respect to its task, goal, human interaction, and feedback learning method. the field of hitl nlp is still relatively nascent and we see many different design choices. we find improving model performance is the most popular goal among surveyed nlp hitl frameworks. however, researchers have found hitl method also enhances nlp model interpretability @xcite and usability @xcite . we encourage future nlp researchers to explore hitl as a mean to better understand their models and improve the experience of end users. one way is to design systems that take feedback from model engineers and end users beyond crowd workers. most of the hitl nlp systems are designed by nlp researchers. as human feedback is the core for hitl design, we believe that this field will be greatly benefited from a deeper involvement of the hci community. for example, with a poorly designed human-machine interface, the collected human feedback are more likely to be inconsistent, incorrect, or even misleading. therefore, better interface design and rigorous user study to evaluate interfaces can greatly enhance the quality of feedback collection, which in turn improve the downstream task performance. to shed light on hitl nlp research from a hci perspective, @xcite explore the effect of adding model interpretation cues in the hitl interface on the quality of collected feedback; @xcite investigate the impacts of question framing imposed on humans; similarly, @xcite study how to ask good questions to which humans are more likely to give helpful feedback. in particular, we recommend future researchers to (1) consider integrating interactive visualization techniques into human-machine interfaces;