M achine learning is a powerful tool for transforming data into computational models that can drive user-facing applications. However, potential users of such applications, who are often domain experts for the application, have limited involvement in the process of developing them. The intricacies of applying machine-learning techniques to everyday problems have largely restricted their use to skilled practitioners. In the traditional applied machine-learning workflow, these practitioners collect data, select features to represent the data, preprocess and transform the data, choose a representation and learning algorithm to construct the model, tune parameters of the algorithm, and finally assess the quality of the resulting model. This assessment often leads to further iterations on many of the previous steps. Typically, any end-user involvement in this process is mediated by the practitioners and is limited to providing data, answering domain-related questions, or giving feedback about the learned model. This results in a design process with lengthy and asynchronous iterations and limits the end users' ability to affect the resulting models. Consider the following case study of machine-learning practitioners working with biochemists to develop a protein taxonomy by clustering low-level protein structures (Caruana et al. 2006) . The project lead recounted their experience in an invited talk at the Intelligent User Interfaces's 2013 Workshop on Interactive Machine Learning (Amershi et al. 2013) . First, the practitioners would create a clustering of the protein structures. Then, they would meet with the biochemists to discuss the results. The biochemists would critique the results (for example, "these two proteins should / should not be in the same cluster" or "this cluster is too small"), providing new constraints for the next iteration. Following each meeting, the practitioners would carefully adjust the clustering parameters to adhere to the given con- (Cohn, Caruana, and McCallum 2003; Caruana et al. 2006) . These algorithms were intended to give people the ability to rapidly iterate and inspect many alternative clusterings within a single sitting. Their later approach is an example of interactive machine learning, where learning cycles involve more rapid, focused, and incremental model updates than in the traditional machine-learning process. These properties enable everyday users to interactively explore the model space and drive the system toward an intended behavior, reducing the need for supervision by practitioners. Consequently, interactive machine learning can facilitate the democratization of applied machine learning, empowering end users to create machine-learning-based systems for their own needs and purposes. However, enabling effective end-user interaction with interactive machine learning introduces new challenges that require a better understanding of end-user capabilities, behaviors, and needs. This article promotes the empirical study of the users of interactive machine-learning systems as a method for addressing these challenges. Through a series of case studies, we illustrate the following propositions: Rapid, focused, and incremental learning cycles result in a tight coupling between the user and the system, where the two influence one another. As a result it is difficult to decouple their influence on the resulting model and study such systems in isolation. Explicitly studying user interaction can challenge assumptions of traditional learning systems about users and better inform the design of interactive learning systems. The ways in which end users interact with learning systems can be expanded to ways in which practitioners do (for example, tuning parameters or defining new constraints); however, novel interaction techniques should be carefully evaluated with potential end users. While the presented case studies paint a broad picture of recent research in user interaction with interactive machine learning, this article does not exhaustively survey the literature in this space. Rather, these case studies are selected to highlight the role and importance of the user within the interactive machine-learning process, serving as an introduction to the topic and a vehicle for considering this body of research altogether. We conclude this article with a discussion of the current state of the field, identifying opportunities and open challenges for future interactive machine-learning research. The applied machine-learning workflow often involves long and complex iterations. The process starts with data provided by domain experts or specifically collected for the target application. Machinelearning practitioners then work with domain experts to identify features to represent the data. Next, the practitioners experiment with different machinelearning algorithms, iteratively tuning parameters, tweaking features, and sometimes collecting more data to improve target performance metrics. Results are then further examined both by practitioners and domain experts to inform the subsequent iteration. At the end of this long cycle, the model is updated in several ways and can be drastically different from the previous iteration. Furthermore, this iterative exploration of the model space is primarily driven by the machinelearning practitioners, who rely on their understanding of machine-learning techniques to make informed model updates in each iteration. In contrast, model updates in interactive machine learning are more rapid (the model gets updated immediately in response to user input), focused (only a particular aspect of the model is updated), and incremental (the magnitude of the update is small; the model does not change drastically with a single update). This allows users to interactively examine the impact of their actions and adapt subsequent inputs to obtain desired behaviors. As a result of these rapid interaction cycles, even users with little or no machine-learning expertise can steer machine-learning behaviors through low-cost trial and error or focused experimentation with inputs and outputs. Perhaps the most familiar examples of interactive machine learning in real-world applications are recommender systems such as Amazon product recommendations, Netflix movie recommendations, and Pandora music recommendations. Users of recommender systems are often asked targeted questions about their preferences for individual items 2 (which they provide by liking or disliking them, for example). These preferences are then promptly incorporated in the underlying learning system for subsequent recommendations. If a recommender system begins recommending undesired items after incorporating new preferences, the user may attempt to redirect the system by correcting it or providing different preference information in the future. We next present two case studies that exemplify the interactive machine-learning process and demonstrate its potential as an end-user tool. Interactive Machine Learning for Image Segmentation Fails and Olsen (2003) were the first to introduce the Articles WINTER 2014 107 term interactive machine learning in the human-computer interaction community, characterizing it with rapid train-feedback-correct cycles, where users iteratively provide corrective feedback to a learner after viewing its output. They demonstrated this process with their Crayons system, which allowed users with no machine-learning background to train pixel classifiers by iteratively marking pixels as foreground or background through brushstrokes on an image. After each user interaction, the system responded with an updated image segmentation for further review and corrective input. Evaluations of Crayons through user studies revealed that the immediate output provided by the system allowed users to quickly view and correct misclassifications by adding new training data in the most problematic areas. As illustrated in figure 2, after an initial classification, the user provides Crayons with more data at the edges of the hand where the classifier failed. When asked what they were thinking while interacting with the system, most users stated that they were focused on seeing parts of the image that were classified incorrectly. Fails and Olsen's work on Crayons demonstrated that users modify their behavior based on a learner's outputs, which is an underlying premise for much of the following research on interactive machine learning. Another example of an interactive machine-learning system comes from the realm of music composition In machine learning, people iteratively supply information to a learning system and then observe and interpret the outputs of the system to inform subsequent iterations. In interactive machine learning, these iterations are more focused, frequent, and incremental than traditional machine learning. The tighter interaction between users and learning systems in interactive machine learning necessitates an increased focus on studying the user's involvement in the process. and performance. This domain is naturally interactive: musicians are accustomed to receiving immediate feedback when interacting with a musical instrument. Fiebrink, Cook, and Trueman (2011) developed the Wekinator, a machine-learning system for enabling people to interactively create novel gesture-based instruments, such as moving an arm in front of a web camera to produce different sounds based on the arm's position, speed, or rotation. In this system, a neural network receives paired gestures and sounds from the user as input and learns how to interpolate from unobserved gesture positions to a range of sounds. Users evaluate their instruments directly by gesturing and assessing the produced sounds. While observing students using Wekinator in an interdisciplinary music and computer science course, the authors found that as students trained their respective instruments, the interactive nature of the system also helped train the students. For example, the students learned how to recognize noise in their training samples and provide clearer examples to the learner. In some cases, students even adjusted their goals to match the observed capabilities of the learner. In a follow-up investigation with a professional cellist (Fiebrink, Cook, and Trueman 2011) , the cellist identified flaws in her playing technique while trying to train a gesture recognizer. The process revealed that the cellist's bowing articulation was not as precise as she had believed. By observing the outputs of the system in real time, Wekinator users were able to modify their behavior in ways that allowed them to create instruments to their satisfaction. These examples illustrate the rapid, focused, and incremental interaction cycles fundamental to interactive machine learning; it is these cycles that facilitate end-user involvement in the machine-learning process. These cycles also result in a tight coupling between the user and the system, making it impossible to study the system in isolation from the user. This necessitates an increased focus on studying how users can effectively influence the machine-learning system and how the learning system can appropriately influence the users. The following section examines how explicitly studying end users can challenge assumptions of traditional machine learning and better inform the development of interactive machine-learning systems. Many of the case studies to follow additionally consider less traditional types of input and output, moving beyond labeled examples and observations of learner predictions. Articles AI MAGAZINE Figure 2. Interactive Training of the Crayons System. The system takes pixels labeled as background/foreground as input (provided through brush strokes), and gives a fully segmented image as output (obtained through a classifier that labels each pixel as foreground/background). The user's input is focused on areas where the classifier is failing in previous iterations (Fails and Olsen 2003). Input DataSegmentation Iteration 1 Iteration 2 Iteration 3 The increased interaction between users and learning systems in interactive machine learning necessitates an increased understanding of how end-user involvement affects the learning process. In this section, we present case studies illustrating how such an understanding can ultimately lead to better-informed system designs. First, we present case studies demonstrating how people may violate assumptions made by traditional machine learners, resulting in unexpected outcomes and user frustration. Next, we present case studies indicating that people may want to interact with machine-learning systems in richer ways than anticipated, suggesting new input and output capabilities. Finally, we present case studies that experiment with increasing transparency about how machine-learning systems work, finding that such transparency can improve the user experience in some scenarios, as well as the accuracy of resulting models. Active learning is a machine-learning paradigm in which the learner chooses the examples from which it will learn (Settles 2010). These examples are selected from a pool of unlabeled samples based on some selection criterion (for example, samples for which the learner has maximum uncertainty). For each selected sample the learner queries an oracle to request a label. This method has had success in accelerating learning (that is, requiring fewer labels to reach a target accuracy) in applications like text classification and object recognition, where oracles are often paid to provide labels over a long period of time. However, Cakmak and colleagues (2010) discovered that when applied to interactive settings, such as a person teaching a task to a robot by example, active learning can cause several problems. Cakmak's study (figure 3) found that the constant stream of questions from the robot during the interaction was perceived as imbalanced and annoying. The stream of questions also led to a decline in the user's mental model of how the robot learned, causing some participants to "turn their brain off" or "lose track of what they were teaching" (according to their self report) (Cakmak, Choa, and Thomaz 2010). Guillory and Bilmes (2011) reported similar findings for an active movie recommendation system they developed for Netflix. These studies reveal that users are not necessarily willing to be simple oracles (that is, repeatedly telling the computer whether it is right or wrong), breaking a fundamental assumption of active learning. Instead, these systems need to account for human factors such as interruptibility or frustration when employing methods like active learning. In reinforcement learning, an agent senses and acts in a task environment and receives numeric reward values after each action. With this experience, the learning agent attempts to find behavioral policies that improve its expected accumulation of reward. A number of research projects have investigated the scenario in which this reward comes as feedback from a human user rather than a function predefined by an expert (Isbell et al. 2006 , Thomaz and Breazeal 2008 , Knox and Stone 2012 . In evaluating the feasibility of nonexpert users teaching through reward signals, these researchers aimed to both leverage human knowledge to improve learning speed and permit users to customize an agent's behavior to fit their own needs. Thomaz and Breazeal (2008) observed that people have a strong tendency to give more positive rewards than negative rewards. Knox and Stone (2012) later confirmed this positive bias in their own experiments. They further demonstrated that such bias leads many agents to avoid the goal that users are teaching it to reach (for example, the water in figure  4 ). This undesirable consequence occurs with a common class of reinforcement learning algorithms: agents that value reward accrued over the long term and are being taught to complete so-called episodic tasks. This insight provided justification for the previously popular solution of making agents that hedonistically pursue only short-term human reward, and it led Knox and Stone (2013) to create an algorithm that successfully learns by valuing human reward that can be gained in the long term. Agents trained through their novel approach were more robust to environmental changes and behaved more appropriately in unfamiliar states than did more hedonistic (that is, myopic) variants. These agents and the algorithmic design guidelines Knox and Stone created were the result of multiple iterations of user studies, which identified positive bias and then verified its hypothesized effects. In an experiment by Thomaz and Breazeal (2008) users trained a simulated agent to bake a cake through a reinforcement learning framework. In their interface, users gave feedback to the learner by clicking and dragging a mouse -longer drags gave larger-magnitude reward values, and the drag direction determined the valence (+/-) of the reward value (figure 4) . Further, users could click on specific objects to signal that the feedback was specific to that object, but they were told that they could not communicate which action the agent should take. Thomaz and Breazeal found evidence that people nonetheless gave positive feedback to objects that they wanted the agent to manipulate, such as an empty bowl that the agent is in position to pick up. These users violated the instructions by applying what could be considered an irrelevant degree of freedom -giving feedback to objects that had not been recently manipulated -to provide guidance to the agent about future actions, rather than actual feedback about previous actions. After Thomaz and Breazeal adapted the agent's interface and algorithm to incorporate such guidance, the agent's learning performance significantly improved. Other researchers have reached similar conclusions. In a Wizard-of-Oz study (that is, the agent's outputs were secretly provided by a human) by Kaochar and colleagues (2011), users taught a simu- lated unmanned aerial vehicle (UAV) to conduct various missions. At any time, these users chose whether to teach by demonstration, by feedback, or by providing an example of a concept. They could also test the agent to see what it had learned. The authors found that users never taught exclusively by feedback, instead generally using it after teaching by the other available means. Together, these two studies provide insight into the design of natural interfaces for teaching agents. People Naturally Want to Provide More Than Just Data Labels Labeling data remains the most popular method for end-user input to interactive machine-learning systems because of its simplicity and ease of use. However, as demonstrated in previous case studies, labelbased input can have drawbacks (for example, negative attitudes toward being treated as an oracle). In addition, emerging research suggests that in some scenarios users may desire richer control over machine-learning systems than simply labeling data. For example, Stumpf and colleagues (2007) conducted a study to understand the types of input end users might provide to machine-learning systems if unrestricted by the interface. The authors generated three types of explanations for predictions from a text classification system operating over email messages. These explanations were presented to people in the form of paper-based mockups to avoid the impression of a finished system and encourage people to provide more feedback. People were then asked to give free-form feedback on the paper prototypes with the goal of trying to correct the classifier's mistakes. This experiment generated approximately 500 feedback instances from participants, which were then annotated and categorized. The authors found that people naturally provided a wide variety of input types to improve the classifier's performance, including suggesting alternative features to use, adjusting the importance or weight given to different features, and modifying the information extracted from the text. These results present an opportunity to develop new machine-learning algorithms that might better support the natural feedback people want to provide to learners, rather than force users to interact in limited, learner-centered ways. In addition to wanting richer controls, people sometimes desire more transparency about how their machine-learning systems work. Kulesza and colleagues (2012) provided users of a content-based music recommender with a 15-minute tutorial discussing how the recommender worked and how various feedback controls (for example, rating songs, steering toward specific feature values, and so on) would affect the learner. Surprisingly, participants responded positively to learning these details about the system. In addition, the researchers found that the more participants learned about the recommender while interacting with it, the more satisfied they were with the recommender's output. This case study provides evidence that users are not always satisfied by "black box" learning systems -sometimes they want to provide nuanced feedback to steer the system, and they are willing and able to learn details about the system to do so. Examining transparency at a more social level, Rashid and colleagues (2006) examined the effect of showing users the value of their potential movie ratings to a broader community in the MovieLens recommendation system. Users who were given information about the value of their contribution to the entire MovieLens community provided more ratings than those who were not given such information, and those given information about value to a group of users with similar tastes gave more ratings than those given information regarding the full MovieLens community. Sometimes users make mistakes while labeling, thus providing false information to the learner. Although most learning systems are robust to the occasional human error, Rosenthal and Dey set out to solve this problem at the source. They sought to reduce user mistakes by providing targeted information when a label is requested in an active learning setting. The information provided to the user included a combination of contextual features of the sample to be labeled, explanations of those features, the learner's own prediction of the label for the sample, and its uncertainty in this prediction (Rosenthal and Dey 2010). They conducted two studies to determine the subset of such information that is most effective in improving the labeling accuracy of users. The first involved people labeling strangers' emails into categories, as well as labeling the interruptability of strangers' activities; the second involved people labeling sensory recordings of their own physical activity. Both studies found that the highest labeling accuracy occurred when the system provided sufficient contextual features and current predictions without uncertainty information. This line of research demonstrates that the way in which information is presented (for example, with or without context) can greatly affect the quality of the response elicited from the user. This case study also shows that not all types of transparency improve the performance of interactive machine-learning systems, and user studies can help determine what information is most helpful to the intended audience. Articles Summary Understanding how people actually interact -and want to interact -with machine-learning systems is critical to designing systems that people can use effectively. Exploring interaction techniques through user studies can reveal gaps in a designer's assumptions about their end users and may suggest new algorithmic solutions. In some of the cases we reviewed, people naturally violated assumptions of the machinelearning algorithm or were unwilling to comply with them. Other cases demonstrated that user studies can lead to helpful insights about the types of input and output that interfaces for interactive machine learning should support. In general, this type of research can produce design suggestions and considerations, not only for people building user interfaces and developing the overall user experience, but for the machine-learning community as well.
Personalization of machine learning itself includes works on interactive machine learning REF , where the goal is to improve a machine learning model in an iterative manner involving humans.
Power to the People: The Role of Humans in Interactive Machine Learning
AI Magazine
AI Magazine
Computer Science