Topic modeling is a common tool for understanding large bodies of text, but is typically provided as a "take it or leave it" proposition. Incorporating human knowledge in unsupervised learning is a promising approach to create high-quality topic models. Existing interactive systems and modeling algorithms support a wide range of refinement operations to express feedback. However, these systems' interactions are primarily driven by algorithmic convenience, ignoring users who may lack expertise in topic modeling. To better understand how non-expert users understand, assess, and refine topics, we conducted two user studies-an in-person interview study and an online crowdsourced study. These studies demonstrate a disconnect between what non-expert users want and the complex, low-level operations that current interactive systems support. In particular, our findings include: (1) analysis of how non-expert users perceive topic models; (2) characterization of primary refinement operations expected by nonexpert users and ordered by relative preference; (3) further evidence of the benefits of supporting users in directly refining a topic model; (4) design implications for future human-in-the-loop topic modeling interfaces. topics (Zhai et al., 2009) , or word pairs that should or should not belong together in a topic (Andrzejewski et al., 2009) . While these approaches are promising, a key limitation to the above work-both what makes for a "bad" topic model and human-in-the-loop methods-is that researchers have focused largely on algorithmic convenience rather than user needs, and-particularly-there is a dearth of studies with users who lack topic modeling experience. For example, the problems with topic model quality identified above come largely from self-reports of machine learning experts, with the exception of two domain experts in one paper (Mimno et al., 2011) . Even the human-in-the-loop approaches offer few user studies. In a study with 20 users who had varied experience with topic models, Hu et al. (2014) showed that having access to their small set of refinement operations impacted how users searched for information in a corpus compared to using a traditional, static topic model. Another exception comes from Hoque and Carenini (2015), who found that users preferred a topic model interface that allows for splitting and merging of topics over a more traditional interface that does not allow for model refinement. While these studies begin to provide insight for a more user-centered approach to topic modeling, important questions remain unanswered: How do non-expert users interpret topic models and what problems do they perceive? Disregarding technical constraints, how would they want to fix or refine those problems? Based on these answers, what are the implications for making human-in-the-loop topic models more accessible to non-expert users? We conducted two user studies with non-experts: a semi-structured interview study with 10 participants followed by an online crowdsourced experiment with 90 participants. The studies were exploratory and formative, meant to guide future algorithmic work on what refinement operations to support for human-in-the-loop topic modeling and how to prioritize those operations if need be. As such, we intentionally asked participants to interact with front-end user interfaces that were not constrained by current technical limitations of topic model implementations. For both studies, participants interacted with topic models built from a corpus of New York Times articles. In one case participants performed a theme summarization task that can act as a general first step for more complex tasks, such as for qualitative content analysis (Chuang et al., 2014) , and in the second case participants performed the more constrained but common low-level task of refining individual topics. The interview study focused on exploring how non-experts interpret, assess, and want to refine topic models without being biased by technical constraints or suggested operations; users did not have preconceptions of how refinements might impact a topic model. For the online crowdsourced experiment, we then designed and implemented a user interface to support the six most popular refinement operations identified in the interviews: Add words, Remove words, Merge words, Change word order, Remove documents, and Split topic. We evaluated the frequency of use and subjective utility of the six operations by asking participants to refine individual topics. While the refinement operations were implemented only in the user interface (the backend model was not updated), this study allows us to compare the operations in terms of frequency of use and measure their perceived impact on topic quality; we also evaluate the quality of the refined topics with automatically computed topic coherence based on NPMI (Lau et al., 2014) . Combined, our studies demonstrate a disconnect between what non-expert users want (e.g., simply adding a word to a topic) and the low-level operations that human-in-the-loop topic modeling approaches currently support, (e.g., specifying a must-link constraint (Andrzejewski et al., 2009; Hu et al., 2014) ). In addition to identifying a set of important refinement operations that should be included to support non-expert users, our findings highlight patterns in how non-experts interpret (and misinterpret) topics and apply refinement operations to individual topics. The online experiment also shows that, albeit without back-end updates, users believed their refinements significantly improved topic model quality; objective measures of topic coherence aligned with this perception. These findings should guide future algorithmic work on human-in-the-loop topic modeling. This paper makes the following contributions: (1) showing how non-expert users perceive topic models, confirming several problems asserted by machine learning researchers; (2) enumerating non-expert users' desired topic-level refinement operations and the relative preference among these refinements, particularly with relevance to the tasks of summarizing themes in a corpus and refining individual topics; (3) providing further evidence of the benefits of allowing users to directly refine a topic model (confirming (Chuang et al., 2013a; Hoque and Carenini, 2015) ), particularly in terms of the perception of topic quality; (4) providing additional implications for the design of future human-in-the-loop topic modeling interfaces, such as specific mixed-initiative support requested by users.
Lee et al. REF specify refinement operations for topic modeling that are intuitive for non-expert users, including splitting and merging topics or adding and removing keywords that have all been implemented in our system.
The Human Touch: How Non-expert Users Perceive, Interpret, and Fix Topic Models
Int. J. Hum. Comput. Stud.
Int. J. Hum. Comput. Stud.
Computer Science