Interactive Concept Learning for Uncovering Latent Themes in Large Text Collections (2023)

Abstract: Experts across diverse disciplines are often interested in making sense of large text collections. Traditionally, this challenge is approached either by noisy unsupervised techniques such as topic models, or by following a manual theme discovery process. In this paper, we expand the definition of a theme to account for more than just a word distribution, and include generalized concepts deemed relevant by domain experts. Then, we propose an interactive framework that receives and encodes expert feedback at different levels of abstraction. Our framework strikes a balance between automation and manual coding, allowing experts to maintain control of their study while reducing the manual effort required.

## introduction researchers and practitioners across diverse disciplines are often interested making sense of large text collections. thematic analysis is one of the most common qualitative research methods used to approach this challenge, and it can be understood as a form of pattern recognition in which the themes (or codes) that emerge from the data become the categories for analysis @xcite . in standard practice, researchers bring their own objectives or questions and identify the relevant themes or patterns recognized while analyzing the data, potentially grounding them in a relevant theory or framework. themes in thematic analysis are broadly defined as "patterned responses or meaning" derived from the data, which inform the research question. with the explosion of data and the rapid development of automated techniques, disciplines that traditionally relied on qualitative methods for the analysis of textual content are turning to computational methods @xcite . topic modeling has long been the go-to nlp technique to identify emerging themes from text collections @xcite @xcite . despite its wide adoption, topic modeling does not afford the same flexibility and representation power of qualitative techniques. for this reason, many efforts have been dedicated to understanding the ways in which topic models can be flawed @xcite , and evaluating their coherence and quality @xcite @xcite . more recently, @xcite showed that human judgements and accepted metrics of topic quality and coherence do not always agree. given the noisy landscape surrounding topic modeling, manual qualitative methods are still prevalent across fields for analyzing nuanced and verbally complex data @xcite @xcite . human-in-the-loop topic modeling approaches aim to address these issues by allowing experts to correct and influence the output of topic models. given that topics in topic models are defined as distributions over words, the feedback received using these approaches is usually limited to identifying representative words and imposing constraints between words @xcite @xcite . in this paper, we argue that themes emerging from a document collection should not just be defined as a word distribution (similar to a topic model), but as a distribution over generalized concepts that can help us explain them. we build on the definition put forward by @xcite , where themes are latent patterned meanings that emerge from the data, and supporting concepts serve as a way to explain themes using theoretical frameworks that are deemed relevant by domain experts. for example, emerging themes in a dataset about covid-19 can be characterized by the strength of their relationship to stances about the covid vaccine and the moral framing of relevant entities (e.g. the theme "government distrust" is strongly correlated to an anti-vax stance and frames dr. fauci as an entity enabling cheating). this representation of a theme aligns more closely with qualitative practices, as experts can introduce their pre-existing knowledge about the domain. moreover, higher-level abstractions expand the capabilities of experts to correct and influence theme discovery, as it allows them to formulate concepts to generalize from observations to new examples @xcite , and to deductively draw inferences via conceptual rules and statements @xcite . following this rationale, we suggest a new computational approach to support and enhance standard qualitative practices for content analysis. we approach both inductive thematic analysis (i.e. identifying the relevant themes that emerge from the data and developing the code-book), and deductive thematic analysis (i.e. identifying the instances where a known theme is observed). to support this process, we allow researchers to shape the space of themes given machine generated candidates. then, we allow them to provide feedback over machine judgments that map text to themes using relevant conceptual frameworks. to showcase our approach, we look at the task of characterizing social media discussions around topics of interest to the computational social science community. namely, we consider two distinct case studies: the covid-19 vaccine debate in the us, and the immigration debate in the us, the uk and europe. for each case, the qualitative researchers use different theories to ground theme discovery, each associated with a different set of concepts. for the covid-19 vaccine debate, the theme discovery process is grounded using vaccination stances and morality frames @xcite . for the immigration debates, the theme discovery is grounded using three framing typologies: narrative frames @xcite , policy frames @xcite and immigration frames @xcite . all of these choices build on previous work and were validated by the qualitative researchers. from a machine learning perspective, these two case studies could be regarded as completely different tasks and have been approached independently in previous work. the reason for this is the data, the context, and the target labels (both the emerging themes and the supporting concepts) are different for each scenario. to aid experts in theme discovery, we propose an iterative two-stage machine-in-the-loop framework. in the first stage, we provide experts with an automated partition of the data, ranked example instances, and visualizations of the concept distribution. then, we have a group of experts work together to explore the partitions, code emerging patterns and identify coherent themes. once themes are identified, we have the experts select representative examples, write down additional examples and explanatory phrases, and explain themes using the set of available concepts. in the second stage, we incorporate the expert feedback using a neuro-symbolic mapping procedure. the symbolic part allows us to explicitly model the dependencies between concepts and the emerging themes using weighted logical formulae (e.g. w : policy_frame(economic) ⇒ theme(economic_migrants). these rules can be interpreted as soft constraints whose weights are learned from the feedback provided by the experts. the neural part allows us to maintain a distributed representation of the data points and themes, which facilitates the live exploration of the data based on distances and similarities, and provides a feature representation for learning the rule weights. after the mapping stage concludes, some instances will be assigned to the identified themes, and the remaining instances will be re-partitioned for a consecutive discovery stage. we conducted extensive evaluations of the different components, design choices, and stages in our methodology. we showed that our framework allows experts to uncover a set of themes that cover a large portion of the data, and that the resulting mapping from tweets to themes is fairly accurate with respect to human judgements. while we focused on polarized discussions, our framework generalizes to any content analysis study where the space of relevant themes is not known in advance. ## related work this paper suggests a novel approach for identifying themes emerging from text collections. the notion of a theme presented in this work is strongly related to topic models @xcite . however, unlike latent topics that are defined as word distributions, our goal is to provide a richer representation that more strongly resembles qualitative practices by connecting the themes to general concepts that help explain them. for example, when identifying themes emerging from polarized discussions in social media, we look at conceptual frameworks such as moral foundations theory (haidt and graham, 2007; amin et al., 2017; chan, 2021 ) and framing theory @xcite @xcite . our work is conceptually similar to recent contributions that characterize themes and issue-specific frames in data, either by manually developing a code-book and annotating the data according to it @xcite , or by using data-driven methods @xcite . unlike these approaches, our work relies on interleaved humanmachine interaction rounds, in which humans can identify and explain themes from a set of candidates suggested by the model, as well as diagnose and adapt the model's ability to recognize these themes in documents. this work is part of a growing trend in nlp that studies how humanmachine collaboration can help improve automated language analysis @xcite . in that space, two lines of works are most similar to ours. interactive topic models @xcite @xcite allow humans to adapt the identified topics, but the feedback is usually limited to lexical information. open framing @xcite allows humans to identify and name frames based on the output of topic models, but lacks our model's ability for sustained interactions that help shape the theme space, as well as the explanatory power of our neuro-symbolic representation. ## the framework we propose an iterative two-stage framework that combines ml/nlp techniques, interactive interfaces and qualitative methods to assist experts in characterizing large textual collections. we define large textual collections as repositories of textual instances (e.g. tweets, posts, documents), where each instance is potentially associated with a set of annotated or predicted concepts. in the first stage, our framework automatically proposes an initial partition of the data, such that instances that are thematically similar are clustered together. we provide experts with an interactive interface equipped with a set of exploratory operations that allows them to evaluate the quality of the discovered partitions, as well as to further explore and partition the space by inspecting individual examples, finding similar instances, and using open text queries. as experts interact with the data through the interface, they following an inductive thematic analysis approach to identify and code the patterns that emerge within the partitions @xcite . next, they group the identified patterns into general themes, and instantiate them using the interface. although intuitively we could expect a single partition to result in a single theme, note that this is not enforced. experts maintain full freedom as to how many themes they instantiate, if any. once a theme is created, experts are provided with a set of intervention operations to explain the themes using natural language, select good example instances, write down additional examples, and input or correct supporting concepts to characterize the theme assignments. the full set of operations are listed in tab. 1 and demonstrated in app. a.1. in the second stage, our framework finds a mapping between the full set of instances and the themes instantiated by the experts. we use the information contributed by the experts in the form of examples and concepts, and learn to map instances to themes using our neuro-symbolic procedure. we allow instances to remain unassigned if there is not a good enough match, and in this case, a consecutive portioning step is done. we refer to instances that are mapped to themes as "named partitions" and unassigned proposed partitions as "unnamed partitions". once instances are assigned to themes, experts have access to a comprehensive visual analysis of the state of the system. the main goal of this analysis is to appreciate the trade-off between coverage (how many instances we can account for with the discovered themes) and quality (how good we are at mapping instances to themes). an illustration of the framework can be observed in fig. 1 . additional details about the coverage and quality analysis are presented in the experimental section. below, we discuss the representation of themes and instances, the protocol followed for interaction, and the mapping and re-partitioning procedures. our visual interface and our analysis code have been made available to the community foot_0 . representing themes and instances we represent example instances and explanatory phrases using their s-bert embedding @xcite . to measure the closeness between an instance and a theme, we compute the cosine similarity between the instance and all of the explanatory phrases and examples for the theme, and take the maximum similarity score among them. our framework is agnostic of the representation used. the underlying embedding objective and the scoring function can easily be replaced. ## case studies we explore two case studies involving discussions on social media: (1) the covid-19 vaccine discourse in the us, and (2) the immigration discourse in the us, the uk and the eu. for the covid-19 case, we build on the corpus of 85k tweets released by @xcite . all tweets in this corpus were posted by users located in the us, are uniformly distributed between jan. and oct. 2021, and contain predictions for vaccination stance (e.g. pro-vax, anti-vax) and morality frames (e.g. fairness/cheating and their actor/targets.) @xcite . for the immigration case, we build on the corpus of 2.66m tweets released by @xcite . all tweets in this corpus were posted by users located in the us, the uk and the eu, written between 2018 and 2019, and contain predictions for three different framing typologies: narrative frames (e.g. episodic, thematic) @xcite , generic policy frames (e.g. economic, security and defense, etc.) @xcite , and immigration-specific frames (e.g. victim of war, victim of discrimination, etc.) @xcite . additional details about the datasets and framing typologies can be found the original publications. our main goal is to evaluate whether experts can leverage our framework to identify prominent themes in the corpora introduced above. we recruited a group of six experts in computational social science, four male and two female, within the ages of 25 and 45. the group of experts included advanced graduate students, postdoctoral researchers and faculty. our studies are irb approved, and we followed their protocols. for each corpus, we performed two consecutive sessions with three experts following the protocol outlined in sec. 3. to evaluate consistency, we did an additional two sessions with a different group of experts for the covid-19 dataset. each session lasted a total of one hour. in app. a.2, a.3 and a.4, we include large tables enumerating the resulting themes, and describing in detail all of the patterns identified and coded by the experts at each step of the process. coverage vs. mapping quality we evaluated the trade-off between coverage (how many tweets to do this evaluation, we sub-sampled a set of 200 mapped tweets for each scenario, uniformly distributed across themes and their proximity to the theme embedding, and validated their assignments manually. the logic behind sampling across different proximities is that we expect mapping performance to degrade the more semantically different the tweets are to the "good" examples and phrases provided by the experts. to achieve this, we look at evaluation metrics at different thresholds using the quartiles with respect to the proximity/similarity distribution. results for q 1 correspond to the 25% most similar instances. for q 2 to the 50% most similar instances, and for q 3 to the 75% most similar instances. note that these are continuous ranges and the quartiles serve as thresholds. to evaluate the impact of our neuro-symbolic mapping procedure (nesym), we compared it against a nearest neighbors (nns) approach that does not leverage conceptual frameworks and looks only at the language embedding of the tweets and theme examples and explanatory phrases. for the first iteration of covid-19, we find that the approximate performance of the nesym mapping at q1 is better (+2 points) than the approximate full mapping for nns, while increasing coverage x1.5. for immigration, we have an even more drastic result, having an approximate 15 point increase at a similar coverage gain. in both cases, experts were able to increase the number of themes in subsequent iterations foot_1 . while the coverage increased in the second iteration for covid, it decreased slightly for immigration. for covid, most of the coverage increase can be attributed to a single theme (vax efforts progression), which accounts for 20% of the mapped data. in the case of covid, this large jump in coverage is accompanied by a slight decrease in mapping performance. in the case of immigration, we have the opposite effect: as the coverage decreases the performance improves, suggesting that the mapping gets stricter. this confirms the expected trade-off between coverage and quality. depending on the needs of the final applications, experts could adjust their confidence thresholds. to perform a fine-grained error analysis, we looked at the errors made by the model using manual validation. in fig. 3 we show the confusion matrix for the covid case. we find that the performance varies a lot, with some themes being more accurate than others. in some cases, we are good at capturing the general meaning of the theme but fail at grasping the stance similarities. for example, anti vax spread missinfo gets confused with pro vax lie, where the difference is on who is doing the lying. in other cases, we find that themes that are close in meaning have some overlap (e.g. alt treatments with vax doesn't work). we also find that unambiguous, neutral themes like vax appointments, got the vax and vax efforts progression have the highest performance. lastly, we observe that for some errors, none of the existing themes are appropriate (last row: other), suggesting that there are still undiscovered themes. upon closer inspection, we found that the majority of these tweets are among the most distant from the theme embedding. the full distribution of other per interval can be observed in app. a.6. we include the confusion matrix for immigration in app. a.6. given our hypothesis that themes can be characterized by the strength of their relationship to high-level concepts, we consider mappings to be better if they are more cohesive. in the covid case, we expect themes to have strong relationships to vaccination stance and morality frames. in the immigration case, we expect themes to have strong relationships to the framing typologies. to measure this, we define a theme purity metric for each for lda, we assigned a tweet to its most probable topic if the probability @xmath0.5. ## limitations the study presented in this paper has three main limitations. (1) while the design of the framework does not prohibit the utilization of longer textual forms, the two case studies presented deal with short texts. when dealing with longer text forms, we need to consider the cognitive load of having experts look at groups of instances. in our ongoing work, we employ strategies such as summarization, highlighting and other visualization techniques to deal with these challenges. (2) in the studies presented, qualitative researchers worked in groups to identify themes. our goal in comparing two independent groups of researchers was to evaluate the degree of subjectivity by observing if the themes identified by the two groups would diverge. this setup might not always be realistic, as a lot of times qualitative researchers work independently or asynchronously. in the future, we will explore the effect of the crowd in minimizing subjectivity, as well as the role that the computational tools play in more challenging settings. (3) finally, we did not include a comprehensive user study to gather input from the experts about their experience with our framework. we consider this to be an important next step and we are actively working in this direction. ## summary we presented a concept-driven framework for uncovering latent themes in text collections. our framework expands the definitions of a theme to account for theoretically informed concepts that generalize beyond word co-occurrence patterns. we suggest an interactive protocol that allows domain experts to interact with the data and provide feedback at different levels of abstraction. we performed an exhaustive evaluation using two case studies and different groups of experts. additionally, we contrasted the extracted themes against the output of traditional topic models, and showed that they are better at capturing conceptual similarities that go beyond word distributions.