Heterogeneous Supervised Topic Models (2022)

Abstract: Researchers in the social sciences are often interested in the relationship between text and an outcome of interest, where the goal is to both uncover latent patterns in the text and predict outcomes for unseen texts. To this end, this paper develops the heterogeneous supervised topic model (HSTM), a probabilistic approach to text analysis and prediction. HSTMs posit a joint model of text and outcomes to find heterogeneous patterns that help with both text analysis and prediction. The main benefit of HSTMs is that they capture heterogeneity in the relationship between text and the outcome across latent topics. To fit HSTMs, we develop a variational inference algorithm based on the auto-encoding variational Bayes framework. We study the performance of HSTMs on eight datasets and find that they consistently outperform related methods, including fine-tuned black-box models. Finally, we apply HSTMs to analyze news articles labeled with pro- or anti-tone. We find evidence of differing language used to signal a pro- and anti-tone.

## introduction researchers in the social sciences are interested in modeling the relationship between text and an outcome of interest. in surveys about elections, how do respondents' open-ended responses relate to evaluations of their anger, fear, or concern @xcite ? in news media, how do articles relate to the tone used by the writer towards the article's topic @xcite ? on social media, how do tweets on mass shootings relate to the political ideology of the user @xcite ? in these settings, the goal is to both uncover latent properties about the text and predict outcomes for unseen texts. we develop the heterogeneous supervised topic model (hstm), a probabilistic approach to modeling outcomes from text. the key innovation in hstms-the hetereogeneity-is the idea that individual words can be predictive of the outcome not just on their own, but also specifically in combination with latent topics in the text. as a running example, consider news articles about us immigration labeled with the pro-or anti-tone of the writer towards us immigration @xcite . as is standard for topic models @xcite , hstms learn the hidden topics in the corpus, which are captured by the ''neutral'' words shown in each topic. for example, us immigration articles discuss legal aspects (first topic) and views of the administration (second topic). however, hstms also analyze the per-topic, heterogeneous associations between the text and outcome. in a fitted hstm can also perform supervised prediction of the outcome from text by using its model of the outcome based on the interactions between the topics and words. for example, an hstm fit to the immigration articles can predict the pro-or anti-immigration tone of unseen articles about us immigration. the hstm contributes to the large body of work on modeling outcomes from text. one common approach is to fine-tune large language models such as bert @xcite . these black-box methods capture complex interactions between parts of text and the outcome, which often leads strong predictive performance. however, it is difficult to visualize the parts of text that drive predictions @xcite . even explanation methods such as saliency maps, rationales, and analysis of attention mechanisms are not robust @xcite @xcite and are often not faithful to the underlying prediction model @xcite . since it is difficult to explain the predictions of black-box methods, there has been a push towards more transparent models @xcite . probabilistic models of text such as topic models @xcite tend to be interpretable to experts @xcite and are a mainstay in social science research for understanding patterns in text data @xcite . their supervised counterparts such as supervised topic models @xcite , sparse additive models @xcite , multinomial text regression @xcite , and related variants @xcite retain this interpretability, enabling exploratory data analysis. however, these probabilistic methods do not capture heterogeneous relationships between text and outcomes, limiting their predictive performance. the hstm enjoys the interpretability of topic models while extending them to capture flexible relationships between text and outcomes. the idea behind hstms is that the observed correlation among words provides evidence for latent topics, and latent topics mediate the associations between words and the outcome of interest. to implement this idea, the hstm posits a joint generative model that captures how latent topics drive both text documents and outcomes. we fit hstms with auto-encoding variational bayes @xcite . we study the hstm across eight datasets that range from product reviews to surveys about immigration. 1 first, we evaluate the predictive performance of several methods for supervised text prediction. we compare hstms to related topic-modeling approaches @xcite @xcite and fine-tuned bert models @xcite . the topic modeling approaches are easy to visualize while the black-box bert models are difficult to interpret. we find that the hstm achieves significantly better predictive performance than its topic modeling counterparts across all settings, and performs competitively with bert in five out of eight settings. next, we perform an ablation study to understand the effect of different modeling choices made in designing the hstm. we find that the heterogeneous model of outcomes improves text prediction. finally, we use news articles to study the use of hstms for qualitative data exploration. ## background consider a corpus of n text documents and their associated outcomes @xmath0, y 1 ), . . . , (w n , y n )}. each document w i is a sequence of m word tokens w @xmath1. . . w im } that come from a vocabulary of size v . the corpus can also be represented as a bag-of-words (bow) with matrix x, where x iv is the frequency (or occurrence count) of word v in document i. the outcome y i can be real-valued, binary, or categorical. this work builds on topic models, probabilistic models of latent variables and observed text. we extend a topic model based on products of experts (poe) @xcite . in a poe topic model, each word w ij comes from a product of k (possibly unnormalized) distributions called experts, each document is associated with a local latent variable θ i on the k-dimensional simplex ( k ). 2 each entry θ ik is the document's affinity for the k-th expert. the k-th expert is governed by the global variable the poe topic model is different than latent dirichl et allocation (lda) @xcite , a mixed membership model. to draw a word from lda, it must have high probability in at least one component. in contrast, in a poe topic model, to draw a word, all k experts must have a high affinity for the word. consequently, poe topic models tend to result in more contrasting topics @xcite . ## the heterogeneous supervised topic model we present the heterogeneous supervised topic model (hstm). it combines a poe topic model of words with a heterogeneous model of the outcomes. hstms capture how associations between words and the outcome vary across latent topics. the hstm is a generative model of a text and outcome pair (w i , y i ). to generate the text w i , the hstm uses the poe topic model described in § 2, and involves the same global and local latent variables β and θ i . additionally, it includes a global intercept term @xmath2. the model of text is each entry b v is an intercept that reflects the baseline popularity of the word v in the corpus. in including this intercept, we follow @xcite , @xcite . the local latent variable θ i is parameterized by a logistic-normal prior, r @xmath3, i) the variables β describe the latent structure of text. in the outcome model involves the global variables β and local variables θ i . additionally, the outcome model includes global latent variables @xmath4, ω ∈ r v , and γ @xmath5, which are vectors of coefficients. let θ @xmath6, θ i , β, γ, a, ω}. the outcome model is the first term in eq. 5 is θ i a. it captures the relationship between the context of a document and the outcome, similar to supervised topic models @xcite @xcite . the second term is x i ω. the coefficients ω capture which words are predictive of the outcome, regardless of the context. when the value ω v is negative, the vocabulary word v has a negative pull on the outcome. the interpretation is similar for positive values. in the last term of eq. 5, the word frequency x iv is multiplied by a quantity that is factorized over the k experts. each term θ ik β kv is the contribution of expert k to the unnormalized log probability of word v in document i. the variable γ modulates this term to predict the outcome. in since the variables γ and β are both real-valued, we cannot immediately interpret positive (and negative) values of γ as pro and anti words. ## inference given a dataset d of documents and outcomes, we perform maximum a posteriori estimation of the global variables μ = {β, γ, a, ω} and variational inference for the local latent variables r i , which is θ i on the logit scale @xcite . we follow the work on auto-encoding variational bayes @xcite . the local latent variable r i comes from an amortized variational family q φ (r i | w i ), a multivariate normal distribution whose mean and diagonal covariance are parameterized by a neural network called the encoder. the encoder has weights φ and take the text w i as input. the expectation is taken with respect to the variational distribution q φ (r i | w i ). the term log p(μ) corresponds to l1 and l2 penalty terms, we maximize the elbo with stochastic gradient descent. when taking gradients, with monte carlo and sample r ∼ q φ (r i | w i ) by drawing from n(0, 1) and applying the locationscale transformation with the mean and variance of q φ (•). then ∇l(φ, μ) can be computed with automatic differentiation. (we use pytorch; implementation details are discussed below.) after fitting hstms, we use the estimated values β, ω, and γ to visualize the results in to form predictions for a new document w d , we obtain its representation θd by passing the words through the encoder. we make predictions using eq. 5 with the fitted variables. ## related work hstms contribute to the work on extending topic models @xcite for supervised prediction @xcite @xcite @xcite and modeling covariates @xcite such as authorship @xcite , political ideology @xcite , voting behavior @xcite , or category tags @xcite . in this line of work, hstms are most closely related to the neural models for documents with metadata (i.e., outcomes) proposed by @xcite . @xcite develop a general framework for topic models fit using auto-encoding variational bayes. in their framework, either the outcomes are predicted from the inferred topics or interactions between the topics and outcomes are used to model text. in the latter task, the interaction effects learned by the neural models are similar to the heterogeneity that hstms capture. however, the neural model framework does not accommodate predicting outcomes and capturing heterogeneity with the same model. in contrast, hstms can simultaneously capture topic-driven heterogeneity and enable supervised prediction based on the heterogeneous patterns. hstms also relate to the literature on neural topic models @xcite @xcite @xcite @xcite @xcite . within this line of work, the hstm builds on @xcite who proposed a poe topic modzel and adapted auto-encoding variational bayes (kingma and welling, 2013) for topic models. in contrast to their work, which proposes a generative model of text, hstms are joint models of text and outcomes, enabling supervised prediction and text analysis that reveals how text relates to the outcome of interest. ## empirical studies we empirically evaluate hstms with eight datasets. the code and data to reproduce the results are available at https://github.com /dsridhar91/hstm . our empirical analysis is driven by three key questions: 1) how does the predictive performance of hstms compare to that of related methods? 2) how do different modeling components of hstms contribute to their performance? 3) how do hstms help exploratory text analysis? for predictive performance, we find that: 1) in all eight settings that we study, the predictive performance of hstms significantly improves upon the results of approaches that also use a bagof-words approach to representing text, making them easy to visualize; 2) in five out of eight settings, hstms are competitive with fine-tuned bert models, a state-of-the-art transformer based approach that is difficult to interpret (in contrast, hstms are easy to visualize); and 3) in four out of eight settings, the terms in eq. 5 that capture heterogeneity in the outcome model significantly improve prediction. for exploratory text analysis, we apply hstms to analyze news articles labeled with the writer's pro-or anti-tone. using the media framing corpus @xcite , we study articles on us immigration, same sex marriage, the death penalty and gun control. in all article subjects, we find evidence of differing pro-issue and anti-issue word choices across learned topics. ## discussion we addressed the problem of modeling the relationship between text and an outcome of interest in service of text analysis and supervised prediction. we proposed the hstm to jointly model the structure in text and capture heterogeneity in the relationship between text and outcomes based on the latent topics that drive the text. we evaluated the hstm on eight prediction settings and find that it outperforms related methods. we conducted an ablation study to examine how different components of the hstm contribute to its performance. the study revealed that the hstm's heterogeneous model of outcomes improves prediction. in addition to forming predictions, the hstm lets us explore text datasets. we applied the hstm to four corpora of news articles labeled with pro/anti tone, and discovered pro and anti wording choices. the hstm opens several avenues of future work. one area to explore is combining black-box language models such as bert @xcite with the hstm to leverage both word embeddings and topics. another area is adapting the hstm to produce dynamic topics that vary over time @xcite .