Rethinking Topic Modelling: From Document-Space to Term-Space (2020)

Abstract: This paper problematizes the reliance on documents as the basic notion for defining term interactions in standard topic models. As an alternative to this practice, we reformulate topic distributions as latent factors in term similarity space. We exemplify the idea using a number of standard word embeddings built with very wide context windows. The embedding spaces are transformed to sparse similarity spaces, and topics are extracted in standard fashion by factorizing to a lower-dimensional space. We use a number of different factorization techniques, and evaluate the various models using a large set of evaluation metrics, including previously published coherence measures, as well as a number of novel measures that we suggest better correspond to real-world applications of topic models. Our results clearly demonstrate that term-based models outperform standard document-based models by a large margin.

## introduction topic models are often used in real-world text analysis scenarios as tools for efficient data exploration. the typical modus operandi in such scenarios is to run a topic model with standard parameter settings on the data, and to extract some fixed number n of topics and some fixed number m of words per topic, and then manually interpret, and draw conclusions from, the resulting term lists. a common choice for both n and m is around 10. this means that the human analyst only needs to look at around 100 terms in total instead of reading a text collection consisting of possibly several hundreds of thousands, or even millions, of running words. in terms of efficiency, this is an invaluable tool for content analysis. topic models extract topics by uncovering (latent) interactions between terms in document space. this methodology obviously assumes that data arrives with clear and consistent document boundaries, and, in the best case, a fairly even distribution of number of words per document. unfortunately, this assumption rarely holds in realworld scenarios, where data may arrive in streams, in batches without clear document boundaries, or with very large variations in document lengths. to handle such scenarios, it would be desirable to use a model that is insensitive to the formatting of the input data. in this paper, we discuss and evaluate one such approach, which embeds the topic modelling process entirely in term space. this makes the model less sensitive to document-formatting, and, as it turns out, also more precise. this work is primarily motivated by the practical usability of topic models in real-world analysis scenarios. in such applications -common in particular in the social sciences, and in security and defence applications -the analyst only cares about the top ranked terms in the resulting term lists. we therefore introduce a number of additional evaluation metrics for topic models, which may correspond better to practical considerations than the commonly used intrinsic (and mostly theoretical) evaluation measures. we also provide an evaluation that casts the topic modelling as a document annotation scenario, and that uses manual annotations as gold standard. our results -across all evaluation metrics -clearly demonstrate that termbased approaches outperform standard documentbased topic models by a large margin. despite the choice of factorization method, all document-based topic models rest on the assumption that latent interactions between terms are due to topical variation in document space. this assumption is neatly summarized by the generative story told by models such as plsa @xcite or lda, which amounts to a subject choosing a (set of) topic(s) to talk about, and for each topic choosing a set of representative terms to utter. this story makes intuitive sense, but note that the notion of a document is completely ad hoc to the story; it only enters the story as the unit of text being output by the subject. we argue that the notion of a document is an unnecessary restriction for topic models that limit the application of such models to data with proper formatting, and that topical term interactions can be better modelled directly in term space. ## from document space to term space we thus suggest to focus entirely on term space, and to remove the dependence on the notion of documents completely. instead of building term vectors for each document in the data, (i.e. a standard vsm), we build word embeddings for all terms in the data from large context windows spanning something like 50 tokens. 1 using such wide context windows ensures that the embeddings have the capacity to encode wider, and thus more topical, contextual information. there are many ways to build word embed-dings. we include four different approaches in this paper: these techniques have their respective merits and drawbacks. the cooc approach is simple and straightforward, but the dimensionality of the embeddings is equivalent to the size of the vocabulary, which can become prohibitive for large data. ri solves the dimensionality problem, since it uses fixed-sized vectors, but at the cost of added noise. w2v is widely acknowledged to be both efficient and precise, but requires sufficient amounts of training data. d2v, on the other hand, is designed primarily for document-processing applications, which might make it an interesting candidate for more topic-oriented applications, such as the present one. for each of the resulting word embeddings, we compute a similarity matrix that contains the pairwise similarities between all term vectors in the embedding space. we prune the similarity matrix by removing entries with too small values, which gives us a sparser similarity space to operate in. this is beneficial from a computational perspective, and it also removes noise from the representations. to extract topics from the similarity space, we can apply any type of algorithm that identifies clusters or latent variables. 2 in our case, we opt for a number of simple factorization methods, including: for each of these factorization techniques, we extract n components (where n defaults to 10) in the same way as in a standard topic model. in the experiments in sections 6.3 to 6.5, all results are averaged over 10 runs of the various factorization techniques. ## related work there have been a few previous studies that explore the use of term-based representations for topic modelling. one example is @xcite , who also base their solution on a termterm matrix, but their term-term matrix is not a co-occurrence matrix but a correlation matrix produced from a standard term-document matrix. as such, their model still relies on the data being properly formatted in a coherent document structure. by contrast, the models we consider do not put any constraints on the formatting of the data, while at the same time adopting a stricter definition of topical relationships in the form of context windows spanning (in our case) 50 terms, which is typically significantly smaller, and thus more precise, than a whole document. another example is rangarajan sridhar (2015), who cluster a word embedding (produced with word2vec) using a gaussian mixture model (gmm). this is the previous work that comes closest to the approaches we consider, but there are a number of significant differences. we explore a range of word embedding techniques, we use wider context windows (50 terms instead of 11-17), and we use a range of standard factorization techniques instead of gmm to extract term clusters. despite these differences, we consider rangarajan sridhar (2015) to be an important inspiration to our work. also similar in spirit to our work is @xcite , who incorporate a word similarity matrix with a standard document-based nmf model. the word similarity matrix is built using the skipgram model from word2vec, and is used as an additional term in the block coordinate descent algorithm used to solve the nmf. the approach, aptly named semantics assisted nmf (seanmf) is primarily designed for data with short documents, in which case the size of the context windows used for the skipgram embeddings equals the length of the documents in the data. @xcite argue that the sparsity of their word similarity matrix is highly beneficial for the efficiency of the model, and the same advantage consequently applies to our case. the most significant difference between the seanmf model and the approaches we consider is that the latter rely only on the word similarity matrix, and thus do not use any term-document matrix at all. in contrast to these previous studies, we focus on the general idea of using word embeddings rather than a vsm as the basis for topic modelling, and we compare a range of different word embeddings using a range of different factorization techniques. we also use a wider range of evaluation methods, and introduce a number of novel measures that better correspond to practical usage of topic models. ## evaluation methods since (most) practical use of topic models only focus on the resulting lists of terms, this should also be our focus for evaluation. a challenge here is that determining the quality of term lists can be a notoriously subjective task, comparable (jokingly) to reading tea leaves @xcite . there have been attempts to arrive at more objective evaluation measures for topic models, which usually take the form of using various forms of intrinsic information measures such as entropy, perplexity, or coherence @xcite @xcite . however, such information theoretic measures do not always correlate with semantic interpretability, as noted by @xcite , and even if they do, it is not clear why semantic interpretability should correlate with topical coherency. in light of these difficulties, it is somewhat remarkable that gold standard topic annotations are not used habitually as standard evaluation metric for topic models. of course, we will probably not be able to find such annotations for individual terms or term lists, but we might be able find such annotations at the text level. even if topic modelling is essentially different from text categorization and text clustering, we can still use text categories as evaluation targets for topic models, given that the categories are topical in nature. that is, if we can find a text collection where the text has been manually labelled with one or more topics, we can simply compare this gold standard topic assignment with that produced by a topic model. one simple way to do this, which also simulates how a human analyst might use the output of a topic model in a practical analysis scenario, is to collect all the documents covered by each topic (i.e. in which one or more of the terms in the topic list occurs), and then count the overlap between this set of documents and the set of documents labelled by topic categories in the gold standard. doing so will arrive at a proportion of overlap between the topic model and the gold standard. we argue that this is a simple and straightforward way to evaluate topic models that maps directly to usability in practical application. we refer to this metric as "truth" in tables 2 to 5 . a human analyst might also be interested in other factors, such as: we quantify this as the proportion of identical terms in the topics; lower is presumably better from an analyst's perspective. we also include the uci @xcite ) and umass @xcite coherence measures as a comparison. these measures sum the pmi values of all pairs of words in the topics; the umass measure considers co-occurrences within the entire document, while the uci measure delimit co-occurrences within a sliding window: following @xcite , we @xmath0, in our case @xmath1.001. ## experiments the following experiments use a number of different datasets, built from two different data sources. the first data source is swedish news, which have been manually collected and annotated with topics by human experts. as such, this dataset corresponds well to a real-world analysis scenario. however, since the swedish dataset is relatively small, and not publicly available, foot_3 we also generate a number of artificially annotated english datasets based on the english wikipedia. the various datasets are detailed in ## discussion as is obvious from our experiments in this paper, different topic models have different properties, and the proper choice of topic model depends on the specific information need of a particular analysis scenario. even if term-based models in general outperform standard document-based models across all data and metrics used in this paper, there may still be situations where a document-based model would be suitable to use. one such scenario would be if the analyst requires a solution with large coverage of the data; document-based models tend to lead to higher coverage of the data, but there tends to be overlap between topics, and the topic assignments (counted as the occurrence of topic terms in documents) are less unique compared to term-based models. term-based models, on the other hand, produce more unique topics with less overlap, and better separation, between topics. the term-based models also reach higher scores on all evaluation metrics (umass and uci coherence, representation separation, and overlap with truth labels) -with the exception of the swedish data, where document-based models lead to higher umass coherence. in general, the difference between document-based and term-based models is lower when considering the umass measure than when looking at the uci measure, which may be explained by the fact that the former uses documents as units for counting co-occurrences, while the latter uses words. we note that there is a high variance between runs, which makes it difficult to draw any definite conclusions regarding the optimal design choice for a term-based topic model. certain factorization techniques seem to be more suitable for certain representations and certain data. nmf in general seems to work best in these experiments for most word embeddings on the swedish data, but dictionary learning works best in these experiments for the english data. on the other hand, if the topics are small, svd seems to work better, in particular for the ri embeddings. with regards to the different types of word embeddings, we note that the cooc model typically leads to the highest coverage of the data, followed by ri, which also tends to have the best overlap with human gold standard annotations, except for the case of large topics where word2vec is significantly better. we note that word2vec and doc2vec both have high average separation of terms within vs. across topics, but that the addition of document information in doc2vec does not seem to be useful for topic inference. note that the data used in these experiments contain only one topic per document, whereas many other topic modelling scenarios operate with multiple topics per document. we do not consider this restriction to have any effect on the generality of our results, since term-based models are eminently applicable to multi-topic scenarios. the proposed gold standard comparison is also directly applicable to multi-topic data. ## conclusion this paper has demonstrated the usefulness of casting the topic inference in topic models as pursuit of latent factors in term-space rather than document-space. we have proposed a simple term-based model that uses standard word embeddings with standard factorization techniques. despite their simplicity, such term-based models outperform all tested document-based models on all evaluation metrics used in this paper. we have also proposed a topic categorization task that utilizes gold standard topic annotations, as well as a range of other metrics that may correspond more closely to a real-world analysis scenario than the type of intrinsic measures commonly used in literature on topic models. the use of these additional measures enables us to characterize the different properties of topic models, and to make informed choices of topic model design for specific information needs. our experiments have demonstrated that the optimal model is likely to be data-and task specific, and that the optimal choice of specific representation and factorization technique will likely be different from case to case. however, as a robust baseline, we suggest to use word2vec representations with nmf factorization. we conclude that term-based models are competitive, if not superior, in comparison with traditional document-based models, with a number of added benefits that include independence of document-formatting, and relative robustness to topic size. although the models investigated in this paper outperform document-based models on all metrics, we consider our term-based approach to be a simple baseline model with a large potential for improvement.