Adaptive Mixed Component LDA for Low Resource Topic Modeling (2021)

Abstract: Probabilistic topic models in low data resource scenarios are faced with less reliable estimates due to sparsity of discrete word co-occurrence counts, and do not have the luxury of retraining word or topic embeddings using neural methods. In this challenging resource constrained setting, we explore mixture models which interpolate between the discrete and continuous topic-word distributions that utilise pre-trained embeddings to improve topic coherence. We introduce an automatic trade-off between the discrete and continuous representations via an adaptive mixture coefficient, which places greater weight on the discrete representation when the corpus statistics are more reliable. The adaptive mixture coefficient takes into account global corpus statistics, and the uncertainty in each topic’s continuous distributions. Our approach outperforms the fully discrete, fully continuous, and static mixture model on topic coherence in low resource settings. We additionally demonstrate the generalisability of our method by extending it to handle multilingual document collections.

## introduction in topic modeling, the goal is to learn key themes in a corpus for exploratory document analysis @xcite . latent dirichlet allocation (lda; @xcite ) has been the bedrock for topic modeling and remains a hard to beat baseline for the general scenario which models with only words and documents. we examine topic modeling in a low resource data setting @xcite , which has seen little attention but is commonly encountered in the digital humanities where document collections are potentially small @xcite @xcite . 1 in such scenarios, word co-occurence statistics are less reliable due to sparsity of discrete counts. with the rise of neural word embeddings @xcite , the defacto approach to improving over discrete models has been to utilise continuous representations (regardless of whether the setting is low resource). early work by @xcite introduced topic dependent word embeddings, while others subsequently use embeddings to influence the discrete topic-word distribution @xcite . however, the low resource scenario constrains us to existing pre-trained embeddings, as the number of train documents is limited to several thousands and thus prohibitively small to train neural models @xcite @xcite @xcite . we therefore consider approaches that do not require further tuning of embeddings, and operate within the well established lda probabilistic inference framework in the continuous space. there have been multiple attempts to replace discrete words with pre-trained embeddings (replacing the multinomial topic-word distribution with a continuous topic-word distribution), doing away with discrete words completely @xcite @xcite . given the dominance of pre-trained word embeddings in modern nlp, would continuous representations outperform discrete representations even in low resource settings? surprisingly, we find that discrete lda outperforms its fully continuous counterpart on topic coherence measures which correlate with human judgement @xcite . how then can we utilise pre-trained continuous representations for learning better topics? a natural direction is hybrid models based on statistical counts and pre-trained neural representations @xcite . early work by @xcite used a mixture of discrete and continuous topic-word distributions with static mixture coefficients. however we find that this does not improve over discrete lda, which motivates a more nuanced treatment of the mixture coefficient. in this work, we introduce an adaptive mixture coefficient specific to each word and each topic, which is updated at every step of collapsed gibbs sampling @xcite . the intuition is as follows, topic anchor words @xcite which have stronger signal from corpus statistics should rely more on the discrete distribution, while infrequent words should rely more on their embeddings (pre-trained on a large external corpus). crucially, we do not assume any prior knowledge of the corpus used to train the word embeddings, and our parameterisation depends on the uncertainty of the continuous topic distributions at the current state of the markov chain during gibbs sampling. our contributions are as follows: gaussian lda @xcite proposes a variant of lda which operates on the continuous vector space rather than on discrete words. each word is represented by an m -dimensional vector @xmath0. that is, for k topics, there are k gaussian distributions. while there have been extensions to more complex continuous distributions such as von mises-fisher @xcite )), we opted to work with a simpler distribution to demonstrate the approach, which can subsequently be extended in future work. polylingual lda @xcite studies lda across more than two languages using parallel corpora. the model assumes that the documenttopic distribution θ d , is shared across languages, and that each language specific topic has a multinomial topic-word distribution, φ ℓ 1 , φ ℓ 2 due to the discrete nature of words. @xcite ; @xcite showed that polylingual topic models can infer topic structure in multilingual corpora. latent feature topic models a natural extension to discrete only or continuous only representations, is to model a word as being sampled with some probability from its discrete or continuous component. @xcite introduced an interpolation between the continuous and discrete representations, but convert the continuous representations back into discrete probability over word types by learning latent feature weights. ## discrete-continuous mixture lda we first establish an incremental extension to the latent feature topic model using mixture of discrete categorical and continuous gaussian distributions. we adopt a mixture model where each word has some probability of either coming from its categorical (discrete) or gaussian (continuous) distribution. the generative process for this model with k topics is as follows: ## computational complexity we consider the computational cost for every gibbs sampling step. the main source of computational complexity comes from inverting σ k which takes o(m 3 ) when computing the probability density of v d,i in row 2 of since the covariance matrix σ k is symmetric and positive semi-definite, we can utilise the cholesky decomposition where σ k can be decomposed as a product of upper and lower triangular matrices, σ @xmath1. although this takes o(m 3 ), we pay this cost only once during initialisation. l k is maintained by performing rank-1 updates and downdates @xcite at every step of collapsed gibbs sampling. as shown in @xcite , calculating the probability density takes o(m 2 ) instead of o(m 3 ). our proposed prior for β k sum the diagonals of l k which takes o(m ) with little to no constant time overhead. therefore each gibbs sampling step takes o(km 2 ) where k is the number of topics whose p(v d,i |z d,@xmath2, ϕ, x) we need to compute. this is parallelisable to o(m 2 ) as each term can be computed independently. ## cross-lingual adaptive lda the following section describes the extension of our work from the monolingual to the cross-lingual setting. to test the robustness of our proposed model and extensibility to other models, we study the topic coherence in multilingual settings where the quality of word embeddings is thought to be worse than monolingual embeddings. we introduce a new topic model for continuous multilingual representations building on our adaptive sampling scheme, cross-lingual adaptive lda in (figure 1 ). @xcite , we assume that the document-topic distribution θ d is shared across paired language documents, and follow a bag-of-words assumption, i.e., they need not be sentence or word-aligned. we additionally assume that multilingual word embeddings v ℓ 1 , v ℓ 2 have been mapped to the same embedding space, by adopting shared gaussian mean µ k and covariance σ k across languages. this reduces the number of parameters and importantly ensures a continuous mapping across languages. although this does not necessarily affect topic-coherence when measured within in each language, this would results in very poor cross-lingual document-topic representations. we checked this assumption by inspecting the learned topics without parameter sharing and found that topic indexes were mismatched across languages. topic 5 in english may be about sports but topic 5 in french may be about medicine. ## conclusion low resource scenarios present an interesting challenge to topic modeling due to sparsity of counts and a lack of data to train neural models. our work proposes an automatic trade-off between externally trained continuous representations and traditional co-occurrence count-based statistics that is specific to each word and topic. the method accounts for variations in number of topics and embedding dimensions, and requires no additional tuning beyond existing methods. importantly, it requires no additional retraining of word embeddings or learning of topic embeddings, allowing us to rely solely on pre-trained representations and existing corpus statistics. we showed the efficacy and extensibility of our approach on a monolingual and a multilingual dataset, while introducing a new cross-lingual adaptive lda topic model in the process. in future work, we aim to study the different scenarios of low resource (e.g., when there are a lot of infrequent words such as named entities) and their interaction with different embedding methods.