Topic Balancing with Additive Regularization of Topic Models (2020)

Abstract: This article proposes a new approach for building topic models on unbalanced collections in topic modelling, based on the existing methods and our experiments with such methods. Real-world data collections contain topics in various proportions, and often documents of the relatively small theme become distributed all over the larger topics instead of being grouped into one topic. To address this issue, we design a new regularizer for Theta and Phi matrices in probabilistic Latent Semantic Analysis (pLSA) model. We make sure this regularizer increases the quality of topic models, trained on unbalanced collections. Besides, we conceptually support this regularizer by our experiments.

## introduction topic modelling is a widespread approach to unsupervised text analysis and clustering. given the number of latent variables -topicstopic models extract hidden word×topic and topic×document probability distributions from text corpora. topic models have proven to be relevant in a wide range of contexts and uni-and multilingual tasks @xcite @xcite . two fundamental topic models are probabilistic latent semantic analysis -plsa @xcite and latent dirichlet allocation -lda @xcite . various extensions of plsa and lda models have emerged over the past years, e.g. additive regularization of topic models (artm) @xcite modification of plsa, where required solution properties are induced by the additional regularizer part in the model. through regularizers one can take into consideration various problem-specific features of data, and this is a reason why we apply artmframework in our work. despite almost 30 years of model development history, lots of problems and issues were raised in the topic modelling field. problem of the "order effect" in lda @xcite , for example. it consists in converging to the different topics set while during training on the unstructured data. even with the structured data solution in the plsa or lda model is non-unique and unstable. such unstability may be reduced by tuning the model with regularizers, as in the artm model. inserting φ and θ prior distribution into the model, according to the @xcite , promotes convergence to the better and stable solution along with regularization. however, many problems with models itself and with quality metrics remain unsolved. in this article, we point out the topic balancing problem. at this moment problem of training topic models on the unbalanced collections is not studied thoroughly and is far from the comprehensive solution. we examine previously suggested approach to the topic balancing and propose a balancing procedure, based on the a priori ratio between topic capacities. ## topic prior regularizer 4.1 description of the regularizer according to our experiments and modelling experience, log-likelihood functional optimization does not preserve topic balance in models and does not converge to the optimal solution from the user's point of view. we want an optimal solution to allow topics with relatively small topic capacities or topics with relatively small p(t|d) for the most of corpora documents. optimality in such terms can be achieved in a solution, where some topic variables, or degrees of freedom, are not fully utilized. current functional during the optimization via em-algorithm tends to redistribute p(t|d) in the most efficient way, without degenerate distributions. thus topic capacities obtain similar values during the training process. we formed the hypothesis from our experiments, that additional shift in tokens-topics φ may influence the em-algorithm as a restriction of the degrees of freedom, supporting topics imbalance. by setting relative collection balance in φ in advance, we can control possible collection balance after the training process. during the optimization, all ϕ wt are specified according to the tokens distribution in documents. we implemented this hypothesis in a new artm regularizer r t opicp rior called topicpriorregularizer with the parameter β to describe a priori topic balance in the collection. to better understand the r t opicp rior influence on the em-algorithm, we calculated the r t opicp rior partial derivative: and modified log-likelihood in case of one additional regularizer with regularization coefficient τ , determining regularizing strength: in most of the cases, we lack knowledge about topic capacities in the researched data collection, therefore we cannot set precise β value. we generalize our regularization approach and propose r t opicp riorsampled regularizer, where β parameter is being sampled from the dirichlet distribution with the parameter γ ∈ r 1 . γ is responsible for the estimated data sparsity, thus γ = 1 stands for the random topic capacities in a model, γ 1 stands for the equal topic capacities, γ 1 stands for the significantly uneven topic capacities. ## discussion and conclusion learning an unbalanced topic model from unbalanced text collection is a non-trivial task for all of the existing modelling methods. in this paper we discussed the problem of training topic models with unbalanced text collections. no previous research provides a thorough analysis of this problem or an efficient training procedure for unbalanced models. after reviewing the problem, we proposed an approach to building topic models, able to maintain relatively high imbalance degree. we described our approach in terms of plsa regularization and brought theoretical justification for the r t opicp rior regularizer.