TAN - NTM : T opic Attention Networks for Neural Topic Modeling (2021)

Abstract: Topic models have been widely used to learn text representations and gain insight into document corpora. To perform topic discovery, most existing neural models either take document bag-of-words (BoW) or sequence of tokens as input followed by variational inference and BoW reconstruction to learn topic-word distribution. However, leveraging topic-word distribution for learning better features during document encoding has not been explored much. To this end, we develop a framework TAN-NTM, which processes document as a sequence of tokens through a LSTM whose contextual outputs are attended in a topic-aware manner. We propose a novel attention mechanism which factors in topic-word distribution to enable the model to attend on relevant words that convey topic related cues. The output of topic attention module is then used to carry out variational inference. We perform extensive ablations and experiments resulting in ~9-15 percentage improvement over score of existing SOTA topic models in NPMI coherence on several benchmark datasets - 20Newsgroups, Yelp Review Polarity and AGNews. Further, we show that our method learns better latent document-topic features compared to existing topic models through improvement on two downstream tasks: document classification and topic guided keyphrase generation.

## introduction topic models @xcite have been popularly used to extract abstract topics which occur commonly across documents in a corpus. each topic is interpreted as a group of semantically coherent words that represent a common concept. in addition to gaining insights from unstructured texts, topic models have been used in several tasks of practical importance such as learning text representations for document classification @xcite , keyphrase extraction @xcite , understanding reviews for e-commerce recommendations @xcite , semantic similarity detection between texts @xcite etc. early works on topic discovery include statistical methods such as latent semantic analysis @xcite , latent dirichlet allocation (lda) @xcite which approximates each topic as a probability distribution over word vocabulary (known as topic-word distribution) and performs approximate inference over documenttopic and topic-word distributions through variational bayes. this was followed by markov chain monte carlo (mcmc) @xcite based inference algorithm -collapsed gibbs sampling (griffiths @xcite . these methods require an expensive iterative inference step which has to be performed for each document. this was circumvented through introduction of deep neural networks and variational autoencoders (vae) @xcite , where variational inference can be performed in single forward pass. neural variational inference topic models @xcite @xcite commonly convert a document to bagof-words (bow) determined on the basis of frequency count of each vocabulary token in the document. the bow input is processed through an mlp followed by variational inference which samples a latent document-topic vector. a decoder network then reconstructs original bow using latent document-topic vector through topic-word distribution (twd). vae based neural topic models can be categorised on the basis of prior enforced on latent document-topic distribution. methods such as nvdm @xcite , ntm-r @xcite , nvdm-gsm @xcite use the gaussian prior. nvlda and prodlda @xcite use approximation to the dirichlet prior which enables model to capture the fact that a document stems from a sparse set of topics. however, improving document encoding in topic models in order to capture document distribution and semantics better has not been explored much. in this work, we build upon vae based topic model and propose a novel framework tan-ntm: topic attention networks for neural topic modeling which process the sequence of tokens in input document through an lstm @xcite whose contextual outputs are attended using topic-word distribution (twd). we hypothesise that twd (being learned by the model) can be factored in the attention mechanism @xcite to enable the model to attend on the tokens which convey topic related information and cues. we perform separate attention for each topic using its corresponding word probability distribution and obtain the topic-wise context vectors. the learned word embeddings and twd are used to devise a mechanism to determine topic weights representing the proportion of each topic in the document. the topic weights are used to aggregate topic-wise context vectors. the composed context vector is then used to perform variational inference followed by the bow decoding. we perform extensive ablations to compare tan-ntm variants and different ways of composing the topicwise context vectors. for evaluation, we compute commonly used npmi coherence @xcite which measures the extent to which most probable words in a topic are semantically related to each other. we compare our tan-ntm model with several state-of-the-art topic models (statistical @xcite , neural vae @xcite and non-variational inference based neural model @xcite ) outperforming them on three benchmark datasets of varying scale and complexity: 20newsgroups (20ng) @xcite , yelp review polarity and agnews @xcite . we verify that our model learns better document feature representations and latent document-topic vectors by achieving a higher document classification accuracy over the baseline topic models. further, topic models have previously been used to improve supervised keyphrase generation @xcite . we show that tan-ntm can be adapted to modify topic assisted keyphrase gener-ation achieving sota performance on stackexchange and weibo datasets. our contributions can be summarised as: development of neural networks has paved path for variational autoencoders (vae) (kingma and welling, 2013) which enables performing variational inference (vi) efficiently. the vae-based topic models use a prior distribution to approximate the posterior for latent document-topic space and compute the evidence lower bound (elbo) using the reparametrization trick. since our work is based on variational inference, we use prodlda and nvlda @xcite as baselines for comparison. the dirichlet distribution has been commonly considered as a suitable prior on the latent document-topic space since it captures the property that a document belongs to a sparse subset of topics. however, in order to enforce the dirichlet prior, vae methods have to resort to approximations of the dirichlet distribution. several works have proposed solutions to impose the dirichlet prior effectively. @xcite enforces dirichlet prior using vi without reparametrization trick through word-level topic assignments. some works address the sparsitysmoothness trade-off in dirichlet distribution by factoring dirichlet parameter vector as a product of two vectors @xcite . wasserstein autoencoders (wae) @xcite have led to the development of non-variational inference based topic model: wasserstein-lda (w-lda) which minimizes the wasserstein distance, a type of optimal transport (ot) distance, by leveraging distribution matching to the dirichlet prior. we compare our work with w-lda as a baseline. @xcite proposed an ot based topic model which directly calculates topic-word distribution without a decoder. adversarial topic model (atm) @xcite was proposed based on gan (generative adversarial network) @xcite but it cannot infer document-topic distribution. a major advantage of w-lda over atm is distribution matching in document-topic space. bidirectional adversarial topic model (bat) @xcite employs a bilateral transformation between document-word and document-topic distribution, while @xcite uses cyclegan @xcite for unsupervised transfer between documentword and document-topic distribution. hierarchical topic models @xcite utilize relationships among the latent topics. supervised topic models have been explored previously where the topic model is trained through human feedback @xcite or with a task specific network simultaneously such that topic extraction is guided through task labels @xcite . @xcite leverages document metadata but without metadata their method is same as prodlda which is our baseline. topic modeling on document networks has been done leveraging relational links between documents @xcite . however our problem setting is completely different, we extract topics from documents in unsupervised way where document links/metadata/labels either don't exist or are not used to extract the topics. some very recent works use pre-trained bert @xcite either to leverage improved text representations @xcite or to augment topic model through knowledge distillation ( hoyle et al., 2020a). zhu et al. (2020) and dieng et al. (2020) jointly train words and topics in a shared embedding space. however, we train topic-word distribution as part of our model, embed it using word embeddings being learned and use resultant topic embeddings to perform attention over sequentially processed tokens. idocnade @xcite is an autoregressive topic model for short texts utilizing pre-trained embeddings as distributional prior. however, it attains poorer topic coherence than prodlda and gnb-ntm as shown in @xcite . some works have attempted to use other prior distributions such as @xcite uses the weibull prior, @xcite uses the beta distribution. gamma negative binomial-neural topic model (gnb-ntm) @xcite is one of the recent neural variational topic models which attempt to combine vi with mixed counting models. mixed counting models can better model hierarchically dependent and over-dispersed random variables while implicitly introducing nonnegative constraints in topic modeling. gnb-ntm uses reparameterization of gamma distribution and gaussian approximation of poisson distribution. we use their model as a baseline for our work. topic models have been used with sequence encoders such as lstm in applications like user activity modeling @xcite . @xcite employs an rnn to detect stop words and merges its output with document-topic vector for next word prediction. @xcite uses a vae pre-trained through topic modeling to perform text classification. we perform document classification and compare our model's accuracy with the accuracy of vae based and other topic models. ltmf @xcite ) combines text features processed through an lstm with a topic model for review based recommendations. fundamentally different from these, we use topic-word distribution to attend on sequentially processed tokens via novel topic guided attention for performing variational inference, learning better document-topic features and improving topic modeling. a key application of topic models is supervised keyphrase generation. some of the existing neural keyphrase generation methods include seq-tag @xcite based on sequence tagging, seq2seq-corr @xcite based on seq2seq model without copy mechanism and seq2seq-copy @xcite which additionally uses copy mechanism. topic-aware keyphrase generation (takg) @xcite is a seq2seq based neural keyphrase generation framework for social media language. takg uses a neural topic model in @xcite and a keyphrase generation (kg) module which is conditioned on latent document-topic vector from the topic model. we adapt our proposed topic model to takg to improve keyphrase generation and discuss it in detail later in the experiments section. 3 background lda is a generative statistical model and assumes that each document is a distribution over a fixed number of topics (say k) and that each topic is a distribution of words over the entire vocabulary. lda proposes an iterative process of document generation where for each document d, we draw a topic distribution θ from dirichlet(α) distribution. for each word in d at index i, we sample a topic t i from m ultinomial(θ) distribution. w i is sampled from p(w i |t i , β) distribution which is a multinomial probability conditioned on topic t i . given the document corpus and the parameters α and β, we need the joint probability distribution of a topic mixture θ, a set of k topics t, and a set of n words w. this is given analytically by an intractable integral. the solution is to use variational inference wherein this problem is converted into an optimization problem for finding various parameters that minimize the kl divergence between the prior and the posterior distribution. this idea is leveraged at scale by the use of variational autoencoders. the encoder processes bow vector of the document x bow by an mlp (multi layer perceptron) which then forks into two independently trainable layers to yield z µ & z log σ 2 . then a re-parametrization trick is employed to sample the latent vector z from a logistic-normal distribution (resulting from an approximation of dirichlet distribution). this is essential since backpropagation through a sampling node is infeasible. our methodology improves upon the document encoder and introduces a topic guided attention whose output is used to sample z. we use the same formulation of decoder as used in prodlda. ## methodology in this section, we describe the details of our framework where we leverage the topic-word distribution to perform topic guided attention over tokens in a document. given a collection c with |c| documents {x 1 , x 2 , .., x |c| }, we process each document x into bow vector x @xmath0, where v represents the vocabulary. as shown in step a in figure 1 @xcite . the embedded sequence {e j } |x| @xmath1, where |x| is the number of tokens in x, is processed through a sequence encoder lstm @xcite to obtain the corresponding hidden states h @xmath2: ## conclusion in this work, we propose topic attention network based neural topic modeling framework: tan13 we use their code and data (link) to conduct experiments. ntm to discover topics in a document corpus by performing attention on sequentially processed tokens in a topic guided manner. attention is performed effectively by factoring topic-word distribution (twd) into attention mechanism. we compare different variants of our method through ablations and conclude that processing tokens sequentially without attention or applying attention without twd gives inferior performance. our tan-ntm model generates more coherent topics compared to state-of-the-art topic models on several benchmark datasets. our model encodes better latent document-topic features as validated through better performance on document classification and supervised keyphrase generation tasks. as future work, we would like to explore our framework with other sequence encoders such as transformers, bert etc. for topic modeling.