Neural Attention-Aware Hierarchical Topic Model (2021)

Abstract: Neural topic models (NTMs) apply deep neural networks to topic modelling. Despite their success, NTMs generally ignore two important aspects: (1) only document-level word count information is utilized for the training, while more fine-grained sentence-level information is ignored, and (2) external semantic knowledge regarding documents, sentences and words are not exploited for the training. To address these issues, we propose a variational autoencoder (VAE) NTM model that jointly reconstructs the sentence and document word counts using combinations of bag-of-words (BoW) topical embeddings and pre-trained semantic embeddings. The pre-trained embeddings are first transformed into a common latent topical space to align their semantics with the BoW embeddings. Our model also features hierarchical KL divergence to leverage embeddings of each document to regularize those of their sentences, paying more attention to semantically relevant sentences. Both quantitative and qualitative experiments have shown the efficacy of our model in 1) lowering the reconstruction errors at both the sentence and document levels, and 2) discovering more coherent topics from real-world datasets.

## introduction topic models are a family of powerful techniques that can effectively discover human-interpretable topics from unstructured corpora for text analysis purposes. among them, bayesian topic models, based on the latent dirichlet allocation (lda) @xcite , have been the mainstream for nearly two decades. they usually adopt count/bag-ofwords (bow) representations for text content and model the generation of the bow data with a probabilistic structure of latent variables. these variables follow pre-specified distributions under bayes' theorem, and are learned through bayesian inference * corresponding author such as variational inference (vi) and monte carlo markov chain (mcmc) sampling. despite their success, conventional bayesian topic models, however, are known to lack flexibility in their model structures and scalability to large volumes of data. to address the above limitations, increasing effort has been made in leveraging deep neural networks (dnns) for topic modelling, which leads to the so-called neural topic models (ntms) @xcite . most of these models follow the framework of variational auto-encoders (vaes) @xcite and adopt an encoder-decoder architecture, in which the encoder transforms the bow data of each document into the corresponding documenttopical embeddings, and the decoder attempts to map these embeddings back to the same data. with a moderate increase in model complexity, ntms have largely outperformed conventional topic models on bow data reconstruction and topic interpretability/coherence @xcite @xcite @xcite . with this being said, most ntms only exploit the bow information of internal documents while ignoring (1) the sentence-level bow information of these documents, and (2) the external (semantic) information regarding the documents, sentences and words (e.g., extracted from other larger relevant corpora). these limitations have hindered the further performance improvement for ntms. therefore, in this paper, we propose a new ntm that address these limitations. it jointly reconstructs the bow data of each document and their sentences with combinations of both internal bow topical embeddings and external pre-trained semantic embeddings. to do this, we design an internal bow data encoder and an external knowledge encoder to respectively transform the bow data and the pretrained embeddings of the same documents and sentences into a shared latent topical space. the resulting internal and external topical embeddings are then combined to decode the bow data. to address the bow data sparsity @xcite at the sentence level, which has been a problem for many topic models, our model enforces hierarchical kl divergence on pairs of sentences and corresponding documents with respect to their topical embeddings derived from both the bow data and the external knowledge. the intent is that both types of topical embeddings for each sentence should be governed by the same types of embeddings of their parent documents. furthermore, the hierarchical kl for a sentence is weighted by its semantic relations to its parent document, so that a document's topical information is more influential to more semantically representative sentences. our contribution can be summarized as follows: ## related work to our best knowledge, most state-of-the-art ntms, like ours, are based on the vae framework. for a detailed discussion of these ntms, we refer readers to @xcite . here, we only discuss the two lines of research that are most relevant to ours. ntms with pre-trained language models recently, pre-trained transformer-based language models such as bert @xcite have started to draw the attention of the topic modelling community thanks to their capability of generating contextualized word embeddings with rich semantic information that is absent from the bow data. thus, an emerging trend focuses on incorporating these contextualized word embeddings into ntms. based on the popular vae framework of srivastava and sutton (2017), @xcite proposed a contextualized topic model (ctm) that incorporates pre-trained document embeddings generated by sentence-transformers @xcite . however, unlike ours, ctm ignores the other levels of pre-trained knowledge and bow information. @xcite proposed to combine an ntm with a fine-tuned bert model by concatenating the topic distribution and the learned bert embedding of a document as the features for document classification. hoyle et al. (2020) proposed bat, a ntm framework "taught" by external knowledge distilled from a pre-trained bert model. bert predicts probabilities for each word of a document which are then averaged to generate a pseudo bow vector for the document. the bat framework can be instantiated with various existing ntms such as scholar @xcite (i.e. bat+scholar), which imposes a logistic normal as the variational posteriors for document embeddings in the vae, and w-lda @xcite (i.e. bat+w-lda), which replaces the kl divergence with the maximum mean discrepancy @xcite . ntms for modelling document structures although document structures, i.e., the structured relationships between documents, paragraphs, and sentences have been modelled in conventional bayesian topic models (e.g., in @xcite ; balikas et al. (2016a); jiang et al. (2019 )), they have not been carefully studied in ntms to our knowledge. the closest work to our idea is @xcite , which proposes an ntm that samples a topic for each sentence of an input document and then generates the word sequence of the sentence with an rnn conditioned on the sentence's topic. however, this work focuses on sentence generation instead of topic modelling. ## preliminaries in this section, we introduce the vae framework of neural variational topic models, based on which our model will be developed. problem formulation consider a corpus that consists of i documents where the i th (1, ..., i) document is represented as a v -dimensional vector of word counts, w d i , also known as the bag-of-words (bow) data. here, v is the size of the vocabulary from the corpus, and w d iv is the number of times the v th (1, ..., v ) word occurs in the i-th document. topic modelling assumes that there exist k topics that can be used to describe each document. its goal is to recover these topics from the bow data symbols description i, v , k, l, ji, m size of corpus, vocabulary, topics, hidden neurons, number of sentences in document @xmath0, ..., i) and pre-trained embedding dimension n a vector of the lengths of each document the i th document and its j th sentence, (z @xmath1, w s i bow matrices over the i documents (z i×@xmath2, x s i , x d matrices of pre-trained non-contextual word embeddings (r v ×m ), pre-trained sentence embeddings (r j i ×m ) and document embeddings (r i×m ) q h (•) the vae encoder for @xmath3, "e"} that generate the topical embeddings of words, sentences and documents f h (•) a shared mlp of the encoder for h that maps r v to r l l h 1 (•), l h 2 (•) linear layers over the shared mlp of the encoder for h that maps r l to r k φ h word-topic embedding matrix over each word specific to the encoder for h, (r v×k ) s h ij , s h i sentence-topic embedding of the j th (1, ..., ji) sentence in document i generated by the encoder for h, (r k ); sentence-topic matrix, (r j i ×k ) z h i , z h topical embedding of document i generated by the encoder for h, (r @xmath4; document-topic matrix, (r i×@xmath5, (r) for computing a shared hidden layer output from w d i , where l is the number of hidden neurons. the functions l b 1 , l b 2 : r @xmath6: as for the nvtm's decoder network, it reconstructs w d using the document-topic matrix z b and the word-topic matrix φ b as follows: ## our model our proposed model, neural attention-aware hierarchical topic model (nahtm), comprises (1) two types of encoders, internal bow data encoder and external knowledge encoder, that capture latent topics of documents, sentences and words from both internal and external sources, respectively; ## experimental setup data we evaluate the efficacy of nahtm using four real-world corpora from a variety of domains, wikitext-103 @xcite , 20newsgroup @xcite , covid-19 open research dataset (cord) foot_1 and nips foot_2 datasets. we adopt the same training-validation-testing split ratios and preprocessing steps from the original papers for the wikitext-103 (i.e. 70%-15%-15%) and 20newsgroup (i.e. 48%-12%-40%) datasets. to construct their vocabularies for topic modelling, we follow the same stemming/lemmatization and stopword removing procedures of @xcite . as for the cord20k and nips datasets, we set the split ratio to be 60%-20%-20%, and use the most frequent 10,000 words (with stemming and stop-words removed) as the vocabulary. specifically, we apply wordnet lemmatizer and english stopword list (both from the nltk 5 toolkit) to preprocess their text. finally, we use the first 50 sentences of each document from the two datasets for the experiments. furthermore, to extract the sentence embedding from the external pre-trained transformer, we have not done any pre-processing (neither stemming/lemmatization nor stopword removing) on the sentence text for all the datasets, as required by the sentence-transformers library. doing so guarantees that the pre-trained language model can capture the full contextual information within each sentence. ## results and discussion following the settings from the previous section, we proceed to conduct both quantitative and qualitative experiments to evaluate nahtm. for the quantitative experiments, we report the results of the average external and internal npmi, and the average test perplexity over 5 runs with different random seeds for initialization. moreover, within each run, all the models were learned twice with 50 and 200 topics each time, and their corresponding metric scores have been summarized in it can be observed that nahtm, equipped with either the β 1 -customising strategy 1 or 2 from section 4.3 (i.e. nahtm s1 and nahtm s2 ), and with uncustomised hierarchical kl (i.e. nahtm hkl ), is overall more coherent, in terms of both types of npmi, than the baseline models and nahtm with independent kl terms for sentences and documents (i.e. nahtm kl ). especially when comparing with bat+{scholar,w-lda} and ctm, which also have leveraged external pre-trained knowledge, nahtm manages to achieve not only higher topic coherence but also lower perplexity on the test data in general. the only exception is on the nips dataset where nahtm follows bat+scholar closely on the perplexity. topics models topic words optimization bat+scholar problem, siam, np, solve, optimization, loss, min, consider, convex, hard nahtms2 loss, gradient, minimization, optimization, solution, min, convex, np, descent, objective nips neural bat+scholar layer, neural, deep, convolutional, network, gradient, architecture, visual, input, recurrent networks nahtms2 network, neural, layer, deep, backpropagation, recurrent, convolutional, descent, feedforward, gradient bat+scholar infection, covid, disease, virus, coronavirus, epidemic, h7n9, vaccine, patient, cases covid nahtms2 disease, covid, virus, infected, coronavirus, vaccine, influenza, patient, epidemic, cord respiratory 20k bat+scholar quarantine, school, lockdown, unemployed, study, control, province, holiday, social, city quarantine nahtms2 quarantine, lockdown, curfew, unemployment, fatality, distancing, holiday, social, province, inequity finally, to gain a more intuitive view on how well nahtm has discovered the underlying topics, we show in ## conclusion in this paper, we have proposed nahtm, a vaebased neural topic model with attention-aware hierarchical kl divergence imposed on the pairs of documents and sentences. nahtm incorporates both the internal bow data information and the external pre-trained knowledge for refining the topical embeddings of words, sentences and documents. both quantitative and qualitative experiments have shown the effectiveness of nahtm on 1) recovering the bow data at different levels of granularity and 2) discovering coherent topics, by making use of the hierarchical kl constraints on the sentencedocument pairs and the external knowledge. as for the future work, we would like to investigate the possibility of combining nahtm with neural language models for topic-aware language understanding and content generation.