Knowledge-Aware Bayesian Deep Topic Model
Anonymous Author(s)
AffiliationAddress
Abstract
We propose a Bayesian generative model for incorporating prior domain knowledgeinto hierarchical topic modeling.
Although embedded topic models (ETMs) and itsvariants have gained promising performance in text analysis, they mainly focus onmining word co-occurrence patterns, ignoring potentially easy-to-obtain prior topichierarchies that could help enhance topic coherence.
While several knowledgebased topic models have recently been proposed, they are either only applicableto shallow hierarchies or sensitive to the quality of the provided prior knowledge.
To this end, we develop a novel deep ETM that jointly models the documentsand the given prior knowledge by embedding the words and topics into the samespace.
Guided by the provided knowledge, the proposed model tends to discovertopic hierarchies that are organized into interpretable taxonomies.
Besides, witha technique for adapting a given graph, our extended version allows the providedprior topic structure to be finetuned to match the target corpus.
Extensive experiments show that our proposed model efficiently integrates the prior knowledge andimproves both hierarchical topic discovery and document representation.
Introduction
Probabilistic topic models (TMs) such as latent Dirichlet allocation
(LDA) have enjoyed success intext mining and analysis in an unsupervised manner [Blei et al., 2003] .
Typically, the goal of TMs isto infer per-document topic proportions as well as a set of latent topics from the target corpus usingword co-occurrences within each document.
The extracted topics are widely used in various machinelearning tasks [Kataria et al., 2011, Jin et al., 2018, Xu et al., 2018].
However, the objective functionof most TMs is to maximize the likelihood of the observed data which causes an over-concentrationon high-frequency words.
Moreover, the infrequent words might be assigned to irrelevant topics dueto the lack of side information.
Those two drawbacks could lead to human-unfriendly topics that failto make sense to end users in practice.
This issue is further exacerbated in hierarchical cases where alarge number of topics and their relevance need to be modeled [Meng et al., 2020, Duan et al., 2021a].
In many cases, users with prior domain knowledge are concerned with a specific topic structure,as shown in Fig.
Such a topic hierarchy provides semantic common sense among topics andwords, which can guide high-quality topic discovery.
Therefore, several researches have attempted toexploit various prior knowledge into topic modeling to improve topic representation.
For example,word correlation based topic models [Andrzejewski et al., 2009, Newman et al., 2011] use mustlinks and cannot-links to improve the interpretability of learned topics.
Word semantic methods[Jagarlamudi et al., 2012, Doshi-Velez et al., 2015] allow users to predefine a set of seed words thatcould appear together in a certain topic to improve topic modeling.
Furthermore, knowledge graphbased topic models [Hu et al., 2016, Yao et al., 2017] combine LDA with entity vectors to injectmulti-relational knowledge, achieving better semantic coherence.
Although the above mentionedmethods improve topic coherence in different ways, they only work with shallow topic models andignore the relationships between topics.
[Figure]
hardwarekeyboardmouse
[Figure]
More recently, embedded topic models (ETMs) and their hierarchical variants [Dieng et al., 2020, Duan et al., 2021a] employ distributed embedding vectors to encode semantic relationship, whichhave gained growing research interest due to their effectiveness and better flexibility.
Viewing wordsand topics as embeddings in a shared latent space makes it possible for ETMs to integrate priorknowledge in the form of pretrained word embeddings or semantic regularities [Dieng et al., 2020, Meng et al., 2020, Duan et al., 2021b].
The main idea behind those models is to constrain thetopic embeddings under the predefined topic structure.
Those models however are built on a strongassumption that the given topic hierarchy is well defined and matched to the target corpus, which isoften unavailable in practice.
Such mismatched structure might hinder the learning process and leadto sub-optimal results.
To address the above shortcomings, in this paper, we first propose TopicKG, a knowledge graphguided deep ETM that views topics and words as nodes in the predefined topic tree.
As a Bayesianprobabilistic model, TopicKG aims to model the document Bag-of-Word (BoW) vector and the giventopic tree jointly by sharing the embedding vectors.
Specifically, we first adopt a deep documentgeneration model to discover multi-layer document representations and topics.
To incorporate thehuman knowledge in the topic tree and guide the learning of topic hierarchy, a graph generativemodel is employed, which refines the node embeddings by exploring the belonging relations hiddenin topic tree, resulting in more interpretable topics.
Besides, to address the mismatch between thegiven topic tree and target corpus, we further extend TopicKG to TopicKGA, which adopts the graph
A daptive technique to explore the missing links in topic tree from the target corpora, achieving betterdocument representation.
The final revised topic tree is obtained by combining the prior structure ofthe given topic tree with the added structures learned from the current corpus.
Thus the topic tree inour TopicKGA has the ability to reinforces itself constantly according to the given corpora.
Finally,both the proposed models are based on variational autoencoders (VAEs) [Kingma and Welling, 2014]and can be trained by maximizing the evidence lower bound (ELBO), enjoying promising flexibilityand scalability.
The main contributions of this paper can be summarized as follows: 1) , A novel knowledge-awaredeep ETM named TopicKG is proposed to incorporate prior domain knowledge into hierarchicaltopic modeling by accounting for the document and the given topic tree in a Bayesian generativeframework.
2) , To overcome the drawback of the mismatch issue between the prior structure of theprovided topic tree and the current corpus, TopicKG is further extended to TopicKGA, which allowsto revise the prior structure to better represent the current document. 3) , Besides achieving promisingperformance on hierarchical topic discovery and document classification task, the proposed modelsgive good flexibility and efficiency, providing a strong baseline for knowledge-based TMs.
Related work
The recent related work has focused on two different knowledge forms, as described below.
Pre-trained linguistic knowledge Pre-trained language models such as BERT [Devlin et al., 2019]and GPT [Brown et al., 2020] have been successful in various natural language processing tasks.
Pre-trained on huge text, such models can serve as a powerful encoder that outputs contextualsemantic embeddings.
Behind this idea, combining a TM with BERT via knowledge distillation, Hoyle et al. [2020] treat BERT as the teacher model to improve the representation of topic models.
CombinedTM [Bianchi et al., 2021] concatenates the Bag-of-Word (BoW) vector with the BERTcontextual embedding as the final input of a TM, providing a consistent increase in topic coherence.
Despite their improvement, those BERT-based TMs focus on the document latent representation, theyignore the relationship between words and topics.
Moreover they require a large amount of memoryto load the pre-trained BERT.
Pre-defined structural knowledge Concept hierarchies are key knowledge sources, and can beobtained easily from knowledge graph ( e.g. , WordNet [Miller, 1995]). Few generative models havebeen developed to hierarchical topic mining task. For example, Meng et al.
[2020] use the categorytree (usually with simple and shallow structure) as the supervised information and preserve suchrelative hierarchical structure in the spherical embedding space to explore user interested topicstructure.
Duan et al.
[2021b] view words and topics as the Gaussian distributions and employ theasymmetrical Kullback–Leibler (KL) divergence to measure the directional similarity in the giventopic hierarchy.
However, computing such KL divergence for each pair is time-consuming thatlimits its application to large-scale knowledge.
Moreover, those knowledge-based models ignore themismatch problem between the given topic tree and the current corpus, leading to sub-optimal resultsin practice.
The proposed model
To incorporate human knowledge into deep TMs, we first propose TopicKG that generates the targetcorpus and the given topic tree in the Bayesian manner.
We further extend TopicKG to its adaptiveversion TopicKGA that allows the topic structure to be refined according to the current corpus,resulting in a better document representation.
Below we introduce the details of our proposed model.
3.1 Problem formulation
Consider a corpus containing J documents with a vocabulary of V unique tokens.
Each documentis represented by the BoW vector x ∈ R V + , where x v denotes the number of occurrences of the v thword.
Unlike other TMs that are purely data-driven, TopicKG intends to inject the knowledge graphto improve topic quality.
Specifically, we introduce a topic tree T as our prior knowledge, whereeach node e i ∈ T denotes a word or a topic.
For each topic node, there is a corresponding definitionas shown in Fig.
Suppose there are L + 1 layers in the topic tree with the bottom word layer and L topic layers, and there are K l nodes T l : { e ( l ) 1 , ..., e ( l ) K l } at the l -th layer, where K 0 = V .
Generally,for the k -th topic node at the l -th layer e ( l ) k , we use S ( e ( l ) k ) to denote the set of its child nodes, and use C ( e ( l ) k ) to denote its key words extracted from the definition sentence.
Mathematically, we adopt thebinary matrix S ( l ) = { 0 , 1 } K l − 1 × K l and C ( l ) = { 0 , 1 } V × K l , l = 1 , ..., L , to denote the belongingrelations between two adjacent layers and the links between the topics and their concept words,respectively.
Guided by this pre-specified topic tree, TopicKG expects to retrieve hierarchical topicsfrom the target corpus that provide a clear taxonomy for the end users.
3.2 Knowledge-aware Bayesian Deep Topic Model
Given the corpus D and topic tree T , we aim to discover hierarchical document representationsand topic hierarchy by jointly modeling the BoW vector x , S ( l ) and C ( l ) , l = 1 , ...
L in a Bayesianframework.
Specifically, we employ the gamma belief network (GBN) of Zhou et al. [2015a] tomodel the count vector, and use the semantic similarity between two node embeddings to generate the topic tree T .
The whole generative model with L layers can be expressed as:
[Equation]
where x j is generated as in Poisson factor analysis [Zhou et al., 2012], in which x j is factorized asthe product of the factor loading matrix (topics) Φ (1) ∈ R K {+
+} × K 1 + and the gamma distributed factorscores (topic proportions) θ (1) j ∈ R K 1 + under the Poisson likelihood; Then the GBN [Zhou et al. ,2015b] is applied to explore multi-layer document representations by stacking hierarchical prior inthe topic proportions θ j .
The topics at each layer Φ ( l ) , l = 1 , ..., L is calculated by the semanticsimilarity ( e.g. , the inner product) between corresponding topic embeddings, followed by a Softmaxlayer to satisfy the probability simplex: (cid:80) {+K l − 1 v =
+}
Φ ( l ) vk = 1 .
e ( l ) k ∈ R
d is the embedding vector of k -thtopic at l -th layer ( k -th node at l -th layer e lk in T ), d is the embedding dimension.
σ ( · ) is the sigmoidfunction and Bern ( · ) denotes the Bernoulli distribution that is employed to model the two edge types S ( l ) and C ( l ) .
As suggested in Meng et al. [2020] and Duan et al. [2021b], we use asymmetry tocapture the directed relations between two adjacent layers in S ( l ) , and use symmetry to capture thesemantic similarity between topics and its concept words.
W is a learnable parameter to guaranteethe directed structure.
Under the Bernoulli likelihood, our TopicKG has the ability to incorporatethe topic hierarchical relations in T into topic modeling by sharing word and topic embeddings withΦ .
In particular, for a node pair in T , the embedding semantics of the source and destination nodesneed to be similar enough to determine whether there is an edge, which will guide the learning of Φ ,resulting in more coherent topics and some appealing model properties as described below.
The flexibility of human knowledge incorporation: As mentioned above, guided by the predefined topic tree, TopicKG models x , S ( l ) and C ( l ) , l = 1 , ...
L jointly, making it possible for thelearned hierarchical topics that not only satisfy the interpretable taxonomy structure but also preventincoherent issues with the help of side information.
Besides, the incorporation of S is simple andflexible.
First of all, as shown in Eq. 1 and Sec. 4, the plug-and-play module of S can be appliedto most of ETMs without changing their model structures, which provides a convenient alternativefor introducing side information to TMs.
Secondly, the pre-defined matrix S can be constructedin various ways so as to encode our beliefs about the graph structure, e.g. , the knowledge graph,taxonomy, and hierarchical label tree [Yao et al., 2017, Hu et al., 2016, Meng et al., 2020].
The relationship between { Φ ( l ) } Ll =1 and { S ( l ) } Ll =1 : The topic distribution matrix Φ ( l ) and thesymmetry matrix S ( l ) are related but play different roles.
The sparse structure matrix S ( l ) isconstructed from the topic tree which will be used to guide the learning of the hierarchy of topics,while the learned topics Φ ( l ) contains more specific knowledge extracted from the text corpus, andcan be viewed as the dense version of S ( l ) .
In other words, the topics learned from TopicKG retainsthe semantic structure of S ( l ) and enriches itself by the current corpora.
3.3 Encoder
The encoder of the proposed model is built around two main components: Weibull upward-downwardvariational encoder and the GCN-based topic aggregation module.
Weibull upward-downward variational encoder:
To approximate the posterior of the topic proportions { θ ( l ) j } , like most of VAE-based methods, we define the variational distribution q ( θ j | x j ) , whichcan be further factorized as [Vahdat and Kautz, 2020]:
[Equation]
In practice, it first obtains the latent feature by feeding the input x j into a residual upward neuralnetworks: h ( l ) j = h ( l − 1)
j + f ( l ) W h ( h ( l − 1) j ) , where l = 1 , ..., L , h (0)
j = x j , f ( l ) W h ( · ) is a two layer fullyconnected network parameterized by W ( l ) h .
To complete the variational distribution, we adopt the Weibull downward stochastic path [Zhang et al., 2018]:
[Equation]
where ⊕ denotes the concatenation at topic dimension, and we use the Softplus to make surethe positive Weibull shape and scale parameters.
f is a single layer fully connected network,correspondingly.
The Weibull distribution is chosen mainly because it is reparameterizable and the
KL divergence from the gamma to Weibull distributions has an analytic expression [Zhang et al.
,2018, Fan et al., 2020].
[Figure]
GCN-based topic aggregation: Attracted bythe excellent ability of graph neural network(GCN)
[Kipf and Welling, 2016a,b] in propagating graph structure information, and the predefined topic tree can be viewed as a directedgraph.
We thus construct a deterministic aggregating module for node embeddings with GCN:
[Equation]
where T is the number of GCN layer,
E (0) ∈
R d × N is the node embedding matrix, and N =(cid:80)
Ll =
K l .
˜
A
= D − 12 AD − 12 is the normalized adjacent matrix with degree matrix D , e.g. ,
D ii =
(cid:80) Nj =
1 A ij , A ∈ { 0 , 1 } N × N is the adjacent matrix of the topic tree T .
Unlike previous ETMs that view embeddings as independentlearnable parameters, the GCN module in TopicKG updates the node embeddings by considering their child-level topics and related words, resultingin more meaningful topic embeddings.
We calculate Φ via the updated node embeddings:
[Equation]
3.4 TopicKG with adaptive structure
One of the main assumptions in the previous section is that the pre-defined topic tree is helpful for thecurrent corpus and the corresponding edges are highly reliable.
However, this is generally unrealisticin practical applications, as T may be (i) noisy, (ii) built on an ad hoc basis, (iii) not closely related tothe topic discovering task [Elinas et al., 2020, Chen et al., 2020].
Consequently, we further propose TopicKGA that overcomes the above mismatching issue and revises the structure of the given topictree based on the corpus at hand.
In detail, TopicKGA first randomly initializes a learnable node embedding dictionaries E A ∈ R d × Nfor all nodes in T .
Then an adaptive graph A ada is generated based on E A using the certain kernelfunction k
: R d × R d −→ R :
[Equation]
Here, we choose the consine similarity to define our kernel function: k ( E A , E A ) =
E A E TA ||
E
A |||| E A
|| , and
Softmax function is used to normalize the adaptive matrix.
Note that, instead of generating A ada , wedirectly generate its normalized version to avoid unnecessary calculations [Bai et al., 2020].
Duringtraining, E A will be updated automatically to learn the hidden dependencies which are ignored by A .
Following Yu et al.
[2020], the final revised adjacency matrix is formed as (here we still use A todenote the revised graph for convenience):
[Equation]
which will enhance the GCN in Eq.
3 by replacing the adjacency matrix with the revised one.
Training
As a deep ETM, TopicKG intends to learn the latent document-topic distribution and the deterministichierarchical topic embeddings.
Like other ETMs, the posterior of the topic proportions and topicembeddings are intractable to compute.
We thus derive an efficient algorithm for approximating theposterior with amortized variational inference [Blei et al., 2017], which makes the proposed modelflexible for downstream task.
The resulting algorithm can either use pre-trained word embeddings, ortrain them from scratch.
The EBLO of the proposed model can be expressed as:
[Equation]
It consists three main parts: the expected log-likelihood of x (first term), the concept structurelog-likelihood (the two middle terms), and the KL divergence from prior p ( θ ( l ) j ) to q ( θ ( l ) j ) .
The graphweight β denotes the belief about the predefined topic tree.
Notably, the two middle terms distinguishthe proposed models from previous deep ETMs.
On the one hand, it acts as a regularization thatconstrains the embedding vectors to conform to the provided prior structure, and on the other hand, itprovides an alternative for ETM to introduce side information to improve its interpretability.
Annealed training for TopicKGA:
In TopicKGA, the structure of the revised graph changes duringthe training.
To incorporate the new edges that are inferred from the target corpus, we develop anannealed training algorithm for TopicKGA.
Specifically, we update S ( l ) and C ( l ) in Eq.
7 every Miterations:
[Equation]
where s is the threshold, and ˜ A ( l ) is the corresponding sub-block in ˜ A .
As E A becomes stable, thestructure of the graph will converge to a good blueprint that balances the prior graph and the currentcorpus.
We summarize all training algorithm in Appendix.
Experiment
In this section, we conduct extensive experiments on several benchmark text datasets to evaluatethe performance of the proposed models against other knowledge-based TMs, in terms of topicinterpretability and document representations.
5.1 Corpora
Our experiments are conducted on four widely-used benchmark text datasets, varying in scale.
Thedatasets include 20 Newsgroups (20NG) [Lang, 1995], Reuters extracted from the Reuters-21578dataset, R8, and Reuters Corpus Volume 2 (RCV2) [Lewis et al., 2004], where 20NG, R8 and RCV2are corpus that associate with the document label.
R8 is a subset of Reuters that collected from 8different classes.
For the multi-label RCV2 dataset, we follow previous works [Miyato et al., 2017] inwhich documents with a single label at the second level topics are left, resulting in 0.1M documentstotally.
Both R8 and RCV2 are already pre-processed.
For other two datasets, we tokenize and cleantext by excluding standard stop words and low-frequency words appearing less than 20 times [Yaoet al., 2019]. The statistics of the preprocessed datasets are summarized in Appendix.
5.2 Baselines
To demonstrate the effectiveness of incorporating human knowledge into deep TMs, we considerseveral baselines for a fair comparison, including representative ETMs and recent knowledge-basedtopic models, described as follows: 1) , ETM [Dieng et al., 2020], the first neural embedded topicmodel that assumes topics and words live in the same embedding space.
The topic distributions in ETM are calculated by the inner product of the word embedding matrix and the topic embedding.2) , CombinedTM [Bianchi et al., 2021], a BERT-based neural topic model that uses the pre-trained BERT as its contextual encoder.
We use it as our BERT baseline. 3) , SawETM [Duan et al., 2021a], ahierarchical ETM that employs the Poisson and gamma distributions to model the BoW vector and thelatent representation, respectively. 4) , JoSH
[Meng et al., 2020], a knowledge-based hierarchical topicmining method that uses category hierarchy as the side information and employs the EM algorithmto learn the spherical tree and text embedding. 5) , TopicNet [Duan et al., 2021b], a semantic graphguided topic model that views the words and topics as the Gaussian distributions and uses the KLdivergence to regularize the structure of the pre-defined tree.
For all baselines, we employ theirofficial codes and default settings obtained from their release repositories.
5.3 External knowledge
Since the learning of TopicKG involves a pre-defined topic tree, some generic knowledge graphssuch as WordNet [Miller, 1995] provide a convenient way to bring in external knowledge.
WordNetis a large lexical database that groups semantically similar words into synonym sets, these sets arefurther linked by the hyponymy relation.
For example, in WordNet the category furniture includesbed, which in turn includes bunkbed.
With these relations, the topic structure can be easily definedaccording to certain heuristic rules.
Specifically, for each dataset we first get the word intersectionof the vocabulary and WordNet, the chosen words are then considered as leaf nodes of the topichierarchy.
Afterwards, each leaf node can continuously find its ancestor nodes based on hyponymyrelations and finally all leaf nodes converge to the same root node, resulting in an adaptive topicstructure.
5.4 Settings
For all experiments, we set the embedding dimension as d = 50 , the knowledge confidence hyperparameter as β = 50 . 0 , the threshold as s = 0 . 4 .
Empirically, we find that the above settings work well for all datasets, we refer readers to the appendix for more detailed analysis.
We initialize thenode embedding from the Gaussian distribution N (0 , 0 . 02) .
We set the batch size as 200 and usethe AdamW
[Loshchilov and Hutter, 2019] optimizer with learning rate 0.01.
We generate a 7-layertopic hierarchy for each dataset with the method described in Sec. 5.3.
From the top to bottom, thenumber of topics at each layer are: 20NG: [1,2,11,66,185,277,270]; R8: [1,2,11,77,287,501,547];
RCV2: [1,2,11,68,263,469,562]; Reuters: [1,2,11,78,300,526,584].
For the single-layer methods(ETM and CombinedTM), we set the number of topics as the same of the first layer of deep models.
For all methods, we run the algorithms in comparison five times by modifying only the random seedsand calculate the mean and standard deviation.
All experiments are performed on an Nvidia {+RTX
 Ti+} GPU and our proposed models are implemented with PyTorch.
[Table]
5.5 Topic interpretability
Generally, topic models are evaluated based on perplexity.
However, perplexity on the held-out testis not an appropriate measure of the topic quality and sometimes can even be contrary to humanjudgements [Chang et al., 2009, Yao et al., 2017].
To this end, we instead adopt three commonmetrics, including Topic coherence ( TC ), Topic diversity ( TD ) and word embedding topic coherence( WE ), to evaluate the learned topics from various aspects [Dieng et al., 2020, Bianchi et al., 2021].
TC measures the average Normalized Pointwise Mutual Information (NPMI) over the top 10 wordsof each topic, and a higher score indicates more interpretable topics.
TD denotes the percentageof unique words in the top 25 words of the selected topics.
WE, as its name implies, provides anembedded measure of how similar the words in a topic are.
WE is calculated by the average pairwisecosine similarity of the word embeddings of the top-10 words in a topic.
We report the topic qualityresults at the first five layer in Fig.
We here focus on topic hierarchy and ignore the single layermethods.
We find that: 1) , Knowledge-based methods including JoSH, CombinedTM, TopicNet andour proposed models are generally better than other likelihood-based ETMs, which illustrates thebenefit of incorporating side information; 2) , Our proposed TopicKG and TopicKGA achieve higherperformance than other knowledge-based models in most cases, especially on TC and WE tasks,which means our proposed model prefer to mine more coherent topics while achieving comparable
TD.
We attribute this to the joint topic tree likelihood that provides an efficient alternative to integratehuman knowledge into ETMs. 3) , Compared to TopicKG, its adaptive version TopicKGA discoversbetter topics at higher layers.
It is not surprise that TopicKGA gives greater robustness and flexibilityby refining the topic structure according to the current corpus.
5.6 Document classification
Besides the topic quality evaluation, we also use document classification to compare the extrinsicpredictive performance.
In detail, we collect the inferred topic proportions, e.g. , θ (1) , and than applylogistic regression to predict the document label.
We report the Micro F1 and Macro F1 score on20NG, R8 and RCV2 datasets in Table.
Overall, the proposed models outperform the baselineson all corpora, which confirms the effectiveness of our innovation of combining human knowledge [Figure] and ETMs in improving document latent representations.
Moreover, with the revised topic structurefine-tuned to the current corpus, TopicKGA surpasses other knowledge graph fixation methods withsignificant gaps.
This conclusion is consistent with one of our motivations that the predefined concepttree may not match the target corpus, leading to suboptimal document representations.
5.7 Qualitative analysis
We visualize the learned topic hierarchies of TopicKG and
TopicKGA on 20NG in Fig.
4(a-b), respectively.
Eachtopic box consists of the pre-specified concept name onthe top bar and its keywords listed in the bottom content.
We can observe that 1) , The mined keywords are highlyrelevant to the corresponding topics, providing a cleandescription of their concepts. 2) , Guided by human knowledge, the connections between topics at two adjacent layerare highly interpretable, resulting in human-friendly topictaxonomies.
3) , More interestingly, to further verify theadaptive ability of our proposed TopicKGA, we manuallyadded several free topics (dashed boxes in Fig. 4(b)) toeach layer of the given topic tree, which can be viewed asthe missing topics in the predefined knowledge and needto be learned from the target corpus.
We find that our TopicKGA indeed has the ability to capture themissing concepts, and revise the prior graph to match the current corpus.
5.8 Time efficiency
To demonstrate the time efficiency of incorporating human knowledge into TMs, we run TopicKG,
TopicKGA and TopicNet (as they are implemented in PyTorch, JoSH is in C) on 20NG datasetwith a 7-layer topic tree.
Fig.
5 shows the Negative log-likelihood of documents x , which showsthat the proposed models not only have faster learning speed than TopicNet, but also achieve betterreconstruction performance.
This result illustrates the efficiency of the introduced graph likelihoodwhich is important in real-time applications.
Conclusion
We develop an efficient Bayesian probabilistic model that integrates pre-specified domain knowledgeinto hierarchical ETMs.
The core idea is the joint document and topic tree modeling with the sharedword and topic embeddings.
Besides, with the graph adaptive technique, TopicKGA has the abilityto revise the given prior structure according to the target corpus, enjoying robustness and flexibilityin practice.
Extensive experiments show that the proposed models outperform competitive methodsin term of both topic quality and document classification task.
Moreover, thanks to the efficientknowledge incorporation algorithm, our proposed models achieve faster learning speed than otherknowledge-base models.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Please refer to the appendix
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Pleaserefer to the appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] The code and
data are included in the supplementary material.
(b) Did you specify all the training details (e g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]