Knowledge-Aware Bayesian Deep Topic Model
Author(
Abstract
We propose a Bayesian generative model for incorporating prior domain knowledgeinto hierarchical topic modeling.
Although embedded topic models (ETMs) and itsvariants have gained promising performance in text analysis, they mainly focus onmining word co-occurrence patterns, ignoring potentially easy-to-obtain prior topichierarchies that could help enhance topic coherence.
While several knowledgebased topic models have recently been proposed, they are either only applicableto shallow hierarchies or sensitive to the quality of the provided prior knowledge.
To this end, we develop a novel deep ETM that jointly models the documentsand the given prior knowledge by embedding the words and topics into the samespace.
Guided by the provided knowledge, the proposed model tends to discovertopic hierarchies that are organized into interpretable taxonomies.
Besides, witha technique for adapting a given graph, our extended version allows the providedprior topic structure to be finetuned to match the target corpus.
Extensive experiments show that our proposed model efficiently integrates the prior knowledge andimproves both hierarchical topic discovery and document representation.
Introduction
Probabilistic topic models (TMs) such as latent Dirichlet allocation
(LDA) have enjoyed successin text mining and analysis in an unsupervised manner [4] .
Typically, the goal of TMs is to inferper-document topic proportions as well as a set of latent topics from the target corpus using wordco-occurrences within each document.
The extracted topics are widely used in various machinelearning tasks [20, 19, 32].
However, the objective function of most TMs is to maximize the likelihoodof the observed data, which causes an over-concentration on high-frequency words.
Moreover, theinfrequent words might be assigned to irrelevant topics due to the lack of side information.
Those twodrawbacks could lead to human-unfriendly topics that fail to make sense to end users in practice.
Thisissue is further exacerbated in hierarchical cases where a large number of topics and their relevanceneed to be modeled [27, 12].
In many cases, users with prior domain knowledge are concerned with a specific topic structure, asshown in Fig.
Such a topic hierarchy provides semantic common sense among topics and words,which can guide high-quality topic discovery.
Therefore, several researches have attempted to exploitvarious prior knowledge into topic modeling to improve topic representation.
For example, wordcorrelation based topic models [1, 30] use must-links and cannot-links to improve the interpretabilityof learned topics.
Word semantic methods [18, 11] allow users to predefine a set of seed words thatcould appear together in a certain topic to improve topic modeling.
Furthermore, knowledge graphbased topic models [17, 33] combine LDA with entity vectors to inject multi-relational knowledge,achieving better semantic coherence.
Although the above mentioned methods improve topic coherencein different ways, they only work with shallow topic models and ignore the relationships betweentopics. [Figure] baseballvolleyball [Figure]
softwarehardwaremonitorcolor windows carbikegashelmets
[Figure]
softwareinstallwindowscomputer
[Figure]
hardwarekeyboardmouse [Figure] More recently, embedded topic models (ETMs) and their hierarchical variants [10, 12] employdistributed embedding vectors to encode semantic relationship, which have gained growing researchinterest due to their effectiveness and better flexibility.
Viewing words and topics as embeddings in ashared latent space makes it possible for ETMs to integrate prior knowledge in the form of pretrainedword embeddings or semantic regularities [10, 27, 13].
The main idea behind those models is toconstrain the topic embeddings under the predefined topic structure.
Those models however are builton a strong assumption that the given topic hierarchy is well defined and matched to the target corpus,which is often unavailable in practice.
Such mismatched structure might hinder the learning processand lead to sub-optimal results.
To address the above shortcomings, in this paper, we first propose TopicKG, a knowledge graphguided deep ETM that views topics and words as nodes in the predefined topic tree.
As a Bayesianprobabilistic model, TopicKG aims to model the document Bag-of-Word (BoW) vector and the giventopic tree jointly by sharing the embedding vectors.
Specifically, we first adopt a deep documentgeneration model to discover multi-layer document representations and topics.
To incorporate thehuman knowledge in the topic tree and guide the learning of topic hierarchy, a graph generativemodel is employed, which refines the node embeddings by exploring the belonging relations hiddenin topic tree, resulting in more interpretable topics.
Besides, to address the mismatch between thegiven topic tree and target corpus, we further extend TopicKG to TopicKGA, which adopts the graph
A daptive technique to explore the missing links in topic tree from the target corpora, achieving betterdocument representation.
The final revised topic tree is obtained by combining the prior structure ofthe given topic tree with the added structures learned from the current corpus.
Thus the topic tree inour TopicKGA has the ability to reinforces itself constantly according to the given corpora.
Finally,both the proposed models are based on variational autoencoders (VAEs) [21] and can be trained bymaximizing the evidence lower bound (ELBO), enjoying promising flexibility and scalability.
The main contributions of this paper can be summarized as follows: 1) , A novel knowledge-awaredeep ETM named TopicKG is proposed to incorporate prior domain knowledge into hierarchicaltopic modeling by accounting for the document and the given topic tree in a Bayesian generativeframework.
2) , To overcome the drawback of the mismatch issue between the prior structure of theprovided topic tree and the current corpus, TopicKG is further extended to TopicKGA, which allowsto revise the prior structure to better represent the current document. 3) , Besides achieving promisingperformance on hierarchical topic discovery and document classification task, the proposed modelsgive good flexibility and efficiency, providing a strong baseline for knowledge-based TMs.
Related work
Embedded topic model: The recent developed ETM is a new topic modeling framework in embedding space. [10] first proposed ETM with the goal of marrying traditional topic models with wordembeddings. In particular, ETM regards words and topics are continue embedding vectors and thetopic’s distribution over words is calculated using the inner to LDA, ETM employs the logistic-normal distribution to estimate the posterior of per-document topicproportion, making it is easier to reparameterization in the inference step.
All parameters in ETM areoptimized by maximizing its evidence lower bound (ELBO).
To explore hierarchical document topicalrepresentations, ETM is extended to SawETM [12] with the gamma belief network [39].
SawETMdesigns the sawtooth connection (SC) to capture the relations between topics at two adjacent layersin the embedding space, resulting in more efficient algorithm for hierarchical topic mining. Note thatour proposed TopicKG is built upon SawETM, but different from SawETM that only considers the BoW representation, which may fail to discover high quality topic hierarchies when a large of topicsand their relations need to be modeled [27, 13], TopicKG introduces a novel Bayesian generativemodel where both document and the pre-specific domain knowledge are considered to integratehuman knowledge into NTMs, resulting in a new knowledge-aware topic modeling framework.topic models with various knowledge: Pre-trained language models such as BERT [9] and GPT [6]have been successful in various natural language processing tasks.
Pre-trained on huge text, suchmodels can serve as a powerful encoder that outputs contextual semantic embeddings.
Behind thisidea, combining a TM with BERT via knowledge distillation [16] treats BERT as the teacher model toimprove the representation of topic models.
CombinedTM [3] concatenates the Bag-of-Word (BoW)vector with the BERT contextual embedding as the final input of a TM, providing a consistent increasein topic coherence.
Despite their improvement, those BERT-based TMs focus on the document latentrepresentation, they ignore the relationship between words and topics.
Moreover they require alarge amount of memory to load the pre-trained BERT.
On the other hand, incorporating knowledgegraph ( e.g. , WordNet [28]) to improve existing NTMs becomes an interesting direction recently.
Forexample, [27] use the category tree (usually with simple and shallow structure) as the supervisedinformation and preserve such relative hierarchical structure in the spherical embedding space. toexplore user interested topic structure.
[13] view words and topics as the Gaussian distributions andemploy the asymmetrical Kullback–Leibler (KL) divergence to measure the directional similarity inthe given topic hierarchy.
However, computing such KL divergence for each pair is time-consumingthat limits its application to large-scale knowledge.
Moreover, those knowledge-based models ignorethe mismatch problem between the given topic tree and the current corpus, leading to sub-optimalresults in practice.
The proposed model
To incorporate human knowledge into deep TMs, we first propose TopicKG that generates the targetcorpus and the given topic tree in the Bayesian manner.
We further extend TopicKG to its adaptiveversion TopicKGA that allows the topic structure to be refined according to the current corpus,resulting in a better document representation.
Below we introduce the details of our proposed model.
3.1 Problem formulation
Consider a corpus containing J documents with a vocabulary of V unique tokens.
Each documentis represented by the BoW vector x ∈ R V + , where x v denotes the number of occurrences of thev th word.
Unlike other TMs that are purely data-driven, TopicKG intends to inject the knowledgegraph to improve topic quality.
Specifically, we introduce a topic tree T as our prior knowledge,where each node e i ∈ T denotes a word or a topic.
For each topic node, there is a correspondingdefinition as shown in Fig.
Suppose there are L + 1 layers in the topic tree with the bottom wordlayer and L topic layers, and there are K l nodes T l : { e ( l ) 1 , ..., e ( l ) K l } at the l -th layer, where K 0 = V .
Generally, for the k -th topic node at the l -th layer e ( l ) k , we use S ( e ( l ) k ) to denote the set of its childnodes, and use C ( e ( l ) k ) to denote its key words extracted from the definition sentence.
Mathematically,we adopt the binary matrix S ( l ) ∈ { 0 , 1 } K l − 1 × K l and C ( l ) ∈ { 0 , 1 } V × K l , l = 1 , ..., L , to denote thebelonging relations between two adjacent layers and the links between the topics and their conceptwords, respectively.
Guided by this pre-specified topic tree, TopicKG expects to retrieve hierarchicaltopics from the target corpus that provide a clear taxonomy for the end users.
3.2 Knowledge-aware Bayesian Deep Topic Model
Given the corpus D and topic tree T , we aim to discover hierarchical document representationsand topic hierarchy by jointly modeling the BoW vector x , S ( l ) and C ( l ) , l = 1 , ...
L in a Bayesianframework.
Specifically, we employ the gamma belief network (GBN) of [38] to model the count- value vector x , and use the semantic similarity between two node embeddings to generate the topictree T .
The whole generative model with L layers can be expressed as:
[Equation]
where x j is generated as in Poisson factor analysis [37], in which x j is factorized as the product ofthe factor loading matrix (topics) Φ (1) ∈ R K [-
-]0 × K 1 + and the gamma distributed factor scores (topicproportions) θ (1) j ∈ R K 1 + under the Poisson likelihood; Then the GBN [39] is applied to exploremulti-layer document representations by stacking hierarchical prior in the topic proportions θ j .
Thetopics at each layer Φ ( l ) , l = 1 , ..., L is calculated by the semantic similarity ( e.g. , the inner product)between corresponding topic embeddings, followed by a Softmax layer to satisfy the probabilitysimplex: (cid:80)[-K l − 1 v =
-]
K l − 1 k l − 1 =1 Φ ( l ) k l − 1 k = 1 .
e ( l ) k ∈ R
d is the embedding vector of k -th topic at l -th layer ( k -thnode at l -th layer e lk in T ), d is the embedding dimension.
e (0) v is the embedding vector for v -thword in vocabulary.
σ ( · ) is the sigmoid function and Bern ( · ) denotes the Bernoulli distribution thatis employed to model the two edge types S ( l ) and C ( l ) .
As suggested in [27] and [13], we useasymmetry to capture the directed relations in S ( l ) , while use symmetry to capture the semanticsimilarity between topics and their concept words.
W is a learnable parameter to guarantee thedirected structure.
Under the Bernoulli likelihood, our TopicKG has the ability to incorporate thetopic hierarchical relations in T into topic modeling by sharing word and topic embeddings withΦ .
In particular, for a node pair in T , the embedding semantics of the source and destination nodesneed to be similar enough to determine whether there is an edge, which will guide the learning of Φ ,resulting in more coherent topics and some appealing model properties as described below.
The flexibility of human knowledge incorporation: As mentioned above, guided by the predefined topic tree, TopicKG models x , S ( l ) and C ( l ) , l = 1 , ...
L jointly, making it possible for thelearned hierarchical topics that not only satisfy the interpretable taxonomy structure but also preventincoherent issues with the help of side information.
Besides, the incorporation of topic tree T issimple and flexible.
First of all, as shown in Eq. 1 and Sec. 4, the plug-and-play module of S ( l ) and C ( lk ) can be applied to most of ETMs without changing their model structures, which provides aconvenient alternative for introducing side information to TMs.
Secondly, the pre-defined T can beconstructed in various ways so as to encode our beliefs about the graph structure, e.g. , the knowledgegraph, taxonomy, and hierarchical label tree [33, 17, 27].
The relationship between { Φ ( l ) } Ll =1 and { S ( l ) } Ll =1 : The topic distribution matrix Φ ( l ) and thesymmetry matrix S ( l ) are related but play different roles.
The sparse structure matrix S ( l ) isconstructed from the topic tree which will be used to guide the learning of the hierarchy of topics,while the learned topics Φ ( l ) contains more specific knowledge extracted from the text corpus, andcan be viewed as the dense version of S ( l ) .
In other words, the topics learned from TopicKG retainsthe semantic structure of S ( l ) and enriches itself by the current corpora.
3.3 Inference network of TopicKG
The inference network of the proposed model is built around two main components: Weibull upwarddownward variational encoder and the GCN-based topic aggregation module.
The former aimsto infer the topic proportions given the document, and the latter updates the node embedding byaggregating the neighbor information via the GCN layer.
Weibull upward-downward variational encoder:
To approximate the posterior of the topic proportions { θ ( l ) j } , like most of VAE-based methods, we define the variational distribution q ( θ j | x j ) , which can be further factorized as [31]: [Equation] L [Equation]
l
[Equation]
l
[Equation]
In practice, it first obtains the latent feature by feeding the input x j into a residual upward neuralnetworks: h ( l ) j = h ( [Equation] j [Equation]
f
[Equation] ) , where l = 1 , ..., L , h [Equation] j = x j , f
[Equation]
l ) W [Equation] ) h ( · ) is a two layerfully connected network parameterized by W ( l ) h .
To complete the variational distribution, we adoptthe Weibull downward stochastic path [36]:
[Equation]
where ⊕ denotes the concatenation at topic dimension, and we use the Softplus to make surethe positive Weibull shape and scale parameters.
f is a single layer fully connected network,correspondingly.
The Weibull distribution is chosen mainly because it is reparameterizable and the
KL divergence from the gamma to Weibull distributions has an analytic expression [36, 15].
[Figure]
GCN-based topic aggregation: Attracted bythe excellent ability of graph neural network(GCN)
[22, 23] in propagating graph structureinformation, and the predefined topic tree canbe viewed as a directed graph.
We thus construct a deterministic aggregating module fornode embeddings with GCN:
[Equation]
where T is the number of GCN layer,
E (0) ∈
R d × N is the node embedding matrix, and N =(cid:80)
Ll =
K l .
˜
A
= D − 12 AD − 12 is the normalized adjacent matrix with degree matrix D , e.g. ,
D [Equation]
= (cid:80) Nj =
1 A ij , A ∈ { 0 , 1 } N × N is the adjacent matrix of the topic tree T .
Unlike previous ETMs that view embeddings as independentlearnable parameters, the GCN module in TopicKG updates the node embeddings by considering their child-level topics and related words, resulting in more meaningful topic embeddings.
Wecalculate { Φ ( l ) } Ll =1 via the updated node embeddings:
[Equation]
3.4 TopicKG with adaptive structure
One of the main assumptions in the previous section is that the pre-defined topic tree is helpful for thecurrent corpus and the corresponding edges are highly reliable.
However, this is generally unrealisticin practical applications, as T may be (i) noisy, (ii) built on an ad hoc basis, (iii) not closely related tothe topic discovering task [14, 8].
Consequently, we further propose TopicKGA that overcomes theabove mismatching issue and revises the structure of the given topic tree based on the corpus at hand.
In detail, TopicKGA first randomly initializes a learnable node embedding dictionaries E A ∈ R d × Nfor all nodes in T .
Then an adaptive graph A ada is generated based on E A using the certain kernelfunction k
: R d × R d −→ R :
[Equation]
Here, we choose the consine similarity to define our kernel function: k ( E A , E A ) = E [Equation]
E TA ||
E
A |||| E A
|| , and
Softmax function is used to normalize the adaptive matrix.
Note that, instead of generating A ada , we directly generate its normalized version to avoid unnecessary calculations [2].
During training, E Awill be updated automatically to learn the hidden dependencies which are ignored by A .
Following[35], the final revised adjacency matrix is formed as (here we still use ˜ A to denote the revised graphfor convenience):
[Equation]
) which will enhance the GCN in Eq.
3 by replacing the adjacency matrix with the revised one.
Training
As a deep ETM, TopicKG intends to learn the latent document-topic distribution and the deterministichierarchical topic embeddings.
Like other ETMs, the posterior of the topic proportions and topicembeddings are intractable to compute.
We thus derive an efficient algorithm for approximatingthe posterior with amortized variational inference [5], which makes the proposed model flexible fordownstream task.
The resulting algorithm can either use pre-trained word embeddings, or train themfrom scratch.
The ELBO of the proposed model can be expressed as:
[Equation]
It consists of three main parts: the expected log-likelihood of x (first term), the concept structurelog-likelihood (the two middle terms), and the KL divergence from prior p ( θ ( l ) j ) to q ( θ ( l ) j ) .
The graphweight β denotes the belief about the predefined topic tree.
Notably, the two middle terms distinguishthe proposed models from previous deep ETMs.
On the one hand, it acts as a regularization thatconstrains the embedding vectors to conform to the provided prior structure, and on the other hand, itprovides an alternative for ETM to introduce side information to improve its interpretability.
Annealed training for TopicKGA:
In TopicKGA, the structure of the revised graph changes duringthe training.
To incorporate the new edges that are inferred from the target corpus, we develop anannealed training algorithm for TopicKGA.
Specifically, we update S ( l ) and C ( l ) in Eq.
7 every Miterations:
[Equation]
where s is the threshold, and ˜ A ( l ) is the corresponding sub-block in ˜ A .
As E A becomes stable, thestructure of the graph will converge to a good blueprint that balances the prior graph and the currentcorpus.
We summarize all training algorithm in Appendix.
Experiment
In this section, we conduct extensive experiments on several benchmark text datasets to evaluatethe performance of the proposed models against other knowledge-based TMs, in terms of topicinterpretability and document representations.
5.1 Corpora
Our experiments are conducted on four widely used benchmark text datasets, varying in scale.
Thedatasets include 20 Newsgroups (20NG) [24], Reuters extracted from the Reuters-21578 dataset, R8,and Reuters Corpus Volume 2 (RCV2) [25].
R8 is a subset of Reuters that collected from 8 differentclasses.
For the multi-label RCV2 dataset, we follow previous works [29] in which documents documents totally
R RCV2 are -processed
standard words than times
5.2 Baselines
To demonstrate the effectiveness of incorporating human knowledge into deep TMs, we considerseveral baselines for a fair comparison, including representative ETMs and recent knowledge-basedtopic models, described as follows: 1) , ETM [10], the first neural embedded topic model that assumestopics and words live in the same embedding space.
The topic distributions in ETM are calculatedby the inner product of the word embedding matrix and the topic embedding. 2) , CombinedTM[3], a BERT-based neural topic model that uses the pre-trained BERT as its contextual encoder.
We use it as our BERT baseline. 3) , SawETM [12], a hierarchical ETM that employs the Poissonand gamma distributions to model the BoW vector and the latent representation, respectively. 4) ,
JoSH [27], a knowledge-based hierarchical topic mining method that uses category hierarchy as theside information and employs the EM algorithm to learn the spherical tree and text embedding. 5) , TopicNet [13], a semantic graph guided topic model that views the words and topics as the Gaussiandistributions and uses the KL divergence to regularize the structure of the pre-defined tree.
For allbaselines, we employ their official codes and default settings obtained from their release repositories.
5.3 External knowledge
Since the learning of TopicKG involves a pre-defined topic tree, some generic knowledge graphs suchas WordNet [28] provide a convenient way to bring in external knowledge.
WordNet is a large lexicaldatabase that groups semantically similar words into synonym sets, these sets are further linked bythe hyponymy relation.
For example, in WordNet the category furniture includes bed, which in turnincludes bunkbed.
With these relations, the topic structure can be easily defined according to certainheuristic rules.
Specifically, for each dataset we first get the word intersection of the vocabulary and WordNet, the chosen words are then considered as leaf nodes of the topic hierarchy.
Afterwards, eachleaf node can continuously find its ancestor nodes based on hyponymy relations and finally all leafnodes converge to the same root node, resulting in an adaptive topic structure.
5.4 Settings
For all experiments, we set the embedding dimension as d = 50 , the knowledge confidence hyperparameter as β = 50 . 0 , the threshold as s = 0 . 4 .
Empirically, we find that the above settings workwell for all datasets, we refer readers to the appendix for more detailed analysis.
We initialize thenode embedding from the Gaussian distribution N (0 , 0 . 02) .
We set the batch size as 200 and use the
AdamW [26] optimizer with learning rate 0.01.
We generate a 7-layer topic hierarchy for each dataset with the method described in Sec. 5.3.
From the top to bottom, the number of topics at each layerare: 20NG: [1,2,11,66,185,277,270]; R8: [1,2,11,77,287,501,547]; RCV2: [1,2,11,68,263,469,562];
Reuters: [1,2,11,78,300,526,584].
For the single-layer methods (ETM and CombinedTM), we set thenumber of topics as the same of the first layer of deep models.
For all methods, we run the algorithmsin comparison five times by modifying only the random seeds and calculate the mean and standarddeviation.
All experiments are performed on an Nvidia [-RTX
 Ti-]RTX 3090-Ti GPU and our proposed modelsare implemented with PyTorch.
[Table]
5.5 Topic interpretability
Generally, topic models are evaluated based on perplexity.
However, perplexity on the held-out testis not an appropriate measure of the topic quality and sometimes can even be contrary to humanjudgements [7, 33].
To this end, we instead adopt three common metrics, including Topic coherence( TC ), Topic diversity ( TD ) and word embedding topic coherence ( WE ), to evaluate the learned topicsfrom various aspects [10, 3].
TC measures the average Normalized Pointwise Mutual Information(NPMI) over the top 10 words of each topic, and a higher score indicates more interpretable topics.
TD denotes the percentage of unique words in the top 25 words of the selected topics.
WE, as its nameimplies, provides an embedded measure of how similar the words in a topic are.
WE is calculated bythe average pairwise cosine similarity of the word embeddings of the top-10 words in a topic.
Wereport the topic quality results at the first five layer in Fig.
We here focus on topic hierarchyand ignore the single layer methods.
We find that: 1) , Knowledge-based methods including JoSH, CombinedTM, TopicNet and our proposed models are generally better than other likelihood-based ETMs, which illustrates the benefit of incorporating side information; 2) , Our proposed TopicKG and TopicKGA achieve higher performance than other knowledge-based models in most cases, especiallyon TC and WE tasks, which means our proposed model prefer to mine more coherent topics whileachieving comparable TD.
We attribute this to the joint topic tree likelihood that provides an efficientalternative to integrate human knowledge into ETMs. 3) , Compared to TopicKG, its adaptive version TopicKGA discovers better topics at higher layers.
It is not surprise that TopicKGA gives greaterrobustness and flexibility by refining the topic structure according to the current corpus.
5.6 Document classification
Besides the topic quality evaluation, we also use document classification to compare the extrinsicpredictive performance.
In detail, we collect the inferred topic proportions, e.g. , θ (1) , and than applylogistic regression to predict the document label.
We report the Micro F1 and Macro F1 score on20NG, R8 and RCV2 datasets in Table.
Overall, the proposed models outperform the baselineson all corpora, which confirms the effectiveness of our innovation of combining human knowledgeand ETMs in improving document latent representations.
Moreover, with the revised topic structurefine-tuned to the current corpus, TopicKGA surpasses other knowledge graph fixation methods withsignificant gaps.
This conclusion is consistent with one of our motivations that the predefined concepttree may not match the target corpus, leading to suboptimal document representations.1 We can’t [Figure]
5.7 Qualitative analysis
We visualize the learned topic hierarchies of TopicKG and
TopicKGA on 20NG in Fig.
4(a-b), respectively.
Eachtopic box consists of the pre-specified concept name onthe top bar and its keywords listed in the bottom content.
We can observe that 1) , The mined keywords arehighly relevant to the corresponding topics, providing aclean description of their concepts. 2) , Guided by humanknowledge, the connections between topics at two adjacentlayers are highly interpretable, resulting in human-friendlytopic taxonomies.
3) , More interestingly, to further verify the adaptive ability of our proposed TopicKGA, wemanually added several free topics (dashed boxes in Fig.4(b)) to each layer of the given topic tree, which can beviewed as the missing topics in the predefined knowledgeand need to be learned from the target corpus.
We findthat our TopicKGA indeed has the ability to capture themissing concepts, and revise the prior graph to match thecurrent corpus.
5.8 Time efficiency
To demonstrate the time efficiency of incorporating humanknowledge into TMs, we run TopicKG, TopicKGA and
TopicNet (as they are implemented in PyTorch, JoSH is in C) on RCV2 and 20NG dataset witha 7-layer topic tree.
Fig.
5 shows the Negative log-likelihood of documents x , which shows thatthe proposed models not only have faster learning speed than TopicNet, but also achieve betterreconstruction performance on both small and large corpus.
This result illustrates the efficiency ofthe introduced graph likelihood which is important in real-time applications.
Conclusion
We develop an efficient Bayesian probabilistic model that integrates pre-specified domain knowledgeinto hierarchical ETMs.
The core idea is the joint document and topic tree modeling with the sharedword and topic embeddings.
Besides, with the graph adaptive technique, TopicKGA has the abilityto revise the given prior structure according to the target corpus, enjoying robustness and flexibilityin practice.
Extensive experiments show that the proposed models outperform competitive methodsin term of both topic quality and document classification task.
Moreover, thanks to the efficientknowledge incorporation algorithm, our proposed models achieve faster learning speed than otherknowledge-base models.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Please refer to the appendix
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please refer to the appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code and
data are included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you citep the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/ A
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]