Meta-Complementing the Semantics of Short Texts in Neural Topic Models
Abstract
Topic models infer latent topic distributions based on observed word co-occurrences in a text corpus.
While typically a corpus contains documents of variable lengths, most previous topic models treat documents of different lengths uniformly, assuming that each document is sufficiently informative.
However, shorter documents may have only a few word co-occurrences, resulting in inferior topic quality.
Some other previous works assume that all documents are short, and leverage external auxiliary data, e.g., pretrained word embeddings and document connectivity.
Orthogonal to existing works, we remedy this problem within the corpus itself by proposing a Meta-Complement Topic Model, which improves topic quality of short texts by transferring the semantic knowledge learned on long documents to complement semantically limited short texts.
As a self-contained module, our framework is agnostic to auxiliary data and can be further improved by flexibly integrating them into our framework.
Specifically, when incorporating document connectivity, we further extend our framework to complement documents with limited edges.
Experiments demonstrate the advantage of our framework.
Introduction
Much of the data on the Web can be represented as text documents. Topic models help to understand the main themes within documents, i e., each document is represented by a topic distribution and each topic is interpreted by its key words.
The quality of topic distribution of each document depends on sufficient word co-occurrences.
However, many real-word corpora contain documents of variable lengths .
Academic papers vary from journal manuscripts to conference papers to extended abstracts.
News articles could be headlines, short or full articles or detailed commentaries.
Fig.
1(a) illustrates an academic paper corpus where the distribution of document lengths exhibits a long-tail distribution.
Despite variable lengths (with different degrees of sufficiency of word co-occurrences), existing works, e.g., ProdLDA [31] and GATON [37], treat documents uniformly, resulting in inferior topic quality for short texts.
Evidentially, Fig. 1(b) presents document classification accuracy on four subsets of corpus with descending lengths.
Accuracies gradually drop as the length decreases.
The inferior topic quality of short texts limits the overall performance of a topic model. [Figure] +
Challenges.
Most existing topic models optimize the learning process by averaging the generative losses of different documents, without paying special attention to semantically limited short texts.
A few studies, e.g., OTLDA [15], take weighted summation of losses based on document lengths, they compute the weights by dividing the length of each document by the length of the whole corpus, which further deemphasizes the importance of short text modeling.
Thus, we seek to improve short text topic modeling within a variable-length corpus, without hurting topic quality of long documents.
To mitigate the scarcity of word co-occurrences in short texts, some works leverage auxiliary knowledge to enhance topic modeling.
ETM [8] exploits pretrained word embeddings [27, 29] to capture word similarities.
RTM [5] constructs a document network, e.g., paper citation network, to aggregate topics of connected documents.
However, they rely on the availability of auxiliary data.
Approach.
We propose M eta- C omplement T opic M odel (MCTM).
Since a corpus contains variable lengths of documents, we are motivated to learn the transferable semantic knowledge on long documents, and complement the semantics-scarce short texts and enhance the latter’s topic distributions.
As illustrated by Fig.
1(c), we learn a function g on long document A with abundant content, and use it to predict missing semantics (red box) for short document B to complement its content for a more expressive topic distribution.
Since meta-learning [10] is emerging to improve model performance with few labeled observations, but no one explores its design in topic modeling.
We are thus motivated to integrate it and optimize the proposed MCTM by a meta-learning objective.
Orthogonal to existing works relying on auxiliary data, our framework is self-contained, assuming only in-corpus information, which offers a new direction to improve short text topic modeling.
When auxiliary data are available, our framework can be further improved by flexibly incorporating them.
In particular, when incorporating document network structure, we discover that document degrees also exhibit a similar long-tail distribution, i.e., some structure-abundant documents link to sufficient neighbors as auxiliary, while others contain scarce links.
Besides textual semantic complement at Fig. 1(c) , we extend MCTM to further complement structural semantics on the document network.
Contributions.
Our contributions are as follows.
First , we propose MCTM, which learns how to complement textual semantics by semantic knowledge transfer.
Second , we derive two alternatives to implement missing semantics prediction function g to capture document similarities.
Third , although agnostic to auxiliary data, MCTM can also flexibly integrate them to further improve the performance.
We demonstrate our adaptability by modeling pretrained word embeddings and document networks.
For the latter, we extend MCTM to further complement structural semantics.
Fourth , extensive experiments verify the effectiveness of MCTM.
Problem Formulation and Preliminaries
We are given a corpus of N documents D
= { d i } Ni =1 .
Each document d ∈ R |V| is a vector in the vocabulary space V .
l d = (cid:80) w ∈V d w is the length of document d where d w is the word count of w in d .
When word embeddings are available, we have H
= { h w } w ∈V where h w is the embedding of word w .
Documents may link to others in a document network G =
{D , E} , with documents D and network connectivity E = { e ij } N i,j =1 .
If document d links to d ′ , e d,d ′ = 1 , otherwise e d,d ′ = 0 .
N ( d ) is the set of neighbors of d .
We consider an undirected network, e d,d ′ =
e d ′ ,d .
Corpus D contains documents of variable lengths { l i } Ni =1 .
We introduce a hyperparameter L as the threshold where short documents are those with fewer observed words, D short = { d i | l i < L } , and long documents are defined symmetrically, D long =
{ d i | l i ≥ L } .
We consider L as a predefined hyperparameter and leave other designs as future work.
See Table 1 for a summary of math notations. [Table]
Given a variable-length corpus D (as well as word embeddings H and network E if observed) as input [Table] we aim to output topic distributions for documents where the topic quality of short documents is improved, without hurting long documents.
Note that our goal is not to allow short texts to reach the performance of long documents [Table] but to improve short text topic modeling as much as possible.
Meta-Learning.
Meta-learning [10] optimizes globally shared parameters, a.k.a.
prior knowledge, over meta-training tasks, so as to rapidly adapt the model to previously unseen meta-testing tasks with only a few observed data.
Since topic models generally learn topics by a content generative process, here we consider generating observed words for a document d as a task T d .
A meta-training task T d corresponds to a training document d and consists of a support set and a query set, T d = {S d , Q d } .
Each set contains randomly sampled words from document d , such that support words and query words are mutually exclusive, S d ∩ Q d = ∅ and S d ∪ Q d = d .
Meta-training has two steps: 1.
Local update.
Given a topic model f θ with parameter θ , f θ is first updated from the globally shared parameter θ to document-specific local parameter θ d w.r.t.
loss on d ’s support words S d .
[Equation]
[Equation]
T d = {S d , Q d } ∈ T tr .
[Equation]
1) α is meta-learning rate, ∇ is gradient, L is loss function, T tr is a set of meta-training tasks (documents).
Global update.
After obtaining θ d for each document d , we compute the loss on query words
L ( θ d , Q d ) .
Together with other training tasks, we optimize the globally shared parameter θ .
[Equation]
θ ∗ is the new globally shared parameter and will replace θ at Eq.
1–2 for the next iteration.
After convergence, the final global parameter θ ∗ can easily be adapted to meta-testing tasks.
Meta-testing tasks T te are unseen test documents.
All the observed words are support words, T d =
S d = d .
During meta-testing, topic model f θ ∗ with optimized global parameter θ ∗ is updated w.r.t.
S d by Eq.
and obtain θ ∗ d .
The topic distribution of testing document is inferred by z d = f θ ∗ d ( d ) .
Methodology
We introduce M eta- C omplement T opic- M odel (MCTM) at Fig.
Below we elaborate three components, graph convolutional encoder, missing semantics prediction, and meta-learning optimization.
pretrained word embeddings words documents 𝐡 𝑤 1 𝐡 𝑤𝐡 𝑤𝛾 𝑑 ⨀ + 𝛽 𝑑 𝐔 𝑑 𝐦 𝑑 𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤𝛾 𝑑 ′ ⨀ + 𝛽 𝑑 ′ 𝐔 𝑑 ′ 𝐦 𝑑 ′ 𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤cluster 𝑅 clustercluster… … 𝐖 𝑟= 𝐖 𝑟= 𝐦 𝑑 𝐦 𝑑 ′ doc 𝑑 pruned short 𝑑 ′ doc 𝑑 pruned short 𝑑 ′ Support loss min 𝜃 ℒ gen (𝜃,𝒮 𝑑 )=
𝜃 𝑑
Query loss 𝜃 ∗ =
min 𝜃 ℒ({𝜃 𝑑 ,𝜇},𝒬 𝑑 ) query word 𝑤query neighbor 𝑑 3 = + 𝐳 𝑑 ′ ∗ 𝐦 𝑑 ′
𝐳 𝑑 ′ =
+ 𝐳 𝑑 ∗ 𝐦 𝑑 𝐳 𝑑 𝐳 𝑑 𝐳 𝑑 ∗ + = 𝐳 𝑑 ′
+ 𝐳
[Figure] [Figure] 3.1 Graph Convolutional Topic Encoding We follow GATON [37] and first present a graph convolutional encoder f θ (Fig. 2(a)), which projects documents to K -dimensional topic distributions.
We defer the discussion on short text modeling to the following subsections.
Given a corpus D , considering documents and words as vertices, we construct a bipartite graph, the links represent word occurrences in the documents.
Since documents and words preserve heterogeneous feature spaces, we project them to the same topic space by learnable matrix,
[Equation]
l is the l -th convolutional layer.
z (0) d and z (0) w are inputs, i.e., Bag-of-Words and one-hot, respectively.
To differentiate the importance of words, we evaluate attention between document d and its observed support words at Eq.
4 and aggregate words at Eq.
5 where [ ·||· ] is concatenation.
[Equation]
Symmetrically, we modify Eq.
4 and obtain a w,d , i.e., the attention between word w and documents it appears in.
After symmetric aggregation at Eq.
5, we obtain z ( l +1) w for word w .
So far, we complete the convolution from l -th to ( l + 1) -th layer.
For simplicity, we summarize aggregation at Eq.
3–5 by
[Equation]
l
[Equation]
We repeat Eq.
6 for maximum L layers and obtain K -dimensional topics
z d = z ( L ) d for document d and z w = z ( L ) w for word w .
The complete encoder is Eq.
θ is the set of all encoding parameters.
[Equation]
2 Missing Semantics Prediction with Contrastive Learning
A short document with few words leaves some content poorly described, resulting in incomplete topic distribution z d .
As a toy example, document B at Fig.
1(c) contains limited words, e.g., gene and clone, and leaves other contents, e.g., protein and cell, uncovered.
We aim to complement the topics of short documents.
For a document d , regardless of long or short, we complement its semantics by
[Equation]
We name m d ∈ R K missing semantics of document d .
If d is a long document with complete semantics, m d is a zero vector.
A function g µ predicts missing semantics at Eq.
9 with topic distributions of d and its support words as inputs.
We will elaborate the design of function g µ shortly.
[Equation]
Contrastive Learning.
Long documents contain relatively more sufficient word co-occurrences than short documents.
Thus, we learn missing semantics prediction function g µ on long documents, and then transfer the learned semantic knowledge to complement short documents.
On one hand, a long document d with enough content does not need semantic complement.
Thus we have below constraint
[Equation]
d
[Equation]
0 where d ∈ D
[Equation]
On the other hand, although we aim to transfer the semantic knowledge from long to short documents, there does not exist a one-to-one correspondence in corpus D .
As a result, we may transfer semantics of a long document (e.g., machine learning concepts) to a short one describing completely distinct content (e.g., biology).
To overcome this limitation, we introduce another constraint with contrastive objective.
For a long document d with support words S d , we randomly hide a proportion of words to mimic a short document, denoted as d ′ with remaining observed words S d ′ ⊂ S d and length l d ′ < L .
For both long text d and its short version d ′ , we predict their missing semantics by Eq. 9, and obtain m d and m d ′ , respectively.
m d ′ should complement the previously hidden semantics, i.e., S d − S d ′ ,
[Equation]
where d ∈ D long .
[Equation]
[Equation]
Together with above Eq. 10, which forces z ∗ d = z d + m d to approach z d , Eq.
11 actually has
[Equation]
where d ∈ D long .
[Equation]
[Equation]
To summarize, we arrive at the following constraint loss.
[Equation]
σ
[Equation]
= 1 1+exp( − x ) is sigmoid function, and cos( · , · ) is cosine similarity.
We here use cosine similarity, mainly due to its superior performance on our datasets.
Besides cosine, other similarity metrics, such as inner product and Euclidean distance, are also possible, depending on different datasets.
Missing Semantics Prediction.
We now define the function of missing semantics prediction g µ .
Given topic distributions of a document and its observed support words as input, a desirable function should aggregate them and output a single missing semantics vector.
We propose two alternatives.
GNN function.
The first is to implement g µ using a graph neural network, see Fig.
[Equation]
A corpus D contains documents with diverse themes.
For example, some documents discuss machine learning, while others describe biology.
However, the assumption of corpus-level shared parameter
U 1 and U 2 is not flexible to model diverse documents for missing semantics prediction, since different documents may have distinct optimal parameters, which are sometimes even in opposing direction.
As a result, the predicted missing semantics m d centers its mass around the most frequent topics and leaves other distinct topics uncovered.
We seek to personalize U
and U 2 for each document d to recover the distinct missing semantics.
Formally, we introduce a function ϕ , which transforms U 1 and
U 2 to document-specific parameters U d, 1 and U d, 2 by scaling and shifting.
Taking U 1 as example,
[Equation]
γ d and β d are document-specific vectors for scaling and shifting shared parameter U 1 .
⊙ is elementwise product.
[( x ) × K ] is a matrix with K identical column vector x . 1 is a vector of ones, ensuring the scaling matrix centers around one.
U d, 2 is similarly defined.
Eq.
15 allows each document d to have its own parameters, while all documents still share the common knowledge.
Similar documents scale and shift U 1 and U 2 to similar directions.
Different documents push them to distinct directions.
We define scaling γ d and shifting β d , parameterized by topics of document d and its support words.
[Equation] [Equation]
¯ z
[Equation]
= 1 |
[Equation] d | [Equation] w ∈S d z w is the average of d ’s words.
In summary, we use Eq.
14 to predict d ’s missing semantics, except that shared parameters
U
and U 2 are replaced by d -specific ones, U d, 1 and U d, 2 .
Clustering function.
We propose an alternative method by semantic clustering to recover distinct missing semantics, Fig. 2(c).
Documents with similar content fall into related clusters, while unique documents belong to different ones.
If we assign each cluster a set of parameters for missing semantics prediction, similar documents would recover their own distinct topics.
Specifically, we introduce R centroids { c r } Rr =1 , each corresponding to one cluster.
Given topic distributions of document d and its support words, we first evaluate the assignment probability between document d and each cluster by
[Equation]
[Equation]
h 0 =
[ z d || ¯ z S d ] is the concatenation of z d and ¯ z S d .
We then assign parameters to each cluster,
[Equation]
Therefore, related documents obtain similar clustering probabilities and missing semantics.
Above process is a flat clustering.
Our function can be extended to multiple clustering layers.
Each layer s consists of R ( s ) centroids.
After obtaining the output from previous layer, h s − 1 , we repeat Eq. 17–18 by replacing h 0 with h s − 1 , and obtain the output of the current layer s , i.e., h s .
For maximum
S layers, we get m d = h S .
We leave adaptive learning of number of clusters R as future work.
3.3 Probabilistic Decoding with Meta-Learning Optimization
After semantic complement, we obtain { z ∗ d } d ∈D at Sec. 3.2.
We use Z V = (cid:2)
z w 1 ; z w 2 ; ... ; z w |V| (cid:3) ∈
R K ×|V| to represent topic-word distribution, each column z w is topic distribution of a word w , and each row is to the distribution of a topic over the vocabulary.
As in previous topic models [3, 31], we generate the observed support words S d by ˆ d S d = σ ( Z V z ∗ d ) .
Compared to the ground-truth support words d S d , we follow [37] and obtain generative loss L gen = ||
d S d − ˆ d S d || 2 2 .
However, this loss requires inefficient computation over the whole vocabulary.
We instead use negative sampling [27].
[Equation]
M is the number of negative samples, and Pr n ( w ) is a noise distribution over vocabulary.
ˆ d w = σ ( z ∗⊤ d z w ) .
d w = 1 if w ∈ S d , otherwise d w = 0 .
In addition, if d is a long document d ∈ D long , we also created its corresponding pruned short version d ′ with a subset of support words S d ′ ⊂ S d at Sec.
Although we do not observe the complete support words for d ′ , its complete topic distribution z ∗ d ′ after semantic complement at Eq.
8 should be able to generate the complete support words S d .
Therefore, together with semantic complement constraint at Eq. 13, we arrive at the complete loss.
[Equation]
where L gen ( d ) =
L gen ( d ) + I ( d ∈ D long ) λ gen L gen ( d ′ ) ,
L con ( d ) = I ( d ∈ D long )
L con ( d ) .
L gen ( d ) is generative loss, consisting of document d in the corpus d ∈ D
= D long ∪ D short and the corresponding pruned short version d ′ (if d ∈ D long ).
I ( d ∈ D long ) =
if d ∈ D long , otherwise 0 .
L reg ( θ ) is L2 regularizer for encoding parameters θ .
λ gen and λ reg are hyperparameters.
Optimization.
Finally, with the objective of meta-learning at Sec. 2, we reach the optimization: 1.
Local update.
Given topic encoder f θ defined at Sec.
3.1, we optimize encoding parameter θ w.r.t. the generative loss L gen ( d ) on support words S d by Eq.
and obtain θ d for each document.
Global update.
With encoding parameter θ d , we compute the overall loss L ( d ) on query words
Q d and optimize all parameters
Φ = { θ, µ } , including encoder parameters θ and parameters µ of missing semantics prediction function g µ .
α 1 and α 2 are local and global learning rate, respectively. Φ
[Equation]
With new parameter Φ ∗ =
{ θ ∗ , µ ∗ } , we substitute it for Φ for the next iteration.
In contrast to previous topic models that generate the content of given documents only, we further create the short version of long documents for semantic complement, and jointly optimize them.
See supplementary materials for learning algorithm.
3.4 Extensions with Auxiliary Data
MCTM with Pretrained Word Embeddings.
Pretrained word embeddings [27, 29] encode word similarity.
As in previous works [7, 45], we incorporate them into topic-word distribution Z V =
[ z k,w ] to improve topic modeling.
We introduce topic embedding { t k } Kk =1 and evaluate cosine similarity between topic k and word w by cos( t k , h w ) .
We then combine it with topic-word distribution by z ′ k,w = 12 ( z k,w + cos( t k , h w )) and obtain a new topic-word distribution Z ′V =
[ z ′ k,w ] for decoding.
[Table]
MCTM with Document Network.
A document network (e.g., citation network) reveals semantic similarities between connected documents (cited papers discuss related research).
A document’s degree or number of links exhibits a long-tail distribution.
Some link to many neighbors, others to a few.
Previously focus was on textual semantic complement.
We extend MCTM to model structural semantic complement for link-scarce documents.
We consider generating both observed words and neighbors as a task T d .
As for words, we split the neighbors N ( d ) of a document d into support and query neighbors T d = {S d , S N ( d ) , Q d , Q N ( d ) } , S d and S N ( d ) denote support words and neighbors, respectively, and ditto for query sets.
We correspondingly extend three modeling components.
Encoding.
Previously, we inferred topic distribution z d of document d by its textual words at Eq. 6.
Here, we extend this process by designing a structural convolutional module.
[Equation]
The topic distribution from encoder f θ is z d := 12 ( z d + κ d ) , with both texts S d and structure S N ( d ) .
Semantics complement.
For a long document d , in addition to randomly hiding some words, we also drop some neighbors and create a pruned version d ′ .
Now the missing semantics m d should contain both textual and structural information.
For GNN function g µ , we extend Eq.
by m d = g µ ( z d , z w , z d ′ | w ∈ S d , d ′ ∈ S
N ( d ) ) =
AGG ( U d, 1 z d , U d, 2 z w , U d, 3 z d ′ | w ∈ S d , d ′ ∈ S N ( d ) ) .
Scaling has extra input, γ d = tanh ( W γ z d + W ′ γ ¯ z S d + W ′′ γ ¯ z S N ( d ) ) , ditto for shifting.
¯ z S N ( d ) = |S N ( d ) | (cid:80) d ′ ∈S N ( d )
z d ′ .
For Clustering g µ , we extend input by h 0 =
[ z d || ¯ z S d
|| ¯ z S N ( d ) ] .
Decoding with meta-learning.
In addition to generating support words using complemented z ∗ d , we also generate support neighbors.
The generative loss is similar to Eq.
19 except that i ) we replace d w with e d,d ′ , the ground-truth link between d and d ′ ; ii ) ˆ e d,d ′ =
σ ( z ∗⊤ d z ∗ d ′ ) .
Finally, meta-learning learns how to accurately predict missing semantics m d for both textual and structural complement.
Experiments
The goal of experiments is to evaluate if our model MCTM can improve short text topic modeling through evaluative tasks, e.g., document classification, link prediction, topic analysis.
Datasets.
Since our model is flexible to incorporate auxiliary data, we rely on four datasets with textual documents, auxiliary word embeddings, and auxiliary network links for experiments.
Cora [24] is corpus of academic papers with citations as links.
We created two independent datasets, Machine Learning ( ML ) and Programming Language ( PL ).
In addition, HEP-TH [21] is a corpus of Physics papers with their citations.
Web [20] is a Web page hyperlink network where each page is a news article, and the hyperlinks connect related articles.
See Table 2 for details.
Baselines.
Since our model has three variants, i.e., MCTM with plain texts, with auxiliary word embeddings, and with auxiliary document networks, we correspondingly compare to three categories of baselines.
i )
Topic models with plain texts , ProdLDA [31], WLDA [28], and GATON [37].
They model all documents uniformly without dealing with short texts.
We compare to them and show the advantage of MCTM on improving short texts.
ii ) Topic models with word embeddings , ETM [8] and NSTM [45].
Since our model is built on top of GATON, we also compare to GATON with word embeddings, denoted as GATON+WE.
By comparing to them, we verify the effectiveness of semantic complement meta-learning to further improve topic quality.
iii )
Topic models with document networks , RTM [5], Adjacent-Encoder [40], LANTM [35], and GATON+DN, which is [Table] the extension of GATON with document networks.
For document network scenario, we include a graph embedding model, meta-tail2vec [23], which uses meta-learning to improve nodes with low degrees, but is not a topic model and ignores variable lengths of node attributes, i.e., texts.
We set L = 2 convolutional layers.
λ gen = 2 and λ reg = 0 .
Number of negative samples M =and number of semantic clusters R = 5 .
L is the median length of the corpus.
Local and global learning rates are α 1 = 0 .
001 and α 2 = 0 .
We use 300D Glove embeddings.
We experiment with 5 independent runs, report mean and std.dev.
All the experiments were done on Linux server with a Tesla K80 GPU with 11441MiB.
4.1 Quantitative Evaluation
Document Classification.
Documents from the same category discuss related topics.
As in LDA [3], we conduct classification to evaluate topic quality.
We split 80% documents for training (10% are for validation).
Labels are not involved during training.
After convergence, we train a k NN classifier ( k = 5 ) [2] with training documents and predict the labels of test documents.
We set 64 topics.
We compare our models within each category of baselines and report classification accuracy at Table 3.
We split the test set into four subsets with descending document length and report the result of both overall test and each subset.
0-25% at Table 3 means the subset with the longest 25% test documents.
MCTM (G) and MCTM (C) denote our model with GNN and Clustering function, respectively.
We use “*” to represent statistically significant improvement with paired t-test at 0.05 significance level.
Our models significantly outperform baselines within each category.
We outperform GATON, the best baseline in the plain text and word embedding category, since textual semantic complement improves short texts, and the overall test set is also improved.
MCTM (G) performs slightly better than MCTM (C), potentially because GNN recognizes importance of words with attention, while the Clustering function takes simple average.
To show our models indeed improve short text quality, we present the improvement of MCTM (G) over both GATON and the best baseline.
Our performance generally improves more as the length decreases, which verifies the advantage of semantic complement. [Table]
Topic Coherence.
Each row of topic-word distribution Z V ∈ R K ×|V| is the distribution of one topic over the vocabulary, and the key words of this topic correspond to the highest values on this row.
As in
ProdLDA [31], we evaluate the coherence of key words by Google Web 1T 5-gram Version 1 [9], with NPMI as metric.
Table 4 (left) summarizes the results.
Topic-word distribution Z V is model parameter and is separate from document length, thus we can not report results of different lengths.
LANTM cannot run on large dataset Web.
Meta-tail2vec is not a topic model, thus is excluded.
Overall, our models outperform baselines on ML and PL and are competitive with the best baseline on HEP-TH and Web.
This indicates that our models at least do not hurt topic coherence, but can significantly improve other tasks, e.g., classification.
Compared to GATON, our models significantly improve it, verifying the [Table] [Table] advantage of semantic complement.
To understand what topics our models capture, we randomly present two topics with top-5 key words at Table 5.
MCTM (G) captures statistics and computational genetics , while MCTM (C) reveals scalability and geometric learning .
Perplexity.
Topic model should generalize to unseen documents.
Following [3], we evaluate perplexity.
Since perplexity is exponential and varies much w.r.t.
its power, we report its power, − logPr
( D test ) (cid:80) d ∈D test l d (lower is better).
Table 4 (right) reveals that our models generate high likelihood to unseen documents, which we attribute to semantic complement meta-learning module.
Link Prediction.
A good model should infer similar topics for potentially linked documents.
Since we model document network as auxiliary data, we follow RTM [5] and predict links.
As in [40], the probability of a link is p ( e d,d ′ )
∝ exp( −|| z ∗ d −
z
∗
d
′ || 22 ) .
We predict the links within test documents and compare with the ground-truth links with AUC as metric.
Since only the third category (network models) incorporate links, we mainly compare the MCTM+DN version to these network baselines.
Table 6 indicates that our models predict links more accurately than baselines.
The comparison to network baselines demonstrates the effectiveness of both textual and structural semantic complement.
4.2 Model Analysis
To better understand our model, we conduct model analysis here.
Effect of Semantic Complement.
To see if semantic complement helps short texts, we remove it from the complete model and present classification accuracy at Table 7 (left).
We show the plain text results, and put the auxiliary data version in supplementary.
Models do better with semantic complement than without.
The accuracy on short subset declines more than the overall test set, which reveals that semantic complement improves short texts, and removing it hurts short documents more.
Effect of Meta-Learning.
To analyze if meta-learning benefits the optimization with a few observed words, we replace it with the commonly used stochastic gradient descend.
Table 7 (middle) shows that result drops more on short subset than on the overall test set, which verifies that meta-learning is good at optimization with only a few observed words and improves short text modeling.
We further remove both semantic complement and meta-learning, and report the result at Table 7 (right), which presents the worst accuracy.
This observation further verifies that both components are important.
[Table]
Effect of Scaling and Shifting.
The GNN version of our model uses scaling-and-shifting method to recover distinct topics.
To test its usefulness [Table] we disregard it and summarize classification accuracy on ML at Table 8 (left).
Removing scaling and shifting leads to worse performance, since we can not personalize shared parameters to each document to recover its distinct missing topics for complement
[Table]
[Table]
Effect of Clustering.
We set the number of clusters R to 1, all documents share the same parameters for missing semantics prediction with no clustering.
Table 8(right) concludes that clustering is helpful to share common semantics for related documents and distinguish documents of different clusters.
Related Work
Traditional topic models are graphical models [3, 13].
Recent ones are neural models [26], mainly variants of VAE [17].
ProdLDA [31] and DVAE [4] analyze Dirichlet prior.
WHAI [43] looks into
Gamma prior.
The variational divergence metric also attracts attention.
WLDA [28] and WAE [32] apply mean maximum discrepancy and generative adversarial network [12].
KATE [6] increases topic sparsity.
More recently, topic models are built with graph neural networks [18], e.g., GATON [37], GraphBTM [46], etc.
They average losses of all documents regardless of lengths.
MCTM designs semantic complement to predict missing semantics for short texts to improve topic quality.
To alleviate scarce word co-occurrences of short texts, existing works leverage auxiliary data, such as word embeddings [27, 29].
Graphical models include GLDA [7], GPUMM [22], LCTM [14],
MetaLDA [44], WDL [36], and OTLDA [15].
Neural models include ETM [8] and NSTM [45].
Another category improves short text modeling with document networks, e.g., citation network.
RTM [5] and NetPLSA [25] are graphical models.
Neural models include NRTM [1], Adjacent-Encoder [40], LANTM [35], NetDTM [42], SemiVN [41].
In contrast, our semantic complement module is agnostic to auxiliary data.
Here, we also briefly review meta-learning works.
One category is metric-based, e.g., ProtoNet [30] and MatchingNet [33], which learn a metric function over tasks.
Gradient-based meta-learning works optimize model parameters for quick adaptation to new tasks [10, 16, 11, 34, 19, 38, 39].
Conclusion
We improve short text topic modeling with semantic complement meta-learning.
We complement the semantics for short documents by contrastive learning and design two alternatives for missing semantics prediction.
Meta-learning helps to optimize and predict the missing semantics.
Experiments on document classification, topic coherence, perplexity, and link prediction verify the effectiveness of our model.
One limitation is to assume a variable-length corpus with both long and short documents for semantic transfer.
We also assume the content is truthful.
If the corpus is infiltrated by fake news, those may appear in some topics.
A future work is exploring online learning to speed up training. Acknowledgement
This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award
No: AISG2-RP-2021-020).
Hady W. Lauw gratefully acknowledges the support by the Lee Kong Chian Fellowship awarded by Singapore Management University.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We aim to design a neural topic model to improve short text topic modeling with semantic complement meta-learning.
The Methodology section and Experiments section reflect these claims.
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not have theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] We do not have theoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include the source code, datasets, and a readme file for reproducibility.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] We specify all the training details at Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 3–8.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We correctly cite the papers that provide datasets and code of baseline models for experiments.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] All the datasets and baseline models are publicly available.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] All the datasets and baseline models are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
The data we are using does not contain personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We did not use crowdsourcing or conduct reseatch with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
We did not use crowdsourcing or conduct reseatch with human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
We did not use crowdsourcing or conduct reseatch with human subjects.