Meta-Complementing the Semantics of Short Texts in Neural Topic Models
Anonymous Author(s)
AffiliationAddress email
Abstract
Topic models infer latent topic distributions based on observed word co-occurrencesin a text corpus.
While typically a corpus contains documents of variable lengths,most previous topic models treat documents of different lengths uniformly, assuming that each document is sufficiently informative.
However, shorter documentsmay have only a few word co-occurrences, resulting in inferior topic quality.
Someother previous works assume that all documents are short, and leverage externalauxiliary data, e.g., pretrained word embeddings and document connectivity.
Orthogonal to existing works, we remedy this problem within the corpus itself byproposing a Meta-Complement Topic Model, which improves topic quality ofshort texts by transferring the semantic knowledge learned on long documentsto complement semantically limited short texts.
As a self-contained module, ourframework is agnostic to auxiliary data and can be further improved by flexiblyintegrating them into our framework.
Specifically, when incorporating documentconnectivity, we further extend our framework to complement documents withlimited edges.
Experiments demonstrate the advantage of our framework.
Introduction
Topic models help to understand the main themes within documents.
The quality of topic distributionof each document depends on sufficient word co-occurrences.
Real-word corpora contain documentsof variable lengths .
Academic papers vary from journal manuscripts to conference papers to extendedabstracts.
News articles could be headlines, short or full articles or detailed commentaries.
Fig.
1(a)illustrates an academic paper corpus where the distribution of document lengths exhibits a long-taildistribution.
Despite variable lengths (with different degrees of sufficiency of word co-occurrences),existing works, e.g., ProdLDA [31] and GATON [37], treat documents uniformly, resulting in inferiortopic quality for short texts.
Evidentially, Fig. 1(b) presents document classification accuracy on foursubsets of corpus with descending lengths.
Accuracies gradually drop as the length decreases.
Theinferior topic quality of short texts limits the overall performance of a topic model.
Challenges.
Most existing topic models optimize the learning process by averaging the generativelosses of different documents, without paying special attention to semantically limited short texts.
A few studies, e.g., OTLDA [15], take weighted summation of losses based on document lengths,they compute the weights by dividing the length of each document by the length of the whole corpus,which further deemphasizes the importance of short text modeling.
Thus, we seek to improve shorttext topic modeling within a variable-length corpus, without hurting topic quality of long documents.
To mitigate the scarcity of word co-occurrences in short texts, some works leverage auxiliaryknowledge to enhance topic modeling.
ETM [8] exploits pretrained word embeddings [27, 29] tocapture word similarities.
RTM [5] constructs a document network, e.g., paper citation network, toaggregate topics of connected documents.
However, they rely on the availability of auxiliary data.
[Figure]
+
Approach.
We propose M eta- C omplement T opic M odel (MCTM).
Since a corpus contains variablelengths of documents, we are motivated to learn the transferable semantic knowledge on long documents, and complement the semantics-scarce short texts and enhance the latter’s topic distributions.
As illustrated by Fig.
1(c), we learn a function g on long document A with abundant content, and useit to predict missing semantics (red box) for short document B to complement its content for a moreexpressive topic distribution.
Since meta-learning [10] is emerging to improve model performancewith few labeled observations, but no one explores its design in topic modeling.
We are thus motivatedto integrate it and optimize the proposed MCTM by a meta-learning objective.
Orthogonal to existing works relying on auxiliary data, our framework is self-contained, assumingonly in-corpus information, which offers a new direction to improve short text topic modeling.
Whenauxiliary data are available, our framework can be further improved by flexibly incorporating them.
In particular, when incorporating document network structure, we discover that document degreesalso exhibit a similar long-tail distribution, i.e., some structure-abundant documents link to sufficientneighbors as auxiliary, while others contain scarce links.
Besides textual semantic complement at Fig.1(c), we extend MCTM to further complement structural semantics on the document network.
Contributions.
First , we propose MCTM, which learns how to complement textual semantics bysemantic knowledge transfer.
Second , we derive two alternatives to implement missing semanticsprediction function g to capture document similarities.
Third , although agnostic to auxiliary data,
MCTM can also flexibly integrate them to further improve the performance.
We demonstrate ouradaptability by modeling pretrained word embeddings and document networks.
For the latter, weextend MCTM to further complement structural semantics.
Fourth , extensive experiments verify theeffectiveness of MCTM.
Code and datasets are submitted and will be released upon publication.
Problem Formulation and Preliminaries
We are given a corpus of N documents D
= { d i } Ni =1 .
Each document d ∈ R |V| is a vector in thevocabulary space V .
l d = (cid:80) w ∈V d w is the length of document d where d w is the word count of win d .
When word embeddings are available, we have H
= { h w } w ∈V where h w is the embedding ofword w .
Documents may link to others in a document network G =
{D , E} , with documents D andnetwork connectivity E
= { e ij } N i,j =1 .
If document d links to d ′ , e d,d ′ = 1 , otherwise e d,d ′ = 0 .
N ( d )is the set of neighbors of d .
We consider an undirected network, e d,d ′ =
e d ′ ,d .
Corpus D containsdocuments of variable lengths { l i } Ni =1 .
We introduce a hyperparameter L as the threshold whereshort documents are those with fewer observed words, D short = { d i | l i < L } , and long documentsare defined symmetrically, D long =
{ d i | l i ≥ L } .
We consider L as a predefined hyperparameterand leave other designs as future work.
See supplementary materials for a summary of math notations.
Given a variable-length corpus D (as well as word embeddings H and network E if observed) asinput, we aim to output topic distributions for documents where the topic quality of short documentsis improved, without hurting long documents.
Note that our goal is not to allow short texts to reachthe performance of long documents, but to improve short text topic modeling as much as possible.
Meta-Learning.
Meta-learning [10] optimizes globally shared parameters, a.k.a.
prior knowledge,over meta-training tasks, so as to rapidly adapt the model to previously unseen meta-testing tasks withonly a few observed data.
Since topic models generally learn topics by a content generative process,here we consider generating observed words for a document d as a task T d .
A meta-training task T dcorresponds to a training document d and consists of a support set and a query set, T d = {S d , Q d } .
Each set contains randomly sampled words from document d , such that support words and querywords are mutually exclusive, S d ∩ Q d = ∅ and S d ∪ Q d = d .
Meta-training has two steps: pretrained word embeddings words documents 𝐡 𝑤 1 𝐡 𝑤𝐡 𝑤𝛾 𝑑 ⨀ + 𝛽 𝑑 𝐔
𝑑 𝐦 𝑑 𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤𝛾 𝑑 ′ ⨀ + 𝛽 𝑑 ′ 𝐔 𝑑 ′ 𝐦 𝑑 ′
𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤𝐳 𝑤 1 𝐳 𝑤𝐳 𝑤cluster 𝑅 clustercluster… … 𝐖 𝑟= 𝐖 𝑟= 𝐦 𝑑 𝐦 𝑑 ′ doc 𝑑 pruned short 𝑑 ′ doc 𝑑 pruned short 𝑑 ′
[Figure]
min 𝜃 ℒ gen (𝜃,𝒮 𝑑 )=
𝜃 𝑑
[Figure]
𝜃 ∗
[Figure]
𝜃 ℒ({𝜃 𝑑 ,𝜇},𝒬 𝑑 )
[Figure]
𝐳 𝑑 ′ ∗ 𝐦 𝑑 ′ 𝐳
[Figure]
𝐳 𝑑 ∗ 𝐦 𝑑 𝐳 𝑑 𝐳
[Figure]
+ =
[Figure]
Local update.
Given a topic model f θ with parameter θ , f θ is first updated from the globally sharedparameter θ to document-specific local parameter θ d w.r.t.
loss on d ’s support words S d .θ d =
[Equation]
−
[Equation]
L ( θ
[Equation]
S d ) where T d = {S d , Q d } ∈ T tr .
(1) α is meta-learning rate, ∇ is gradient, L is loss function, T tr is a set of meta-training tasks (documents).2.
Global update.
After obtaining θ d for each document d , we compute the loss on query words
L ( θ d , Q d ) .
Together with other training tasks, we optimize the globally shared parameter θ .
[Equation]
T d ∈T tr
L ( θ − α ∇ θ L ( θ, S d ) , Q d ) .
(2) θ ∗ is the new globally shared parameter and will replace θ at Eq.
1–2 for the next iteration.
Afterconvergence, the final global parameter θ ∗ can easily be adapted to meta-testing tasks.
Meta-testing tasks T te are unseen test documents.
All the observed words are support words, T d =
S d = d .
During meta-testing, topic model f θ ∗ with optimized global parameter θ ∗ is updated w.r.t.
S d by Eq.
and obtain θ ∗ d .
The topic distribution of testing document is inferred by z d = f θ ∗ d ( d ) .
Methodology
We introduce Meta - C omplement T opic- M odel (MCTM) at Fig. 2.
Below we elaborate three components, graph convolutional encoder, missing semantics prediction, and meta-learning optimization.
3.1 Graph Convolutional Topic Encoding
We follow GATON [37] and first present a graph convolutional encoder f θ (Fig. 2(a)), which projectsdocuments to K -dimensional topic distributions.
We defer the discussion on short text modeling to thefollowing subsections.
Given a corpus D , considering documents and words as vertices, we constructa bipartite graph, the links represent word occurrences in the documents.
Since documents and wordspreserve heterogeneous feature spaces, we project them to the same topic space by learnable matrix,˜ z ( l +1) d
= W ( l +1)z ( l ) d , ˜ z
( l +1) w = W
( l +1)z ( l ) w , where d ∈ D , w ∈ V .
(3) l is the l -th convolutional layer.
z (0) d and z (0) w are inputs, i.e., Bag-of-Words and one-hot, respectively.
To differentiate the importance of words, we evaluate attention between document d and its observedsupport words at Eq.
4 and aggregate words at Eq.
5 where [ ·||· ] is concatenation.
[Equation]
w
[Equation] [Equation]
[Equation]
Symmetrically, we modify Eq.
4 and obtain a w,d , i.e., the attention between word w and documentsit appears in.
After symmetric aggregation at Eq.
5, we obtain z ( l +1) w for word w .
So far, we completethe convolution from l -th to ( l + 1) -th layer.
For simplicity, we summarize aggregation at Eq.
3–5 byz ( l +
[Equation]
d
[Equation]
We repeat Eq.
6 for maximum L layers and obtain K -dimensional topics
z d = z ( L ) d for document dand z w =
z ( L ) w for word w .
The complete encoder is Eq.
θ is the set of all encoding parameters.
[Equation]
3.2 Missing Semantics Prediction with Contrastive Learning
A short document with few words leaves some content poorly described, resulting in incomplete topicdistribution z d .
As a toy example, document B at Fig.
1(c) contains limited words, e.g., gene andclone, and leaves other contents, e.g., protein and cell, uncovered.
We aim to complement the topicsof short documents.
For a document d , regardless of long or short, we complement its semantics byz
[Equation]
We name m d ∈ R K missing semantics of document d .
If d is a long document with completesemantics, m d is a zero vector.
A function g µ predicts missing semantics at Eq.
9 with topicdistributions of d and its support words as inputs.
We will elaborate the design of function g µ shortly.
[Equation]
Contrastive Learning.
Long documents contain relatively more sufficient word co-occurrences thanshort documents.
Thus, we learn missing semantics prediction function g µ on long documents, andthen transfer the learned semantic knowledge to complement short documents.
On one hand, a longdocument d with enough content does not need semantic complement.
Thus we have below constraint
[Equation]
where d ∈ D
[Equation]
On the other hand, although we aim to transfer the semantic knowledge from long to short documents,there does not exist a one-to-one correspondence in corpus D .
As a result, we may transfer semanticsof a long document (e.g., machine learning concepts) to a short one describing completely distinctcontent (e.g., biology).
To overcome this limitation, we introduce another constraint with contrastiveobjective.
For a long document d with support words S d , we randomly hide a proportion of words tomimic a short document, denoted as d ′ with remaining observed words S d ′ ⊂ S d and length l d ′ < L .
For both long text d and its short version d ′ , we predict their missing semantics by Eq. 9, and obtainm d and m d ′ , respectively.
m d ′ should complement the previously hidden semantics, i.e., S d − S d ′ ,
[Equation]
where d ∈ D long .
[Equation]
Together with above Eq. 10, which forces z ∗ d = z d + m d to approach z d , Eq.
11 actually has
[Equation]
where d ∈ D long .
[Equation]
To summarize, we arrive at the following constraint loss.
[Equation]
where
[Equation]
σ
[Equation]
= 1 1+exp( − x ) is sigmoid function, and cos( · , · ) is cosine similarity.
We here use cosine similarity,mainly due to its superior performance on our datasets.
Besides cosine, other similarity metrics, suchas inner product and Euclidean distance, are also possible, depending on different datasets.
Missing Semantics Prediction.
We now define the function of missing semantics prediction g µ .
Given topic distributions of a document and its observed support words as input, a desirable functionshould aggregate them and output a single missing semantics vector.
We propose two alternatives.1.
GNN function.
The first is to implement g µ using a graph neural network, see Fig. 2(b).
[Equation]
A corpus D contains documents with diverse themes.
For example, some documents discuss machinelearning, while others describe biology.
However, the assumption of corpus-level shared parameter
U 1 and U 2 is not flexible to model diverse documents for missing semantics prediction, since differentdocuments may have distinct optimal parameters, which are sometimes even in opposing direction.
As a result, the predicted missing semantics m d centers its mass around the most frequent topics andleaves other distinct topics uncovered.
We seek to personalize U
and U 2 for each document d torecover the distinct missing semantics.
Formally, we introduce a function ϕ , which transforms U 1 and
U 2 to document-specific parameters U d, 1 and U d, 2 by scaling and shifting.
Taking U 1 as example,
[Equation]
(15) γ d and β d are document-specific vectors for scaling and shifting shared parameter U 1 .
⊙ is elementwise product.
[( x ) × K ] is a matrix with K identical column vector x . 1 is a vector of ones, ensuring the scaling matrix centers around one.
U d, 2 is similarly defined.
Eq.
15 allows each document d tohave its own parameters, while all documents still share the common knowledge.
Similar documentsscale and shift U 1 and U 2 to similar directions.
Different documents push them to distinct directions.
We define scaling γ d and shifting β d , parameterized by topics of document d and its support words. [Equation] [Equation] ¯ z
[Equation]
= 1 |S d | (cid:80) w ∈S d z w is the average of d ’s words.
In summary, we use Eq.
14 to predict d ’s missingsemantics, except that shared parameters
U
and U 2 are replaced by d -specific ones, U d, 1 and U d, 2 .2.
Clustering function.
We propose an alternative method by semantic clustering to recover distinctmissing semantics, Fig. 2(c).
Documents with similar content fall into related clusters, while uniquedocuments belong to different ones.
If we assign each cluster a set of parameters for missing semanticsprediction, similar documents would recover their own distinct topics.
Specifically, we introduce Rcentroids { c r } Rr =1 , each corresponding to one cluster.
Given topic distributions of document d and itssupport words, we first evaluate the assignment probability between document d and each cluster by
[Equation]
h 0 =
[ z d || ¯ z S d ] is the concatenation of z d and ¯ z S d .
We then assign parameters to each cluster,
[Equation] [Equation]
Therefore, related documents obtain similar clustering probabilities and missing semantics.
Above process is a flat clustering.
Our function can be extended to multiple clustering layers.
Eachlayer s consists of R ( s ) centroids.
After obtaining the output from previous layer, h s − 1 , we repeat Eq.17–18 by replacing h 0 with h s − 1 , and obtain the output of the current layer s , i.e., h s .
For maximum
S layers, we get m d = h S .
We leave adaptive learning of number of clusters R as future work.
3.3 Probabilistic Decoding with Meta-Learning Optimization
After semantic complement, we obtain { z ∗ d } d ∈D at Sec. 3.2.
We use Z V = (cid:2)
z w 1 ; z w 2 ; ... ; z w |V| (cid:3) ∈
R K ×|V| to represent topic-word distribution, each column z w is topic distribution of a word w , andeach row is to the distribution of a topic over the vocabulary.
As in previous topic models [3, 31], wegenerate the observed support words S d by ˆ d S d = σ ( Z V z ∗ d ) .
Compared to the ground-truth supportwords d S d , we follow [37] and obtain generative loss L gen = ||
d S d − ˆ d S d
|| 22 .
However, this lossrequires inefficient computation over the whole vocabulary.
We instead use negative sampling [27].
[Equation]
M is the number of negative samples, and Pr n ( w ) is a noise distribution over vocabulary.
ˆ d w =σ ( z ∗⊤ d z w ) .
d w = 1 if w ∈ S d , otherwise d w = 0 .
In addition, if d is a long document d ∈ D long , wealso created its corresponding pruned short version d ′ with a subset of support words S d ′ ⊂ S d at Sec.3.2.
Although we do not observe the complete support words for d ′ , its complete topic distributionz ∗ d ′ after semantic complement at Eq.
8 should be able to generate the complete support words S d .
Therefore, together with semantic complement constraint at Eq. 13, we arrive at the complete loss.
[Equation]
where L gen ( d ) =
L gen ( d ) + I ( d ∈ D long ) λ gen L gen ( d ′ ) ,
L con ( d ) = I ( d ∈ D long )
L con ( d ) .
L gen ( d ) is generative loss, consisting of document d in the corpus d ∈ D
= D long ∪ D short and thecorresponding pruned short version d ′ (if d ∈ D long ).
I ( d ∈ D long ) =
if d ∈ D long , otherwise 0 .
L reg ( θ ) is L2 regularizer for encoding parameters θ .
λ gen and λ reg are hyperparameters.
Optimization.
Finally, with the objective of meta-learning at Sec. 2, we reach the optimization:
Local update.
Given topic encoder f θ defined at Sec.
3.1, we optimize encoding parameter θ w.r.t.the generative loss L gen ( d ) on support words S d by Eq.
and obtain θ d for each document.2.
Global update.
With encoding parameter θ d , we compute the overall loss L ( d ) on query words
Q d and optimize all parameters
Φ = { θ, µ } , including encoder parameters θ and parameters µ ofmissing semantics prediction function g µ .
α 1 and α 2 are local and global learning rate, respectively.
[Equation]
With new parameter Φ ∗ =
{ θ ∗ , µ ∗ } , we substitute it for Φ for the next iteration.
In contrast toprevious topic models that generate the content of given documents only, we further create the shortversion of long documents for semantic complement, and jointly optimize them.
See supplementarymaterials for learning algorithm.
3.4 Extensions with Auxiliary Data
MCTM with Pretrained Word Embeddings.
Pretrained word embeddings [27, 29] encode wordsimilarity.
As in previous works [7, 43], we incorporate them into topic-word distribution Z V =
[ z k,w ]to improve topic modeling.
We introduce topic embedding { t k } Kk =1 and evaluate cosine similaritybetween topic k and word w by cos( t k , h w ) .
We then combine it with topic-word distribution byz ′ k,w = 12 ( z k,w + cos( t k , h w )) and obtain a new topic-word distribution Z ′V =
[ z ′ k,w ] for decoding.
MCTM with Document Network.
A document network (e.g., citation network) reveals semanticsimilarities between connected documents (cited papers discuss related research).
A document’sdegree or number of links exhibits a long-tail distribution.
Some link to many neighbors, others to afew.
Previously focus was on textual semantic complement.
We extend MCTM to model structuralsemantic complement for link-scarce documents.
We consider generating both observed words andneighbors as a task T d .
As for words, we split the neighbors N ( d ) of a document d into support andquery neighbors T d = {S d , S N ( d ) , Q d , Q N ( d ) } , S d and S N ( d ) denote support words and neighbors, 204 respectively, and ditto for query sets.
We correspondingly extend three modeling components.1.
Encoding.
Previously, we inferred topic distribution z d of document d by its textual words at Eq. 6.
Here, we extend this process by designing a structural convolutional module.
[Equation]
The topic distribution from encoder f θ is z d := 12 ( z d + κ d ) , with both texts S d and structure S N ( d ) .2.
Semantics complement.
For a long document d , in addition to randomly hiding some words, wealso drop some neighbors and create a pruned version d ′ .
Now the missing semantics m d shouldcontain both textual and structural information.
For GNN function g µ , we extend Eq.
14 bym d
[Equation]
[Equation]
Scaling has extra input, γ d = tanh ( W γ z d + W ′ γ ¯ z S d + W ′′ γ ¯ z S N ( d ) ) , ditto for shifting.
¯ z S N ( d ) =1 |S N ( d ) | (cid:80) d ′ ∈S N ( d )
z d ′ .
For Clustering g µ , we extend input by h 0 =
[ z d || ¯ z S d
|| ¯ z S N ( d ) ] .3.
Decoding with meta-learning.
In addition to generating support words using complemented z ∗ d , wealso generate support neighbors.
The generative loss is similar to Eq.
19 except that i ) we replaced w with e d,d ′ , the ground-truth link between d and d ′ ; ii ) ˆ e d,d ′ =
σ ( z ∗⊤ d z ∗ d ′ ) .
Finally, meta-learninglearns how to accurately predict missing semantics m d for both textual and structural complement.
Experiments
[Table] The goal of experiments is to evaluate if our model MCTM can improveshort text topic modeling through evaluative tasks, e.g., document classification, link prediction, topic analysis.
Datasets.
Since our model is flexibleto incorporate auxiliary data, we rely [Table] on with for experiments.
Cora [24] is corpus of academic papers with citations as links.
We created twoindependent datasets, Machine Learning ( ML ) and Programming Language ( PL ).
In addition, HEPTH [21] is a corpus of Physics papers with their citations.
Web [20] is a Web page hyperlink networkwhere each page is a news article, and the hyperlinks connect related articles.
See Table 1 for details.
Baselines.
Since our model has three variants, i.e., MCTM with plain texts, with auxiliary wordembeddings, and with auxiliary document networks, we correspondingly compare to three categoriesof baselines.
i )
Topic models with plain texts , ProdLDA [31], WLDA [28], and GATON [37].
Theymodel all documents uniformly without dealing with short texts.
We compare to them and show theadvantage of MCTM on improving short texts.
ii ) Topic models with word embeddings , ETM[8] and NSTM [43].
Since our model is built on top of GATON, we also compare to GATON withword embeddings, denoted as GATON+WE.
By comparing to them, we verify the effectivenessof semantic complement meta-learning to further improve topic quality.
iii )
Topic models withdocument networks , RTM [5], Adjacent-Encoder [40], LANTM [35], and GATON+DN, which isthe extension of GATON with document networks.
For document network scenario, we include agraph embedding model, meta-tail2vec [23], which uses meta-learning to improve nodes with lowdegrees, but is not a topic model and ignores variable lengths of node attributes, i.e., texts.
We set L = 2 convolutional layers. λ gen = 2 and λ reg = 0 .
Number of negative samples M = 5and number of semantic clusters R = 5 .
L is the median length of the corpus.
Local and globallearning rates are α 1 = 0 .
001 and α 2 = 0 .
We use 300D Glove embeddings.
We experimentwith 5 independent runs, report mean and std.dev.
All the experiments were done on Linux server witha Tesla K80 GPU with 11441MiB. Code and datasets are submitted, to be released upon publication.
4.1 Quantitative Evaluation
Document Classification.
Documents from the same category discuss related topics.
As in LDA [3],we conduct classification to evaluate topic quality.
We split 80% documents for training (10% are forvalidation).
Labels are not involved during training.
After convergence, we train a k NN classifier( k = 5 ) [2] with training documents and predict the labels of test documents.
We set 64 topics.
Wecompare our models within each category of baselines and report classification accuracy at Table 2.
We split the test set into four subsets with descending document length and report the result of bothoverall test and each subset.
0-25% at Table 2 means the subset with the longest 25% test documents.
MCTM (G) and MCTM (C) denote our model with GNN and Clustering function, respectively.
Weuse “*” to represent statistically significant improvement with paired t-test at 0.05 significance level.
Our models significantly outperform baselines within each category.
We outperform GATON, the bestbaseline in the plain text and word embedding category, since textual semantic complement improvesshort texts, and the overall test set is also improved.
MCTM (G) performs slightly better than MCTM(C), potentially because GNN recognizes importance of words with attention, while the Clusteringfunction takes simple average.
To show our models indeed improve short text quality, we present theimprovement of MCTM (G) over both GATON and the best baseline.
Our performance generallyimproves more as the length decreases, which verifies the advantage of semantic complement.
Topic Coherence.
Each row of topic-word distribution Z V ∈ R K ×|V| is the distribution of one topicover the vocabulary, and the key words of this topic correspond to the highest values on this row. As
[Table] [Table] in ProdLDA [31], we evaluate the coherence of key words by Google Web 1T 5-gram Version 1 [9],with NPMI as metric.
Table 3 (left) summarizes the results.
Topic-word distribution Z V is modelparameter and is separate from document length, thus we can not report results of different lengths.
LANTM cannot run on large dataset Web.
Meta-tail2vec is not a topic model, thus is excluded.
Overall, our models outperform baselines on ML and PL and are competitive with the best baselineon HEP-TH and Web.
This indicates that our models at least do not hurt topic coherence, but cansignificantly improve other tasks, e.g., classification.
Compared to GATON, our models significantlyimprove it, verifying the advantage of semantic complement.
To understand what topics our modelscapture, we randomly present two topics with top-5 key words at Table 4.
MCTM (G) capturesstatistics and computational genetics , while MCTM (C) reveals scalability and geometric learning .
[Table]
Perplexity.
Topic model should generalizeto unseen documents.
Following [3], weevaluate perplexity.
Since perplexity is exponential and varies much w.r.t.
its power,we report its power, − logPr
( D test ) (cid:80) d ∈D test l d (loweris better).
Table 3 (right) reveals that ourmodels generate high likelihood to unseendocuments, which we attribute to semanticcomplement meta-learning module.
Link Prediction.
A good model should infer similar topics for potentially linked documents.
Sincewe model document network as auxiliary data, we follow RTM [5] and predict links.
As in [40], theprobability of a link is p ( e d,d ′ )
∝ exp( −|| z ∗ d −
z
∗
d
′ || 22 ) .
We predict the links within test documentsand compare with the ground-truth links with AUC as metric.
Since only the third category (networkmodels) incorporate links, we mainly compare the MCTM+DN version to these network baselines.
Table 5 indicates that our models predict links more accurately than baselines.
The comparison tonetwork baselines demonstrates the effectiveness of both textual and structural semantic complement.
4.2 Model Analysis
Effect of Semantic Complement.
To see if semantic complement helps short texts, we remove itfrom the complete model and present classification accuracy at Table 6 (left).
We show the plaintext results, and put the auxiliary data version in supplementary.
Models do better with semantic [Table] complement than without.
The accuracy on short subset declines more than the overall test set, whichreveals that semantic complement improves short texts, and removing it hurts short documents more.
Effect of Meta-Learning.
To analyze if meta-learning benefits the optimization with a few observedwords, we replace it with the commonly used stochastic gradient descend.
Table 6 (middle) showsthat result drops more on short subset than on the overall test set, which verifies that meta-learning isgood at optimization with only a few observed words and improves short text modeling. We furtherremove both semantic complement and meta-learning, and report the result at Table 6 (right), whichpresents the worst accuracy. This observation further verifies that both components are important.
Effect of Scaling and Shifting.
The GNN version of our model uses scaling-and-shifting method torecover distinct topics.
To test its usefulness, we disregard it and summarize classification accuracyon ML at Table 7 (left).
Removing scaling and shifting leads to worse performance, since we can notpersonalize shared parameters to each document to recover its distinct missing topics for complement. [Table]
Effect of Clustering.
We set the number ofclusters R to 1, all documents share the sameparameters for missing semantics predictionwith no clustering.
Table 7(right) concludesthat clustering is helpful to share commonsemantics for related documents and distinguish documents of different clusters.
Related Work
Traditional topic models are graphical models [3, 13].
Recent ones are neural models [26], mainlyvariants of VAE [17].
ProdLDA [31] and DVAE [4] analyze Dirichlet prior.
WHAI [41] looks into
Gamma prior.
The variational divergence metric also attracts attention.
WLDA [28] and WAE [32]apply mean maximum discrepancy and generative adversarial network [12].
KATE [6] increasestopic sparsity.
More recently, topic models are built with graph neural networks [18], e.g., GATON[37], GraphBTM [44], etc.
They average losses of all documents regardless of lengths.
MCTMdesigns semantic complement to predict missing semantics for short texts to improve topic quality.
To alleviate scarce word co-occurrences of short texts, existing works leverage auxiliary data, suchas word embeddings [27, 29].
Graphical models include GLDA [7], GPUMM [22], LCTM [14],
MetaLDA [42], WDL [36], and OTLDA [15].
Neural models include ETM [8] and NSTM [43].
Another category improves short text modeling with document networks, e.g., citation network.
RTM[5] and NetPLSA [25] are graphical models.
Neural models include NRTM [1], Adjacent-Encoder[40], and LANTM [35].
In contrast, our semantic complement module is agnostic to auxiliary data.
Here, we also briefly review meta-learning works.
One category is metric-based, e.g., ProtoNet [30]and MatchingNet [33], which learn a metric function over tasks.
Gradient-based meta-learning worksoptimize model parameters for quick adaptation to new tasks [10, 16, 11, 34, 19, 38, 39].
Conclusion
We improve short text topic modeling with semantic complement meta-learning.
We complementthe semantics for short documents by contrastive learning and design two alternatives for missingsemantics prediction.
Meta-learning learns how to optimize and predict the missing semantics.
Onelimitation is to assume a variable-length corpus with both long and shot documents for semantictransfer.
We also assume the content is truthful.
If the corpus is infiltrated by fake news, those mayappear in some topics.
As future work, we will explore online learning to speed up training.
Checklist
For all authors...
(a ) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] We aim to design a neural topic model to improve
short text topic modeling with semantic complement meta-learning.
The Methodology
section and Experiments section reflect these claims.
(b) Did you describe the limitations of your work?
[Yes] See Line 337 at Section 6.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Line338 at Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A] We do not
have theoretical results.
(b) Did you include complete proofs of all theoretical results?
[N/A] We do not havetheoretical results.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include thesource code, datasets, and a readme file for reproducibility.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We specify all the training details at Line 243-247.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 2–7.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Line 246-247. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We correctly cite thepapers that provide datasets and code of baseline models for experiments. [Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
All the datasets and baseline models are publicly available.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table]
using/curating?
[N/A] All the datasets and baseline models are publicly available. [Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
The data we are using does not contain
[Table]
personally identifiable information or offensive content.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
We did not use crowdsourcing or conduct reseatch with human
[Table]
subjects.
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
We did not use crowdsourcing or conduct
[Table]
reseatch with human subjects.
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
We did not use crowdsourcing or conduct
[Table]
reseatch with human subjects.
[Table]