textTOvec : D EEP C ONTEXTUALIZED N EURAL A UTOREGRESSIVE T OPIC M ODELS OF L ANGUAGE WITH D ISTRIBUTED C OMPOSITIONAL P RIOR
A BSTRACT
We address two challenges of probabilistic topic modelling in order to better estimate the probability of a word in a given context, i e., P ( word | context ) : (1) No language structure in context : Probabilistic topic models ignore word order by summarizing a given context as a “bag-of-word” and consequently the semantics of words in the context is lost.
In this work, we incorporate language structure by combining a neural autoregressive topic model (TM) (e g., DocNADE) with a LSTM based language model (LSTM-LM) in a single probabilistic framework.
The LSTM-LM learns a vector-space representation of each word by accounting for word order in local collocation patterns, while the TM simultaneously learns a latent representation from the entire document.
In addition, the LSTM-LM models complex characteristics of language (e g., syntax and semantics), while the TM discovers the underlying thematic structure in a collection of documents.
We unite two complementary paradigms of learning the meaning of word occurrences by combining a topic model and a language model in a uniﬁed probabilistic framework, named as ctx-DocNADE.
(2) Limited context and/or smaller training corpus of documents : In settings with a small number of word occurrences (i e., lack of context) in short text or data sparsity in a corpus of few documents, the application of TMs is challenging.
We address this challenge by incorporating external knowledge into neural autoregressive topic models via a language modelling approach: we use word embeddings as input of a LSTM-LM with the aim to improve the word-topic mapping on a smaller and/or short-text corpus.
The proposed DocNADE extension is named as ctx-DocNADEe.
We present novel neural autoregressive topic model variants coupled with neural language models and embeddings priors that consistently outperform state-of-theart generative topic models in terms of generalization (perplexity), interpretability (topic coherence) and applicability (retrieval and classiﬁcation) over 7 long-text and 8 short-text datasets from diverse domains.
I NTRODUCTION
Probabilistic topic models, such as LDA (Blei et al., 2003), Replicated Softmax (RSM) (Salakhutdinov & Hinton, 2009) and Document Neural Autoregressive Distribution Estimator (DocNADE) variants (Larochelle & Lauly, 2012; Zheng et al., 2016; Lauly et al., 2017; Gupta et al., 2019) are often used to extract topics from text collections, and predict the probabilities of each word in a given document belonging to each topic.
Subsequently, they learn latent document representations that can be used to perform natural language processing (NLP) tasks such as information retrieval (IR), document classiﬁcation or summarization.
However, such probabilistic topic models ignore word order and represent a given context as a bag of its words, thereby disregarding semantic information.
To motivate our ﬁrst task of extending probabilistic topic models to incorporate word order and language structure, assume that we conduct topic analysis on the following two sentences:
Bear falls into market territory and Market falls into bear territory
Deal with stock index falls
Brace for market share drops
Cosine similarity in Word
Embedding space 0.
Coarse Granuality: Global View
Topic due to Word distributions across documents b illi on r e c o r d c r ed i t do ll a r f a ll t r ade e x po r t s a l e m one
y m a r k e t s rise fall falling falls drop tumble
Fine Granuality: Local View
Nearest Neighbors of fall
[Figure]
o
[Figure]
When estimating the probability of a word in a given context (here: P (“ bear ” | context ) ), traditional topic models do not account for language structure since they ignore word order within the context and are based on “bag-of-words” (BoWs) only.
In this particular setting, the two sentences have the same unigram statistics, but are about different topics.
On deciding which topic generated the word “bear” in the second sentence, the preceding words “market falls” make it more likely that it was generated by a topic that assigns a high probability to words related to stock market trading , where “bear territory” is a colloquial expression in the domain.
In addition, the language structure (e.g., syntax and semantics) is also ignored.
For instance, the word “bear” in the ﬁrst sentence is a proper noun and subject while it is an object in the second.
In practice, topic models also ignore functional words such as “into”, which may not be appropriate in some scenarios.
Recently, Peters et al.
(2018) have shown that a deep contextualized LSTM-based language model (LSTM-LM) is able to capture different language concepts in a layer-wise fashion, e.g., the lowest layer captures language syntax and topmost layer captures semantics.
However, in LSTM-LMs the probability of a word is a function of its sentence only and word occurrences are modeled in a ﬁne granularity .
Consequently, LSTM-LMs do not capture semantics at a document level.
To this end, recent studies such as TDLM (Lau et al., 2017), Topic-RNN (Dieng et al., 2016) and TCNLM (Wang et al., 2018) have integrated the merits of latent topic and neural language models (LMs); however, they have focused on improving LMs with global (semantics) dependencies using latent topics.
Similarly, while bi-gram LDA based topic models (Wallach, 2006; Wang et al., 2007) and n-gram based topic learning (Lauly et al., 2017) can capture word order in short contexts, they are unable to capture long term dependencies and language concepts.
In contrast, DocNADE variants (Larochelle & Lauly, 2012; Gupta et al., 2019) learns word occurrences across documents i.e., coarse granularity (in the sense that the topic assigned to a given word occurrence equally depends on all the other words appearing in the same document); however since it is based on the BoW assumption all language structure is ignored.
In language modeling, Mikolov et al. (2010) have shown that recurrent neural networks result in a signiﬁcant reduction of perplexity over standard n-gram models.
Contribution 1 : We introduce language structure into neural autoregressive topic models via a
LSTM-LM, thereby accounting for word ordering (or semantic regularities), language concepts and long-range dependencies.
This allows for the accurate prediction of words, where the probability of each word is a function of global and local (semantics) contexts, modeled via DocNADE and LSTMLM, respectively.
The proposed neural topic model is named as contextualized - Document Neural
Autoregressive Distribution Estimator ( ctx-DocNADE ) and offers learning complementary semantics by combining joint word and latent topic learning in a uniﬁed neural autoregressive framework.
For instance, Figure 1 (left and middle) shows the complementary topic and word semantics, based on TM and LM representations of the term “fall”.
Observe that the topic captures the usage of “fall” in the context of stock market trading , attributed to the global (semantic) view.
While this is a powerful approach for incorporating language structure and word order in particular for long texts and corpora with many documents, learning from contextual information remains challenging in settings with short texts and few documents, since (1) limited word co-occurrences or little context (2) signiﬁcant word non-overlap in such short texts and (3) small training corpus of documents lead to little evidence for learning word co-occurrences.
However, distributional word representations (i.e. word embeddings) (Pennington et al., 2014) have shown to capture both the semantic and syntactic relatedness in words and demonstrated impressive performance in NLP tasks.
For example, assume that we conduct topic analysis over the two short text fragments: Deal with stock index falls and Brace for market share drops .
Traditional topic models v
La t en t
T op i cs vv i-
i v i ctx-DocNADEe
v i+ v Dv D
W
Coarse Granuality: Global View
Word occurrences across document ...
h
i DN h
i LM v
i h
i x
i+1 x i x i-1 x
Fine Granuality: Local View Word occurences in collocation patterns x
LSTM-LM DocNADE (DN) =
W or = W + E Embedding layer ...
c i = x 1 , x 2 ,..., x iv
DocNADE
U v iv
i v D vvv D
v i
hhh
i h D
U
W ...
[Figure]
with “BoW” assumption will not be able to infer relatedness between word pairs such as ( falls , drops ) due to the lack of word-overlap and small context in the two phrases.
However, in the distributed embedding space, the word pairs are semantically related as shown in Figure 1 (left).
Related work such as Sahami & Heilman (2006) employed web search results to improve the information in short texts and Petterson et al.
(2010) introduced word similarity via thesauri and dictionaries into LDA.
Das et al. (2015) and Nguyen et al.
(2015) integrated word embeddings into LDA and Dirichlet Multinomial Mixture (DMM) (Nigam et al., 2000) models.
Recently, Gupta et al.
(2019) extends DocNADE by introducing pre-trained word embeddings in topic learning.
However, they ignore the underlying language structure, e.g., word ordering, syntax, etc.
In addition, DocNADE and its extensions outperform LDA and RSM topic models in terms of perplexity and IR.
Contribution 2 : We incorporate distributed compositional priors in DocNADE: we use pre-trained word embeddings via LSTM-LM to supplement the multinomial topic model (i.e., DocNADE) in learning latent topic and textual representations on a smaller corpus and/or short texts.
Knowing similarities in a distributed space and integrating this complementary information via a LSTM-LM, a topic representation is much more likely and coherent.
Taken together, we combine the advantages of complementary learning and external knowledge, and couple topic- and language models with pre-trained word embeddings to model short and long text documents in a uniﬁed neural autoregressive framework, named as ctx - DocNADEe .
Our approach learns better textual representations, which we quantify via generalizability (e.g., perplexity), interpretability (e.g., topic extraction and coherence) and applicability (e.g., IR and classiﬁcation).
To illustrate our two contributions , we apply our modeling approaches to 7 long-text and 8 short-text datasets from diverse domains and demonstrate that our approach consistently outperforms state-ofthe-art generative topic models.
Our learned representations, result in a gain of: (1) 4 .
6 % (.790 vs .755) in topic coherence, (2) 6 .
5 % (.615 vs .577) in precision at retrieval fraction 0.02, and (3) 4 .
4 % (.662 vs .634) in F 1 for text classiﬁcation, averaged over 6 long-text and 8 short-text datasets.
When applied to short-text and long-text documents, our proposed modeling approaches generate context ualized topic vectors , which we name textTOvec .
The code is available at https: //github.com/pgcool/textTOvec .
N EURAL A UTOREGRESSIVE T OPIC M ODELS
Generative models are based on estimating the probability distribution of multidimensional data, implicitly requiring modeling complex dependencies.
Restricted Boltzmann Machine (RBM) (Hinton et al., 2006) and its variants (Larochelle & Bengio, 2008) are probabilistic undirected models of binary data.
RSM (Salakhutdinov & Hinton, 2009) and its variants (Gupta et al., 2018) are generalization of the RBM, that are used to model word counts.
However, estimating the complex probability distribution of the underlying high-dimensional observations is intractable.
To address this challenge, NADE (Larochelle & Murray, 2011) decomposes the joint distribution of binary observations into autoregressive conditional distributions, each modeled using a feed-forward network.
Unlike for RBM/RSM, this leads to tractable gradients of the data negative log-likelihood.
2. D OCUMENT N EURAL A UTOREGRESSIVE T OPIC M ODEL (D OC NADE)
An extension of NADE and RSM, DocNADE (Larochelle & Lauly, 2012) models collections of documents as orderless bags of words (BoW approach), thereby disregarding any language structure.
In other words, it is trained to learn word representations reﬂecting the underlying topics of the documents only, ignoring syntactical and semantic features as those encoded in word embeddings (Bengio et al., 2003; Mikolov et al., 2013; Pennington et al., 2014; Peters et al., 2018).
DocNADE (Lauly et al., 2017) represents a document by transforming its BoWs into a sequence v =
[ v 1 , ..., v D ] of size D , where each element v i ∈ { 1 , 2 , ..., K } corresponds to a multinomial observation (representing a word from a vocabulary of size K ).
Thus, v i is the index in the vocabulary of the i th word of the document v .
DocNADE models the joint distribution p ( v ) of all words v i by decomposing it as p ( v ) =
(cid:81)
Di =1 p ( v i | v <i ) , where each autoregressive conditional p ( v i | v <i ) for the word observation
v i is computed using the preceding observations v <i ∈ { v 1 , ..., v i − 1 } in a feed-forward neural network for i ∈ { 1 , ...
D } , h DNi ( v <i ) =
g ( e + (cid:88) k
<i
W : ,v k ) and p ( v i =
w | v <i ) =
exp( b w + U w, : h DNi ( v <i )) (cid:80) w (cid:48) exp( b w (cid:48) + U w (cid:48) , : h DNi ( v <i ))
(1) where g ( · ) is an activation function,
U ∈ R K × H is a weight matrix connecting hidden to output, e ∈ R H and b ∈ R K are bias vectors,
W ∈ R H
× K is a word representation matrix in which a column W : ,v i is a vector representation of the word v i in the vocabulary, and H is the number of hidden units (topics).
The log-likelihood of any document v of any arbitrary length is given by:
L DN ( v ) = (cid:80) Di =1 log p ( v i | v <i ) .
Note that the past word observations v <
i are orderless due to
BoWs, and may not correspond to the words preceding the i th word in the document itself.
Algorithm 1 Computation of v model h
i p ( v i | v
DocNADE h DNi ← g ( a )
[Table]
h
[Table]
h
[Table]
ctx
[Table] h [Table] ( c
[Table]
[Table]
[Table]
h
[Table]
+ λ h
[Table]
ctx
[Table]
h
[Table] [Table] 2. D EEP C ONTEXTUALIZED D OC NADE WITH D ISTRIBUTIONAL S EMANTICS We propose two extensions of the DocNADE model: (1) ctx-DocNADE : introducing language structure via LSTM-LM and (2) ctx-DocNADEe : incorporating external knowledge via pre-trained word embeddings E , to model short and long texts.
The uniﬁed network(s) account for the ordering of words, syntactical and semantic structures in a language, long and short term dependencies, as well as external knowledge, thereby circumventing the major drawbacks of BoW-based representations.
Similar to DocNADE, ctx-DocNADE models each document v as a sequence of multinomial ob- servations.
Let [ x 1 , x 2 , ..., x N ] be a sequence of N words in a given document, where x i is rep- resented by an embedding vector of dimension, dim .
Further, for each element v i ∈ v , let c i =
[ x 1 , x 2 , ..., x i − 1 ] be the context (preceding words) of i th word in the document.
Unlike in DocNADE, the conditional probability of the word v i in ctx-DocNADE
or ctx-DocNADEe) is a function of two hidden vectors: h DN i ( v <i ) and h LM i ( c i ) , stemming from the DocNADE-based and
[Table]
LSTM-based components of ctx-DocNADE, respectively:
[Equation]
where h DN i ( v <i ) is computed as in DocNADE (equation 1) and λ is the mixture weight of the LM component, which can be optimized during training (e.g., based on the validation set).
The second term h LMi is a context-dependent representation and output of an LSTM layer at position i −over input sequence c i , trained to predict the next word v i .
The LSTM offers history for the i th word via modeling temporal dependencies in the input sequence, c i .
The conditional distribution for each word v i is estimated by equation 2, where the uniﬁed network of DocNADE and LM combines global and context-dependent representations.
Our model is jointly optimized to maximize the pseudo log likelihood, log p ( v ) ≈ (cid:80)
Di =1 log p ( v i | v <i ) with stochastic gradient descent.
See
Larochelle & Lauly (2012) for more details on training from bag of word counts.
In the weight matrix W of DocNADE (Larochelle & Lauly, 2012), each row vector W j, : encodes topic information for the j th hidden topic feature and each column vector W : ,v i is a vector for the word v i .
To obtain complementary semantics, we exploit this property and expose W to both global and local inﬂuences by sharing W in the DocNADE and LSTM-LM componenents.
Thus, the embedding layer of LSTM-LM component represents the column vectors.
ctx-DocNADE , in this realization of the uniﬁed network the embedding layer in the LSTM component is randomly initialized.
This extends DocNADE by accounting for the ordering of words and language concepts via context-dependent representations for each word in the document.
ctx-DocNADEe , the second version extends ctx-DocNADE with distributional priors, where the embedding layer in the LSTM component is initialized by the sum of a pre-trained embedding matrix E and the weight matrix W .
Note that W is a model parameter; however E is a static prior.
Algorithm 1 and Table 1 show the log p ( v ) for a document v in three different settings: DocNADE , ctx-DocNADE and ctx-DocNADEe .
In the DocNADE component, since the weights in the matrix W are tied, the linear activation a can be re-used in every hidden layer and computational complexity reduces to O ( HD ) , where H is the size of each hidden layer.
In every epoch, we run an LSTM over the sequence of words in the document and extract hidden vectors h LMi , corresponding to c i for every target word v i .
Therefore, the computational complexity in ctx-DocNADE or ctx-DocNADEe is O ( HD + N ) , where N is the total number of edges in the LSTM network (Hochreiter & Schmidhuber, 1997; Sak et al., 2014).
The trained models can be used to extract a textTOvec representation, i.e., h ( v ∗ ) =
h DN ( v ∗ )
+ λ h LM ( c ∗ N +1 ) for the text v ∗ of length D ∗ , where h DN ( v ∗ ) =
g ( e + (cid:80) k
≤ D ∗ W : ,v k ) and h LM ( c ∗ N +1 ) =
LSTM ( c ∗ N +1 , embedding = W
or ( W + E )) .
ctx-DeepDNEe : DocNADE and LSTM can be extended to a deep, multiple hidden layer architecture by adding new hidden layers as in a regular deep feed-forward neural network, allowing for improved performance.
In the deep version, the ﬁrst hidden layer is computed in an analogous fashion to DocNADE variants (equation 1 or 2).
Subsequent hidden layers are computed as: h DNi,d ( v <i )
=
g ( e d + W i,d · h i,d − 1 ( v <i )) or h LMi,d ( c i ) =
deepLSTM ( c i , depth = d ) for d = 2 , ...n , where n is the total number of hidden layers (i.e., depth) in the deep feed-forward and LSTM networks.
For d = 1 , the hidden vectors h DNi, 1 and h LMi, 1 correspond to equations 1 and 2.
The conditional p ( v i = w | v <i ) is computed using the last layer n , i.e., h i,n = h DN i,n + λ h LM i,n .
[Table]
IR
F
IR
F
IR
F
IR
F
IR
[Table]
IR
F
IR
F
IR
[Table]
E VALUATION
We apply our modeling approaches (in improving topic models,
i
[Table]
e, DocNADE using language concepts from LSTM-LM) to 8 short-text and 7 long-text datasets of varying size with single/multiclass labeled documents from public as well as industrial corpora.
We present four quantitative measures in evaluating topic models: generalization (perplexity), topic coherence, text retrieval and categorization.
See the appendices for the data description and example texts.
Table 2 shows the data statistics, where 20NS and R21578 signify 20NewsGroups and Reuters21578, respectively.
Baselines : While, we evaluate our multi-fold contributions on four tasks: generalization (perplexity), topic coherence, text retrieval and categorization, we compare performance of our proposed models ctx-DocNADE and ctx-DocNADEe with related baselines based on: (1) word representation: glove (Pennington et al., 2014), where a document is represented by summing the embedding vectors of it’s words, (2) document representation: doc2vec (Le & Mikolov, 2014), (3) LDA based BoW TMs: ProdLDA (Srivastava & Sutton, 2017) and SCHOLAR 1 (Card et al., 2017) (4) neural BoW TMs: DocNADE and NTM (Cao et al., 2015) and , (5) TMs, including pre-trained word embeddings: Gauss-LDA ( GaussianLDA )
(Das et al., 2015), and glove-DMM ,
glove-LDA
(Nguyen et al., 2015).
(6) jointly 2 trained topic and language models: TDLM (Lau et al., 2017),
Topic-RNN (Dieng et al., 2016) and TCNLM (Wang et al., 2018).
[Table] [Table]
Experimental Setup : DocNADE is often trained on a reduced vocabulary (RV) after pre-processing (e.g., ignoring functional words, etc.); however, we also investigate training it on full text/vocabulary (FV)
(Table 2) and compute document representations to perform different evaluation tasks.
The FV setting preserves the language structure, required by LSTM-LM
[Table]
and allows a fair comparison of
DocNADE+FV and ctx-DocNADE variants.
We use the glove embedding of 200 dimensions.
All the baselines and proposed models ( ctx-DocNADE , ctx-DocNADEe and ctx-DeepDNEe ) were run in the FV setting over 200 topics to quantify the quality of the learned representations.
To better initialize the complementary learning in ctx-DocNADEs, we perform a pre-training for 10 epochs with λ set to 0 .
See the appendices for the experimental setup and hyperparameters for the following tasks, including the ablation over λ on validation set.
[Table]
We run TDLM 3 (Lau et al., 2017) for all the short-text datasets to evaluate the quality of representations learned in the spare data setting.
For a fair comparison, we set 200 topics and hidden size, and initialize with the same pre-trained word embeddings (i.e., glove) as used in the ctx-DocNADEe.
3. G ENERALIZATION : P ERPLEXITY (PPL)
To evaluate the generative performance of the topic models, we estimate the log-probabilities for the test documents and compute the average held-out perplexity ( PPL ) per word as, PPL = exp (cid:0) −z (cid:80)
zt =1 1 | v t | log p ( v t ) (cid:1) , where z and | v t | are the total number of documents and words in a document v t .
For DocNADE, the log-probability log p ( v t ) (cid:1) is computed using L DN ( v ) ; however, we ignore the mixture coefﬁcient, i.e., λ =
(equation 2) to compute the exact log-likelihood in ctx-DocNADE versions.
The optimal λ is determined based on the validation set.
Table 5 quantitatively shows the
PPL scores, where the complementary learning with λ = 0 .
01 (optimal) in ctx-DocNADE achieves lower perplexity than the baseline DocNADE for both short and long texts, e.g., ( 822 vs 846 ) and ( 1358 vs 1375 ) on AGnewstitle and 20NS 4 datasets, respectively in the FV setting.
3. I NTERPRETABILITY : T OPIC C OHERENCE
We compute topic coherence (Chang et al., 2009; Newman et al., 2009; Gupta et al., 2018) to assess the meaningfulness of the underlying topics captured.
We choose the coherence measure proposed by R ¨ oder et al.
(2015) , which identiﬁes context features for each topic word using a sliding window over the reference corpus.
Higher scores imply more coherent topics.
We use the gensim module ( radimrehurek.com/gensim/models/coherencemodel.html , coherence type = c v ) to estimate coherence for each of the 200 topics (top 10 and 20 words).
Table 6 shows average coherence over 200 topics, where the higher scores in ctx-DocNADE compared to DocNADE ( .
772 vs . 755 ) suggest that the contextual information and language structure help in generating more coherent topics.
The introduction of embeddings in ctx-DocNADEe boosts the topic coherence, leading to a gain of 4 .
6 % (.790 vs .755) on average over 11 datasets.
Note that the proposed models also outperform the baselines methods glove-DMM and glove-LDA.
Qualitatively, Table 8 illustrates an example topic from the 20NSshort text dataset for DocNADE, ctx-DocNADE and ctx-DocNADEe, where the inclusion of embeddings results in a more coherent topic.
Additional Baslines : We further compare our proposed models to other approaches that combining topic and language models, such as TDLM (Lau et al., 2017), Topic-RNN (Dieng et al., 2016) and
TCNLM (Wang et al., 2018).
However, the related studies focus on improving language models using topic models: in contrast, the focus of our work is on improving topic models for textual representations (short-text or long-text documents) by incorporating language concepts (e.g., word ordering, syntax, semantics, etc.) and external knowledge (e.g., word embeddings) via neural language models, as discussed in section 1.
To this end, we follow the experimental setup of the most recent work, TCNLM and quantitatively compare the performance of our models (i.e., ctx-DocNADE and ctx-DocNADEe) in terms of topic coherence (NPMI) on BNC dataset.
Table 7 (left) shows NPMI scores of different models, where the results suggest that our contribution (i.e., ctx-DocNADE) of introducing language concepts into
BoW topic model (i.e., DocNADE) improves topic coherence 5 .
The better performance for high val-
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
ues of λ illustrates the relevance of the LM component for topic coherence (DocNADE corresponds to λ =0).
Similarly, the inclusion of word embeddings (i e., ctx-DocNADEe) results in more coherent topics than the baseline DocNADE.
Importantly, while ctx-DocNADEe is motivated by sparse data settings, the BNC dataset is neither a collection of short-text nor a corpus of few documents.
Consequently, ctx-DocNADEe does not show improvements in topic coherence over ctx-DocNADE.
In Table 7 (right), we further qualitatively show the top 5 words of seven topics (topic name summarized by Wang et al.
(2018)) from TCNML and our models. Observe that ctx-DocNADE captures a topic expression that is a collection of only verbs in the past participle.
Since the BNC dataset is unlabeled, we are here restricted to comparing model performance in terms of topic coherence only.
3. A PPLICABILITY : T EXT R ETRIEVAL AND C ATEGORIZATION
Text Retrieval : We perform a document retrieval task using the short-text and long-text documents with label information.
We follow the experimental setup similar to Lauly et al.
(2017), where all test documents are treated as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity measure between their textTOvec representations (section 2.2).
To compute retrieval precision for each fraction (e.g., 0 . 0001 , 0 . 005 , 0 . 01 , 0 . 02 , 0 . 05 , etc.), we average the number of retrieved training documents with the same label as the query.
For multi-label datasets, we average the precision scores over multiple labels for each query.
Since, Salakhutdinov & Hinton (2009) and Lauly et al.
(2017) have shown that RSM and DocNADE strictly outperform
LDA on this task, we solely compare DocNADE with our proposed extensions.
Table 3 and 4 show the retrieval precision scores for the short-text and long-text datasets, respectively at retrieval fraction 0 .
Observe that the introduction of both pre-trained embeddings and language/contextual information leads to improved performance on the IR task noticeably for short texts.
We also investigate topic modeling without pre-processing and ﬁltering certain words, i.e. the
FV setting and ﬁnd that the DocNADE(FV) or glove(FV) improves IR precision over the baseline
RV setting.
Therefore, we opt for the FV in the proposed extensions.
On an average over the 8 shorttext and 6 long-text datasets, ctx-DocNADEe reports a gain of 7 . 1 % ( .
630 vs . 588 )
(Table 3) 6 .
( . 601 vs . 567 )
(Table 4), respectively in precision compared to DocNADE(RV).
To further compare with TDLM, our proposed models (ctx-DocNADE and ctx-DocNADEe) outperform it by a notable margin for all the short-text datasets, i.e., a gain of 14.5% (.630 vs .550: ctx-DocNADEe vs TDLM)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
in IR-precision.
In addition, the deep variant ( d =3) with embeddings, i.e., ctx-DeepDNEe shows competitive performance on TREC6 and Subjectivity datasets.
Figures (3a, 3b, 3c, 3d, 3e and 3f) illustrate the average precision for the retrieval task on 6 datasets.
Observe that the ctx-DocNADEe outperforms DocNADE(RV) at all the fractions and demonstrates a gain of 6 . 5 % ( . 615 vs . 577 ) in precision at fraction 0.02, averaged over 14 datasets.
Additionally, our proposed models outperform TDLM and ProdLDA 6 (for 20NS) by noticeable margins.
Text Categorization : We perform text categorization to measure the quality of our textTovec representations.
We consider the same experimental setup as in the retrieval task and extract textTOvec of 200 dimension for each document, learned during the training of ctx-DocNADE variants.
To perform text categorization, we employ a logistic regression classiﬁer with L 2 regularization.
While, ctx-DocNADEe and ctx-DeepDNEe make use of glove embeddings, they are evaluated against the topic model baselines with embeddings.
For the short texts (Table 3), the glove leads DocNADE in classiﬁcation performance, suggesting a need for distributional priors in the topic model.
Therefore, the ctx-DocNADEe reports a gain of 4 .
8 % ( . 705 vs . 673 ) and 3 .
618 vs . 596 ) in F 1 , compared to DocNADE(RV) on an average over the short (Table 3) and long (Table 4) texts, respectively.
In result, a gain of 4 .
662 vs . 634 ) overall.
In terms of classiﬁcation accuracy on 20NS dataset, the scores are: DocNADE (0.734), ctxDocNADE (0.744), ctx-DocNADEe (0.751), NTM (0.72) and SCHOLAR (0.71).
While, our proposed models, i.e., ctx-DocNADE and ctx-DocNADEe outperform both NTM (results taken from
Cao et al.
(2015), Figure 2) and SCHOLAR (results taken from Card et al.
Table 2), the
DocNADE establishes itself as a strong neural topic model baseline.
3. I NSPECTION OF L EARNED R EPRESENTATIONS
To further interpret the topic models, we analyze the meaningful semantics captured via topic extraction.
Table 8 shows a topic extracted using 20NS dataset that could be interpreted as computers , which are (sub)categories in the data, conﬁrming that meaningful topics are captured.
Observe that
[Figure]
rec i s i o n a t f r a c t i o n
[Figure]
the ctx-DocNADEe extracts a more coherent topic due to embedding priors.
To qualitatively inspect the contribution of word embeddings and textTOvec representations in topic models, we analyse the text retrieved for each query using the representations learned from DocNADE and ctxDoocNADEe models.
Table 9 illustrates the retrieval of the top 3 texts for an input query, selected from TMNtitle dataset, where #match is YES if the query and retrievals have the same class label.
Observe that ctx-DocNADEe retrieves the top 3 texts, each with no unigram overlap with the query.
[Figure]
[Table] [Table]
Additionally, we show the quality of representations learned at different fractions (20%, 40%, 60%, 80%, 100%) of training set from TMNtitle data and use the same experimental setup for the IR and classiﬁcation tasks, as in section 3.3.
In Figure 4, we quantify the quality of representations learned and demonstrate improvements due to the proposed models, i.e., ctx-DocNADE and ctx-DocNADEe over DocNADE at different fractions of the training data.
Observe that the gains in both the tasks are large for smaller fractions of the datasets.
For instance, one of the proposed models, i.e., ctxDocNADEe (vs DocNADE) reports: (1) a precision (at 0.02 fraction) of 0.580 vs 0.444 at 20% and 0.595 vs 0.525 at 100% of the training set, and (2) an F1 of 0.711 vs 0.615 at 20% and 0.726 vs 0.688 at 100% of the training set.
Therefore, the ﬁndings conform to our second contribution of improving topic models with word embeddings, especially in the sparse data setting.
3. C ONCLUSION
In this work, we have shown that accounting for language concepts such as word ordering, syntactic and semantic information in neural autoregressive topic models helps to better estimate the probability of a word in a given context.
To this end, we have combined a neural autoregressive topic-
(i e.,
DocNADE) and a neural language (e g., LSTM-LM) model in a single probabilistic framework with an aim to introduce language concepts in each of the autoregressive steps of the topic model.
This facilitates learning a latent representation from the entire document whilst accounting for the local dynamics of the collocation patterns, encoded in the internal states of LSTM-LM.
We further augment this complementary learning with external knowledge by introducing word embeddings.
Our experimental results show that our proposed modeling approaches consistently outperform stateof-the-art generative topic models, quantiﬁed by generalization (perplexity), topic interpretability (coherence) and applicability (text retrieval and categorization) on 15 datasets
R
A D ATA D ESCRIPTION
We use 14 different datasets: (1) 20NSshort : We take documents from 20NewsGroups data, with document size less (in terms of number of words) than 20.
(2) TREC6 : a set of questions (3) Reuters21578title : a collection of new stories from nltk.corpus .
We take titles of the documents.
(4) Subjectivity : sentiment analysis data.
(5) Polarity : a collection of positive and negative snippets acquired from Rotten Tomatoes (6) TMNtitle :
Titles of the Tag
My News (TMN) news dataset.
(7) AGnewstitle : Titles of the AGnews dataset.
Reuters8 : a collection of news stories, processed and released by (9) Reuters21578 : a collection of new stories from nltk.corpus .
(10) 20NewsGroups : a collection of news stories from nltk.corpus .
(11) RCV1V2 (Reuters): www.ai.mit.edu/projects/jmlr/papers/ volume5/lewis04a/lyrl2004_rcv1v2_README.htm (12) 20NSsmall :
We sampledocument for training from each class of the 20NS dataset.
For validation and test, 10 document for each class.
(13) TMN :
The Tag My News (TMN) news dataset.
(14) Sixxx Requirement
OBjects ( SiROBs ): a collection of paragraphs extracted from industrial tender documents (our industrial corpus). [Table]
Instructors shall have tertiary education and experience in the operation and maintenance of the equipment or sub-system of Plant.
They shall be proﬁcient in the use of the English language both written and oral.
They shall be able to deliver instructions clearly and systematically.
The curriculum vitae of the instructors shall be submitted for acceptance by the Engineer at least 8 weeks before the commencement of any training. Label: maintenance The Contractor shall provide experienced staff for 24 hours per Day, 7 Days per week, throughout the Year, for call out to carry out On-call Maintenance for the Signalling System. [Table]
Unless otherwise speciﬁed, this standard is applicable to all cables which include single and multi-core cables and wires, Local Area Network (LAN) cables and Fibre Optic (FO) cables.
[Table]
The Contractor shall provide and permanently install the asset labels onto all equipment supplied under this Contract.
The Contractor shall liaise and co-ordinate with the Engineer for the format and the content of the labels.
The Contractor shall submit the ﬁnal format and size of the labels as well as the installation layout of the labels on the respective equipment, to the Engineer for acceptance.
[Table]
It shall be possible to switch any station Interlocking capable of reversing the service into “Auto-Turnaround Operation”.
This facility once selected shall automatically route Trains into and out of these stations, independently of the ATS system.
At stations where multiple platforms can be used to reverse the service it shall be possible to select one or both platforms for the service reversal.
[Table]
The SiROBs is our industrial corpus, extracted from industrial tender documents.
The documents contain requirement speciﬁcations for an industrial project for example, railway metro construction .
There are 22 types of requirements i.e. class labels (multi-class), where a requirement is a paragraph or collection of paragraphs within a document.
We name the requirement as Requirement Objects (ROBs).
Some of the requirement types are project management , testing , legal , risk analysis , ﬁnancial cost , technical requirement , etc.
We need to classify the requirements in the tender documents and assign each ROB to a relevant department(s).
Therefore, we analyze such documents to automate decision making, tender comparison, similar tender as well as ROB retrieval and assigning
ROBs to a relevant department(s) to optimize/expedite tender analysis.
See some examples of ROBs from SiROBs corpus in Table 10.
B E XPERIMENTAL S ETUP B. E XPERIMENTAL S ETUP AND H YPERPARAMETERS FOR G ENERALIZATION TASK
See Table 11 for hyperparameters used in generalization
. B. E XPERIMENTAL S ETUP AND H YPERPARAMETERS FOR IR TASK
We set the maximum number of training passes to 1000, topics to 200 and the learning rate to 0.
with tanh hidden activation.
For model selection, we used the validation set as the query set and used the average precision at 0.02 retrieved documents as the performance measure.
Note that the [Table] [Table]
have the same label the
See Table 12 for the hyperparameters in the document retrieval task.
B. E XPERIMENTAL S ETUP FOR DOC 2 VEC MODEL
We used gensim ( https://github.com/RaRe-Technologies/gensim ) to train
Doc2Vec models for 12 datasets.
Models were trained with distributed bag of words, foriterations using a window size of 5 and a vector size of 500.
B. C LASSIFICATION TASK
We used the same split in training/development/test as for training the Doc2Vec models (also same split as in IR task) and trained a regularized logistic regression classiﬁer on the inferred document vectors to predict class labels.
In the case of multilabel datasets ( R21578 , R21578title ,
RCV1V2 ), we used a one-vs-all approach.
Models were trained with a liblinear solver using L regularization and accuracy and macro-averaged F1 score were computed on the test set to quantify predictive power.
B. E XPERIMENTAL S ETUP FOR GLOVE -DMM AND GLOVE -LDA MODELS
We used LFTM ( https://github.com/datquocnguyen/LFTM ) to train glove-DMM and glove-LDA models.
Models were trained for 200 iterations with 2000 initial iterations usingtopics.
For short texts we set the hyperparameter beta to 0.1, for long texts to 0.01; the mixture parameter lambda was set to 0.6 for all datasets.
The setup for the classiﬁcation task was the same as for doc2vec; classiﬁcation was performed using relative topic proportions as input (i.e. we inferred the topic distribution of the training and test documents and used the relative distribution as input
[Table]
for the logistic regression classiﬁer).
Similarly, for the IR task, similarities were computed based on the inferred relative topic distribution.
B. E XPERIMENTAL S ETUP FOR P ROD LDA
We run
ProdLDA ( https://github.com/akashgit/autoencoding_vi_for_ topic_models ) on the short-text datasets in the FV setting to generate document vectors for
IR-task.
We use 200 topics for a fair comparison with other baselines used for the IR tasks.
We infer topic distribution of the training and test documents and used the relative distribution as input for the IR task, similar to section 3.3.
To fairly compare PPL scores of ProdLDA and DocNADE in the RV setting, we take the preprocessed 20NS dataset released by ProdLDA and run DocNADE for 200 topics.
To further compare them in the FV setting, we run ProdLDA ( https://github.com/akashgit/ autoencoding_vi_for_topic_models ) on the processed 20NS dataset for 200 topics used in this paper.
C A BLATION OVER THE MIXTURE WEIGHT λ C. λ FOR G ENERALIZATION TASK
See Table 13.
C. λ FOR IR TASK
See Table 14.
D A DDITIONAL B ASELINES D. D OC NADE VS SCHOLAR
PPL scores over 20 topics : DocNADE (752) and SCHOLAR (921), i.e., DocNADE outperforms
SCHOLAR in terms of generalization.
Topic coherence (NPMI) using 20 topics : DocNADE (.18) and SCHOLAR (.35), i.e., SCHOLAR (Card et al., 2017) generates more coherence topics than DocNADE, though worse in PPL and text classiﬁcation (see section 3.3) than DocNADE, ctx-DocNADE and ctx-DocNADEe.
IR tasks : Since, SCHOLAR (Card et al., 2017) without meta-data equates to ProdLDA
and we have shown in section 3.3 that ProdLDA is worse on IR tasks than our proposed models, therefore one can infer the performance of SCHOLAR on IR task.
The experimental results above suggest that the DocNADE is better than SCHOLAR in generating good representations for downstream tasks such as information retrieval or classiﬁcation, however falls behind SCHOLAR in interpretability.
The investigation opens up an interesting direction for future research.