N EURAL N ONNEGATIVE CP D ECOMPOSITION FOR H IERARCHICAL T ENSOR A NALYSIS
Anonymous authors
Paper under double-blind review
A BSTRACT
There is a signiﬁcant demand for topic modeling on large-scale data with complex multi-modal structure in applications such as multi-layer network analysis, temporal document classiﬁcation, and video data analysis; frequently this multi-modal data has latent hierarchical structure.
We propose a new hierarchical nonnegative
CANDECOMP/PARAFAC (CP) decomposition (hierarchical NCPD) model and a training method, Neural NCPD, for performing hierarchical topic modeling on multi-modal tensor data.
Neural NCPD utilizes a neural network architecture and backpropagation to mitigate error propagation through hierarchical NCPD.
I NTRODUCTION
The recent explosion in the collection and availability of data has led to an unprecedented demand for scalable data analysis techniques.
Furthermore, data that has a multi-modal tensor format has become ubiquitous across numerous ﬁelds (Cichocki et al., 2009).
The need to reduce redundant dimensions (across modes) and to identify meaningful latent trends within data has rightly become an integral focus of research within signal processing and computer science.
An important application of these dimension-reduction techniques is topic modeling , the task of identifying latent topics and themes of a dataset in an unsupervised or partially supervised approach.
A popular topic modeling approach for matrix data is the dimension-reduction technique nonnegative matrix factorization (NMF) (Lee &
Seung, 1999), which is generalized to multi-modal tensor data by the nonnegative CP decomposition (NCPD) (Carroll & Chang, 1970; Harshman et al., 1970).
These models identify r latent topics within the data; here the rank r is a user-deﬁned parameter that can be challenging to select without a priori knowledge or a heuristic selection procedure.
In topic modeling applications, one often additionally wishes to understand the hierarchical topic structure (i.e., how the topics are naturally related and combine into supertopics).
For matrices (tensors), a naive approach is to apply NMF (NCPD) ﬁrst with rank r and then again with rank j < r , and simply identify the j supertopics as linear (multilinear) combinations of the original r subtopics.
However, due to the nonconvexity of the NMF (NCPD) objective function, the supertopics identiﬁed in this way need not be linearly (multi-linearly) related to the subtopics.
For this reason, hierarchical models which enforce these relationships between subtopics and supertopics have become a popular direction of research.
A challenge of these models is that the nonconvexity of the model at each level of hierarchy can yield cascading error through the layers of models; several works have proposed techniques for mitigating this cascade of error (Flenner & Hunter, 2018; Trigeorgis et al., 2016;
Le Roux et al., 2015; Sun et al., 2017; Gao et al., 2019).
In this work, we propose a hierarchical NCPD model and Neural NCPD, an algorithm for training this model which exploits backpropagation techniques to mitigate the effects of error introduced at earlier (subtopic) layers of hierarchy propagating downstream to later (supertopic) layers.
This approach allows us to (1) explore the topics learned at different ranks simultaneously, and (2) illustrate the hierarchical relationship of topics learned at different tensor decomposition ranks.
Notation.
We follow the notational conventions of Goodfellow et al.
(2016); e.g., tensor X , matrix X , vector x , and (integer or real) scalar x .
In all models, we use variable r (with superscripts denoting layer of hierarchical models) to denote model rank and use j when indexing through rank-one components.
In all tensor decomposition models, we use k to denote the order (number of modes) of the tensor and use i when indexing through modes of the tensor.
In all hierarchical models, we use L to denote the number of layers in the model and use ` to index layers.
We let ⌦ denote the vector outer product and adopt the CP decomposition notation
[Equation]
where x ( i ) j is the j th column of the i th factor matrix
X i (Kolda & Bader, 2009). Contributions.
Our main contributions are two-fold.
First, we propose a novel hierarchical nonnegative tensor decomposition model that we denote hierarchical NCPD (HNCPD). Our model treats all tensor modes alike and the output is not affected by the order of the modes in the tensor representation; this is a property not shared by other hierarchical tensor decomposition models such as that of Cichocki et al. (2007a). Second, we propose an effective neural network-inspired training method that we call Neural NCPD. This method builds upon the Neural NMF method proposed in Gao et al.
but is not a direct extension; Neural NCPD consists of a branch of Neural NMF for each tensor mode, but the backpropagation scheme must be adapted for factorization information ﬂow between branches.
Organization. In the remainder of Section 1, we present related work on tensor decompositions and training methods.
In Section 2, we present our main contributions, hierarchical NCPD and the Neural
NCPD method.
In Section 3, we test Neural NCPD on real and synthetic data, and offer some brief conclusions in Section 4.
We include justiﬁcation of several computational details of our method and further experimental results in Appendix A.
1. R ELATED W ORK
In this section, we introduce NMF, hierarchical NMF, the Neural NMF method, and NCPD, and then summarize some relevant work.
Nonnegative Matrix Factorization (NMF).
Given a nonnegative matrix X 2 R n 1 ⇥ n 2 , and a desired dimension r 2 N , NMF seeks to decompose X into a product of two low-dimensional nonnegative matrices; dictionary matrix A 2 R n 1 ⇥ r and representation matrix S 2 R r ⇥ n 2 so that
[Equation]
where a j is a column (topic) of A and s j is a row of S .
Typically, r is chosen such that r < min { n 1 , n 2 } to reduce the dimension of the original data matrix or reveal latent themes in the data.
Each column of S provides the approximation of the respective column in X in the lower-dimensional space spanned by the columns of A .
The nonnegativity of the NMF factor matrices yields clear interpretability; thus, NMF has found application in document clustering (Xu et al., 2003; Gaussier & Goutte, 2005; Shahnaz et al., 2006), and image processing and computer vision (Lee & Seung, 1999; Guillamet & Vitria, 2002; Hoyer, 2002), amongst others.
Popular training methods include multiplicative updates (Lee & Seung, 1999; 2001; Lee et al., 2009), projected gradient descent (Lin, 2007), and alternating least-squares (Kim et al., 2008; Kim & Park, 2008).
Hierarchical NMF (HNMF).
HNMF seeks to illuminate hierarchical structure by recursively factorizing the NMF S matrices; see e.g., (Cichocki et al., 2009).
We ﬁrst apply NMF with rank r (0) and then apply NMF with rank r (1) to the S matrix, collecting the r (0) subtopics into r (1) supertopics.
HNMF with L layers approximately factors the data matrix as
[Equation]
Here the A ( i ) matrix represents how the subtopics at layer i collect into the supertopics at layer i + 1 .
Note that as L increases, the error k X  A (0) A (1) · · · A ( L  1) S
( L  1)
k F necessarily increases as error propagates with each step.
As a result, signiﬁcant error is introduced when L is large.
Choosing r (0) , r (1) , · · · , r ( L  1) in practice proves difﬁcult as the number of possibilities grow combinatorially.
Neural NMF (NNMF).
In the previous work of Gao et al.
(2019), the authors developed an iterative algorithm for training HNMF that uses backpropagation techniques to mitigate cascading error through the layers.
To form this hierarchical factorization, the Neural NMF algorithm uses a neural net architecture.
Each layer ` of the network has weight matrix A ( ` ) .
In the forward propagation step, the network accepts a matrix S ( `  1) , calculates the nonnegative least-squares solution [Equation]
[Equation]
and sends the matrix S ( ` ) to the next layer.
In the backpropagation step, the algorithm calculates gradients and updates the weights of the network, which in this case are the A matrices.
Nonnegative CP Decomposition (NCPD).
The NCPD generalizes NMF to higher-order tensors; speciﬁcally, given an order- k tensor X 2 R n 1 ⇥ n 2 ⇥ ··· ⇥ n k and a ﬁxed integer r , the approximate NCPD of X seeks X [Equation] [Equation]
so that
[Equation]
The X i matrices will be referred to as the NCPD factor matrices.
A nonnegative approximation with ﬁxed r is obtained by approximately minimizing the reconstruction error between X and the
NCPD reconstruction.
This decomposition has found numerous applications in the area of dynamic topic modeling where one seeks to discover topic emergence and evolution (Cichocki et al., 2007b;
Traor´e et al., 2018; Saha & Sindhwani, 2012).
Methods for training NMF models can often be generalized to NCPD; for example, multiplicative updates (Welling & Weber, 2001) and alternating least-squares (Kim et al., 2014).
Other Related Work.
Other works have sought to mitigate error propagation in HNMF models with techniques inspired by neural networks (Trigeorgis et al., 2016; Le Roux et al., 2015; Sun et al., 2017; Flenner & Hunter, 2018).
Additionally, previous works have developed hierarchical tensor decomposition models and methods (Vasilescu & Kim, 2019; Song et al., 2013; Grasedyck, 2010).
The model most similar to ours is that of Cichocki et al. (2007a), which we refer to as hierarchical nonnegative tensor factorization (HNTF).
This model consists of a sequence of NCPDs, where a factor matrix for one mode is held constant, the remaining factor matrices produce the tensor which is decomposed at the second layer, and this decomposition is combined with the ﬁxed matrix from the previous layer.
We note that HNTF is dependent upon the ordering of the modes, and speciﬁcally which data mode appears ﬁrst in the representation of the tensor.
We refer to ‘HNTF- i ‘ as HNTF applied to the representation of the tensor where the modes are reordered with mode i ﬁrst.
O UR C ONTRIBUTIONS
In this section, we present our two main contributions.
We ﬁrst describe the proposed hierarchical
NCPD (HNCPD) model, and then propose a training method, Neural NCPD, for the model.
2. H IERARCHICAL NCPD (HNCPD)
Given an order- k tensor X 2 R n 1 ⇥ ... ⇥ n k , HNCPD consists of an initial rank- r NCPD layer with factor matrices X 1 , X 2 , . . .
, X k , each with r columns, and an HNMF with ranks r (0) , r (1) , · · · , r ( L  2) for each of these factors matrices; that is, for each X i , we factorize X i as
[Equation]
(6) where A ( ` ) i has r ( ` ) columns; see Figure 1 for a visualization.
The ﬁrst layer of HNCPD is the NCPD layer of rank r , but the later layers require collection of the columns of the hierarchical NMF factor matrices to form the NCPD models of ranks r (0) , r (1) , · · · , r ( L  2) .
We must do so carefully due to the topic dependencies introduced by the HNMF layers.
For simplicity, we ﬁrst consider the two layer case.
We note that [
[Equation]
where ↵ j 1 ,j 2 ,...
j k =
P rp =1 ( S (0)1 ) j 1 ,p ( S (0)2 ) j 2 ,p . . .
( S (0) k ) j k ,p ; we justify this statement in Appendix A. We refer to decomposition summands in (7) where j 1 = j 2 = · · · = j k as vector outer products of same-index factor matrix topics, and all other summands as vector outer products of different-index factor matrix topics.
Note that, by further factorizing the initial NCPD factors, we can inadvertantly increase NCPD rank of the later-layer decompositions since these involve vector outer products of different-index factor matrix topics.
[Figure]
We develop an approximation scheme that avoids these different-index column outer products, elucidates the hierarchical NCPD structure, and empirically is seen to provide good ﬁt even at later layers.
This approximation scheme computes matrices [Equation] [Figure] [Equation]
whose columns provide the desired r (0)
NCPD topics along each mode while avoiding different-index column outer products in the decomposition. We approximate the summation (7) by replacing all summands that include column p 2 of A (0) k with a single rank-one vector outer product, ( [Equation] e
[Equation]
A
[Equation] [Equation] p 2 . To minimize error introduced by this approximation, we transform factor matrices A (0) i {+for i 
+}
= k to
[Equation]
by collecting into ( e
A (0) i ) : ,p 2 the approximate contribution of all columns of A (0) i in vector outer products with ( A (0) k ) : ,p 2 in (7).
That is, for 1  p 1 , p 2  r (0) and 1  i < k , let W
i
R r (0) ⇥ r (0) be a matrix with ( W i ) p 1 ,p 2 =
X j i = p 1 ,j k = p 2 , 1  j 1 ,j 2 ,... j k  r (0) ↵ j 1 ,j 2 ,...,j k and e
A (0) i = A (0) i W i , (8) and let the approximate NCPD at the second layer be [[ e
A (0)1 , · · · , e A (0) k  1 , A (0) k ]] ; that is, we avoid the different-index column vector outer products and produce a NCPD approximation of rank r (0) .
While this approximation scheme has lost some information computed in the HNMF models, empirically we ﬁnd that it constructs NCPD approximations with relatively good ﬁt.
Furthermore, we can identify the topic hierarchy from the S (0) k matrix.
We can generalize this process to later layers ` by noting that we can group the A i matrices together, so
X i ⇡
[Equation]
S (
[Equation]
) i .
Thus, we can treat this approximation as above, replacing A (0) i with the product A (0)
i A (1) i . . .
A ( ` ) i .
Like in HNMF, errors in earlier layers can propagate through to later layers and produce highly suboptimal approximations.
Challenges encountered during computation of HNMF are exacerbated in an HNCPD model.
For this reason, we exploit approaches developed for HNMF in Gao et al.
(2019) in our training method Neural NCPD.
Furthermore, the computation of HNMF factor matrices for X i are independent from X j if the factorizations are applied sequentially; Neural NCPD allows factor matrices for all other modes to inﬂuence the factorization of a given mode.
2. N EURAL NCPD
Our iterative method consists of two subroutines, a forward-propagation and a backpropagation.
In
Algorithms 1 and 2, we display the pseudocode for our proposed method. Forward Propagation.
The forward-propagation treats A ( ` ) i matrices as neural network weights and uses A ( ` ) i and previous layer output to compute [Equation] where q is as deﬁned in equation 4 and S (  1) i
=
X i , producing the matrices S (0) i , . . .
, S ( L  2) i for 1  i  k .
The function q ( A ( ` ) , S ( `  1) ) , as a nonnegative least-squares problem, can be calculated via any convex optimization solver; we utilize an implementation of the Hanson-Lawson algorithm (Lawson & Hanson, 1995).
Finally, we pass the A ( ` ) i and S ( ` ) i matrices and X into a loss function, which we differentiate and backpropagate.
procedure F ORWARD P ROP ( { X i } ki =0 , { A ( ` ) i } k, L  2 i =0 , ` =0 ) for i = 1 , · · · , k do for ` = 0 , · · · , L  2 do
S ( ` ) i q
( A ( ` ) i , S ( `  1) i ) .
see equation
Input: Tensor X 2 R n 1 ⇥ n 2 ⇥ ... ⇥ n k , cost C
X 1 , X 2 , . . .
X k
NCPD ( X ) , initialize { A ( ` ) i } k, L  2 i =0 , ` = for iterations = 1 , . .
, T do
ForwardProp ( { X i } ki =0 , { A ( ` ) i } k, L  2 i =0 , ` =0 ) .
Alg.for i = 1 , · · · , k, ` = 0 , · · · , L  2 do
A ( ` ) i ⇣ optimizer
A ( ` ) i , @
C @
A ( ` ) i ⌘ + .
any ﬁrst-order method
Backpropagation.
Our goal is to differentiate our cost function C with respect to the weights in each layer, the A ( ` ) i matrices and backpropagate.
This algorithm accepts any ﬁrstorder optimization method, denoted optimizer (e.g., SGD (Robbins &
Monro, 1951), Adam (Kingma & Ba, 2014)), but projects the updated weight matrix into the positive orthant to maintain nonnegativity.
For the NCPD task, the most natural loss function is the reconstruction loss ,
[Equation]
In order to encourage optimal ﬁt at each layer, we also introduce a loss function that we refer to as energy loss .
First we denote the approximation of X at layer ` of our network as
[Equation]
Then, we calculate energy loss as
[Equation]
The derivatives of q ( A , X ) with respect to A and X are derived and exploited to differentiate a generic cost function for the hierarchical NMF model in Gao et al.
(2019); here we summarize these derivatives and illustrate how to combine them with simple multilinear algebra for HNCPD.
Gao et al.
(2019) show that, if
[Equation]
is the derivative of C with respect to A ( ` 1 ) i holding the S matrices constant, then
[Equation]
13) where U ( ` 1 , ` 2 ) ,j i relates C to A ( ` 1 ) i through S ( ` 2 ) i and S ( ` 1 ) i , is deﬁned column-wise ( j ), and depends upon ✓
[Equation]
◆ * , the derivative of C with respect to S ( ` 2 ) i holding S ( ` 2 +1) i , . .
, S ( L  2)
i constant.
The deﬁnition of U ( ` 1 , ` 2 ) ,j i is given in Gao et al.
(2019) and utilizes, via the chain-rule, the partial derivative of q ( A ` i , S `  1 i ) for all ` 2 [ ` 1 , ` 2 ] .
Example.
The derivative of the previously deﬁned, or other differentiable cost functions can be calculated using these results of Gao et al.
(2019) and some simple multi-linear algebra.
As an example, we directly compute the backpropagation step for the reconstruction loss function C given in equation 10.
Let X ( i ) be the mode- i matricized version of X , and deﬁne [Equation]
[Equation]
where  denotes the Khatri-Rao product (see e.g., (Kolda & Bader, 2009)).
Then we have that @
[Equation] [Equation]
and @
[Equation]
These derivatives are justiﬁed in Appendix A. With equation 13, these derivatives are sufﬁcient to calculate the partial derivative of C with respect to any A matrix.
E XPERIMENTAL R ESULTS
We test Neural NCPD on three datasets: one synthetic, one video, and one collected from Twitter.
The synthetic dataset is constructed as a simple block tensor with hierarchical structure.
The Twitter dataset consists of tweets from political candidates during the 2016 United States presidential election (Littman et al., 2016). We pull the video, a time-lapse of a forest over the span of one year, from (Solheim).
We also compare Neural NCPD to Standard NCPD , in which we perform an independent NCPD decomposition at each rank, and to Standard HNCPD , in which we perform
NCPD ﬁrst on the full dataset, and apply HNMF to the ﬁxed factor matrices; here we sequentially apply NMF to the factor matrices using multiplicative updates and do not update previous layer factorizations as in Neural NCPD. In all experiments, we use Tensorly (Kossaiﬁ et al., 2018) for Standard NCPD calculations and to initialize the NCPD layer of our hierarchical NCPD, and in
Neural NCPD we do not backpropagate to this layer as the initialization has usually found a stationary point. We use Energy Loss (Eq. 11) for all experiments to encourage ﬁt at every layer. Because we do not backpropagage to the initial factor matrices, the ﬁrst term in (Eq. 11) is ﬁxed.
For the Twitter and video experiments we use the approximation scheme of Section 2.1 to recover the relationship between the columns of the A ( ` ) i matrices.
3. E XPERIMENT ON S YNTHETIC D ATA
We test the Neural NCPD algorithm ﬁrst using a synthetic dataset.
This dataset is a rank seven tensor of size 40 ⇥ 40 ⇥ 40 with positive noise added to each entry; we generate noise as n = | g | where g ⇠ N (0 ,  2 ) .
To generate this dataset, we begin with the all-zeros tensor and create three large nonoverlapping blocks with value 1, and then overlay each block with either two or three additional blocks with value 3.
We display this tensor with two levels of noise at the left of Figure 2; here we plot projections of all tensors (and all approximations) along the third mode; that is, we construct a matrix with entries equal to the largest entries of the mode-three ﬁbers (see e.g., (Kolda & Bader, 2009) for relevant deﬁnitions).
The projections on the remaining two modes are included in the
Appendix A, and are all similar to the third mode. Method  2 = 0 . 2 = 0 .r = r (0) =
r (1) =
r =r (0) = r (1) = Neural NCPD 0 .0 .0 .0 .0 .0 . Standard HNCPD 0 .0 .
[Table]
We run Neural NCPD, Standard HNCPD, and Chichocki et al.
on this synthetic dataset at two different levels of noise with three layers of ranks 7, 5, and 3, and display the results in Figureand Table 1; we present the relative reconstruction loss C rel =
k X  [[ f
X 1 , f
X 2 , · · · , f
X k ]] k F / k X k F .
For each level of noise, we display the rank 7 approximation shared by all methods, and the rank [Figure] and rank 3 approximations produced by Neural NCPD and Standard HNCPD.
We also display the transposed Neural NCPD S (0)3 and S (1)3 matrices, which show how rank 7 topics collect into ranksand 3 topics.
From Table 1, we see that the loss for Neural NCPD is at or below that of Standard HNCPD and HNTF at each rank and level of noise.
3. T EMPORAL D OCUMENT A NALYSIS
We next apply Neural NCPD to a dataset of tweets from four Republican [R] and four Democratic [D] 2016 presidential primary candidates, (1) Hillary Clinton
[D], (2) Tim Kaine [D], (3) Martin
O’Malley [D], (4) Bernie Sanders [D], (5) Ted Cruz
[R], (6) John Kasich [R], (7) Marco Rubio [R], and (8) Donald Trump [R]; this is constructed from a subset of the dataset of Littman et al.
We use a bag-of-words (12,721 words in corpus) representation of all tweets made by a candidate within bins of 30 days (from February to December 2016), and cap each of these groups at 100 tweets to avoid oversampling from any candidate; resulting in a tensor of size 8 ⇥ 10 ⇥ 12721 .
(cid:53)(cid:68)(cid:81)(cid:78)(cid:3)(cid:27)(cid:3)(cid:55)(cid:82)(cid:83)(cid:76)(cid:70)(cid:86)
[Table]
At rank, display
In Table 2, we display the relative reconstruction loss on the Twitter political dataset for all models.
We see that Neural NCPD signiﬁcantly outperforms Standard HNCPD, slightly outperforms Standard
NCPD while offering a hierarchical topic structure, and outperforms all HNTF-
i , for which loss varies signiﬁcantly based on the arrangement of the tensor.
In Figure 3, we show the topic keywords and factor matrices of a rank 8, 4, and 2 hierarchical NCPD approximation computed by Neural
NCPD.
Note that in the rank 8 candidates mode factor and keywords we see that nearly every topic is identiﬁed with a single candidate.
Topic two of the rank 8 approximation aligns with
[Table]
[Figure]
political issues (the Zika virus and the Venezuelan government) rather than a single candidate, and is temporally most present in May to July 2016 (during the Zika outbreak and the Venezualan state of emergency).
Topics one and eight, corresponding to candidates Clinton and Trump, are most
[Table]
present in the months immediately leading up to the election.
At rank 4, we see that topics one and four are inherited from the rank 8 approximation, topic two combines the rank 8 topics of candidates Trump and Clinton (ﬁnal candidates), and topic 3 combined the topics of candidates Cruz and Kasich (Republicans).
Meanwhile, the rank 2 NCPD topics are nearly identical to rank 4 NCPD topics two and three.
We display HNTF for each ordering of the tensor modes in Appendix A.
In Figure 4, we display the S (0)3 (top) and S (1)3 (bottom) matrices produced by Neural NCPD on the
Twitter dataset, which illustrate how topics collect at each rank.
We see topics 5 and 6 from the rank 8 factorization combine to form topic 3 at rank 4 and topic 2 at rank 2.
This is expected because both topics include keywords from Cruz and Kasich, who had high presence in topics 5 and 6 respectively in the rank 8 factorization.
In Figure 5, we display the results of performing separate NCPD decomposition of ranks 4 and 2 on the Twitter dataset.
We see that the results are similar to those of Neural NCPD, but these independent decompositions lack the clear hierarchical structure provided by Neural NCPD.
Note that while the topics corresponding to candidates Kasich and Clinton combine in the rank 4 NCPD, these candidates are present in different topics in the rank 2 NCPD; Neural NCPD prevents this breach of hierarchy.
We next apply Neural NCPD to video data constructed from a year-long time-lapse video of a forest; see Figure 6 for a selection of frames and Figure 11 in Appendix A for more details.
We extract 37 frames and ﬂatten each frame (RGB image) into a single matrix, to form a tensor X of size 37 ⇥ 3 ⇥ 57600 ; here the ﬁrst mode represents frames (temporal mode), the second colors (chromatic mode), and the third pixels (spatial mode).
In Figure 7, we show the three-layer Neural NCPD decompositions of the video tensor with ranks 8, 6, and 3.
For each rank, we plot the topics in the spatial (left), temporal (top right), and chromatic (bottom right) modes.
We note that many of the identiﬁed topics represent visual seasonal changes.
Topic six of the rank 8 decomposition represents the green and leafy late-summer to early fall.
Topic one of the rank 6 decomposition represents the winter sky and leaﬂess trees.
Topic three of the rankdecomposition represents the summer and fall sky and tree leaves.
[Figure]
display
We additionally apply NMF to the slices of the tensor along a single mode.
Slicing along the temporal or spatial modes would make interpretation of the resulting topics challenging, so we choose to slice along the chromatic mode, producing three matrices.
In Figure 8, we visualize a rank 3 NMF [Figure] s on each of the three chromatic slices of the video tensor.
The chromatic factorizations are nearly identical, illustrating little salient dynamic information.
While similar to the rank 3 Neural NCPD layer, the chromatic NMFs obscure much of the chromatic interaction evidenced by Neural NCPD.
In particular, Neural NCPD illustrates the spatial and temporal dynamics of multi-colored features and their cooccurrence hierarchy, while NMF provides only single-colored features and requires far more work to glean multi-colored feature co-occurrence information.
C ONCLUSIONS
In this paper, we introduced the hierarchical NCPD model and presented a novel algorithm, Neural NCPD, to train this decomposition.
We empirically demonstrate the promise of this method on both real and synthetic datasets; in particular, this model reveals the hierarchy of topics learned at different NCPD ranks, which is not available to standard NCPD or NMF-based approaches.