Alleviating “Posterior Collapse” in Deep Topic Models via Policy Gradient
Abstract
Deep topic models have been proven as a promising way to extract hierarchical latent representations from documents represented as high-dimensional bag-of-words vectors.
However, the representation capability of existing deep topic models is still limited by the phenomenon of “ posterior collapse ”, which has been widely criticized in deep generative models, resulting in the higher-level latent representations exhibiting similar or meaningless patterns.
To this end, in this paper, we first develop a novel deep-coupling generative process for existing deep topic models, which incorporates skip connections into the generation of documents, enforcing strong links between the document and its multi-layer latent representations.
After that, utilizing data augmentation techniques, we reformulate the deep-coupling generative process as a Markov decision process and develop a corresponding
Policy Gradient (PG)
based training algorithm, which can further alleviate the information reduction at higher layers.
Extensive experiments demonstrate that our developed methods can effectively alleviate “ posterior collapse ” in deep topic models, contributing to providing higher-quality latent document representations.
Introduction
Topic modeling has become a successful technique for text analysis and been widely applied to various problems in machine learning (ML)
[1, 2, 3] and natural language processing (NLP)
[4, 5] over the past two decades.
Representing documents as bag-of-words (BoW) vectors, vanilla probabilistic topic models (PTMs), with latent Dirichlet allocation (LDA) [6] being the best known representative, typically formulate each document as a mixture over latent topics, where each topic is characterized by a distribution over the terms of the vocabulary and describes an interpretable semantic concept.
Although being widely used, the modeling capability of these shallow topic models is still restricted by their single-layer structure and has difficulty in exploring hierarchical thematic structures.
To this end, a series of deep topic models [7, 8, 9] have been developed to extract multi-layer document representations from a text corpus, providing a more intuitive way for users to understand text data.
Recently, benefiting from the development of deep neural networks (DNNs), there has been an emerging research interest to develop neural topic models (NTMs) to boost the performance, efficiency, and usability of topic modeling with DNNs.
Specifically, following the framework of variational autoencoder (VAE) [10], most NTMs [11, 12, 13] construct a variational inference network (encoder) to project each document into its stochastic latent representation, and then reconstruct the corresponding BoW observation with a stochastic/deterministic decoder.
By modeling the inference/generative process with DNNs, these NTMs are more flexible and scalable than traditional Bayesian PTMs, contributing to performing large-scale downstream tasks, especially in NLP tasks [14, 15].
[Table]
“ Posterior collapse ” has been widely criticized in the field of generative model [16, 17], and the occurrence of this phenomenon will cause the approximated posterior q ϕ ( z | x ) collapses to it noninformation prior distribution p θ ( z ) , leading their KL divergence to be close to zero [10, 16, 17, 18].
For deep topic models, despite achieving attractive performance, existing PTMs or NTMs still suffer from different degrees of “ posterior collapse ”, which causes their exhibiting similar or meaningless patterns at higher layers [19, 20, 21].
Although there have been several deep NTMs [20, 21] trying to alleviate this issue by constructing more flexible inference networks, the collapse phenomenon in deep NTMs may not be solved in essence, because the true posterior provided by the generative model and the objective function for optimization remain almost unchanged [18].
To extract higher-quality hierarchical latent document representations, in this paper, we develop a deep-coupling generative process equipped with a Policy Gradients (PG) based training algorithm for existing deep topic models.
The main contributions of this work are as follows
: • We develop a deep-coupling generative process for deep topic models, which incorporates skip connections into the generation of documents to alleviate “ posterior collapse ”.
• We take a specific NTM as an example to explain how to construct a deep topic model with the deep coupling generation process, and develop a deep-coupling hierarchical Embedding Topic Model ( dc -ETM), which can be extended to other deep topic models.
• Utilizing the property of sequence-like generation process, we design a PG-based training algorithm for dc -ETM, which can further alleviate the information reduction at higher layers.
• Compared to existing deep topic models, extensive experimental results show that dc -ETMs can lead to less “ posterior collapse ” and provide higher-quality latent representations.
Related Work
Probabilistic Topic Model: Deep PTMs [7, 8, 9, 22] are developed to infer multi-layer document representations, whose adjacent layers are connected with specific factorization.
For instance, gamma belief network (GBN) [8] is constructed via factorizing the shape parameters of the gamma distributed latent representations; DPFA [7] extends PFA [23] into a multi-layer version but is restricted to model binary topic usage patterns; DirBN [9] is developed via factorizing the Dirichlet distributed topic matrix.
Although providing readily interpretable multi-layer latent document representations, the representation capability of these deep PTMs is limited by adopting CRT distribution to upward propagate data information to higher layers with their backbones [19].
Neural Topic Model: Most existing NTMs [12, 20, 21, 24, 25, 26] can be viewed as extensions of
PTMs under the VAE framework and focus on modeling the generative/inference process with DNNs.
For instance, one popular research direction of NTMs is to develop more flexible inference network with reparametrization tricks [12, 20] and the other could be incorporating word embeddings into the generative model [21, 24].
However, as far as we know, few efforts have been made to alleviate the phenomenon of “ posterior collapse ” in NTMs by modifying its generative process, which is a great challenge under the framework of topic modeling and also the main contribution of this work.
Besides, distinct from the way of combining reinforcement learning (RL) with topic models in previous works [27, 28, 29], our work is the first to formulate the topic modeling generative process as a sequential decision making one to incorporate RL-based training algorithms, which focuses on providing higher-quality latent document representations by alleviating “ posterior collapse ”.
Deep-Coupling Generative Process for Deep Topic Models
To give an intuitive insight on “ posterior collapse ” in deep topic models, we provide a detailed introduction for “ posterior collapse ” in Appendix J and then visualize the higher-level topics learned by a recent popular NTM named SawETM [21] in Fig. 3, which exhibit similar semantic patterns and limit its representation capability.
Then, we take SawETM as an example, but not limited to this, to illustrate how to construct a deep topic model with the deep coupling generation process, leading to a novel dc -ETM in Fig.
1(c).
Compared to the usual structures of deep PTMs and NTMs shown in Fig. 1(a) and 1(b), besides the design of inference network, the main difference of dc -ETM is incorporating skip connections into the generation of documents, enforcing strong links between the document and its multi-layer latent representations to alleviate “ posterior collapse ”.
(1) n  (2) n  (3) n  × (0) (0) T K D    (3) (3) D K    (3) ˆ  × (0) (0) T K D    (2) (2) D K    (2) ˆ  × (0) (0) T K D    (1) (1) D K    (1) ˆ  × (1) (1) T K D    (2) (2) D K    (2)  × (3) (3) D K    (2) (2) T K D    (1) n x (2) n x (3) n x n x (1) n h (2) n h (3) n h n x
[Table]
n h (3)  n x (1) n h (2) n h (3) n h (3) n  (2) n  (1) n  (3)  (2)  (3) n  (2) n  (1) n  n x (3)  (2)  (1)  (3) n  (2) n  (1) n  n x (3)  (2)  (1)  (3) n  (2) n  (1) n  (3)  (2)  n x (1) n h (2) n h (3) n h (3) n  (2) n  (1) n  (3)  (2)
 n x (3) n x (2) n x (1) n x
[Table]
Inference
Generative
Inference
Generative (1)  (2)  (3)  (a) Deep PTM (1) n  (2) n  (3) n  × (0) (0) T K D    (3) (3) D K    (3) ˆ  × (0) (0) T K D    (2) (2) D K    (2) ˆ  × (0) (0) T K D    (1) (1) D K    (1) ˆ  × (1) (1) T K D    (2) (2) D K    (2)  × (3) (3) D K    (2) (2) T K D    (1) n x (2) n x (3) n x n x (1) n h (2) n h (3) n h
[Table]
x
[Table]
n h (3)  n x (1) n h (2) n h (3) n h (3) n  (2) n  (1) n  (3)  (2)  (3) n  (2) n  (1) n  n x (3)  (2)  (1)  (3) n  (2) n  (1) n  n x (3)  (2)  (1)  (3) n  (2) n  (1) n  (3)  (2)  n x (1) n h (2) n h (3) n h (3) n  (2) n  (1) n  (3)  (2)
 n x (3) n x (2) n x (1) n x
Generative
Inference
Generative
Inference
Generative (1)  (2)  (3)  (b)
NTM (1) n  (2) n  (3) n  × (0) (0) T K D    (3) (3) D K    (3) ˆ  × (0) (0) T K D    (2) (2) D K    (2)
ˆ  × (0) (0) T K D    (1) (1) D K    (1) ˆ  × (1) (1) T K D    (2) (2) D K    (2)  × (3) (3) D K    (2) (2) T K D    (1)
[Figure]
[Figure]
developed in this paper, where the symbol definitions are consistent with those in Sec. 3.1.
3.1 Deep-Coupling Hierarchical Embedding Topic Model
As a usual VAE-like model, the developed dc -ETM consists of a generative model (decoder) and an inference network (encoder).
Below, we focus on presenting the generative model of dc -ETM, which can be flexibly applied for other deep topic models to alleviate “ posterior collapse ”, and leave the details of the inference network to Appendix A.
Generative Model:
Given a text corpus consisting of N documents X = { x n } Nn =1 , each document can be represented as a high-dimensional sparse BoW vector x n ∈ Z K (0) , where
Z = { 0 , 1 , ... } and
K (0) denotes the vocabulary size.
Then, from top to bottom, the generative model of the dc -ETM with L hidden layers can be formulated as
[Equation]
where, Φ ( l ) ∈ R K ( l − 1) × K ( l ) + denotes the topic matrix (factor loading) and each column ϕ ( l ) k ∈
R K ( l − 1) + indicates a specific topic (factor) at layer l ; θ ( l ) n ∈ R K ( l ) + denotes the gamma distributed latent representation (topic proportions) at layer l , K ( l ) denotes the number of hidden units (topics) at layer l .
Under the Poisson likelihood, the observed multivariate count vector x n is first factorized into L equal-size latent matrix { α ( l ) ˆ Φ ( l ) θ ( l ) n } Ll =1 , where, ˆ Φ ( l ) ∈ R K (0)
× K ( l ) + can be regarded as the projection of topic matrix Φ ( l ) to the observation space and the detailed definition will be discussed in the next paragraph; α ( l ) denotes the importance weight of ˆ Φ ( l ) θ ( l ) n for generating the observation x n , and the summation of the whole weight vector α ∈ R L + is constrained to be equal to one with a
Softmax normalization.
Then, the latent representation θ ( l ) n at layer l is further factorized into the product of the topic matrix Φ
( l +1) ∈ R K ( l ) × K ( l +1) + and topic proportions θ ( l +1) n ∈ R K ( l +1)
+ at the next layer under the shape of gamma distribution.
The top layer’s latent representation θ ( L ) n shares the same gamma shape parameters r ∈ R K ( L )
+
and we apply a gamma distributed prior on the scale parameters c ( l ) n for l ∈ { 2 , ..., L + 1 } .
With the recent popular distributed topic representation in NTMs [24, 30], each topic ϕ ( l ) k is treated as the result of applying a Softmax normalization on the inner product of its distributed representation β ( l ) k ∈ R D and topic embedding matrix β ( l − 1) ∈
R D × K ( l − 1) at the previous layer, where D denotes the dimension of the embedding space.
The projections of topic matrices to the observation space, denoted as { ˆ Φ ( l ) } Ll =1 , build the straightforward connections between the document x n and its multi-layer latent representations { θ ( l ) n } L l =1 , which alleviates the information reduction at higher layers by sharing the pressure of document modeling with all hidden layers.
To reduce the computation and storage cost of the developed dc -ETM, we develop two variants for ˆ ϕ ( l ) k ∈ R K (0) + without introducing any extra parameter.
The one variant is adopting the property of topic hierarchy elaborated in Sec.
3.2 to obtain each ˆ ϕ ( l ) k by successively multiplying topic matrices at lower layers as
[Equation]
and the other variant is treating the projection ˆ ϕ ( l ) k as the result of the inner product of its distributed representation β ( l ) k
and the word embedding matrix β (0) at the observed space, as follows
[Equation]
We emphasize that the first variant can be used to extend most existing deep topic models, while the latter is limited to NTMs equipped with topic embedding techniques.
We use the suffix − α and − β to distinguish the variants defined in Eq.
(2) and Eq.
(3), and their detailed implementations can be found in Appendix I.
Generally speaking, the deep-coupling generative process in dc -ETM not only preserves the hierarchy of traditional deep topic models, leading to multi-layer document representations to enhance the modeling capability and interpretability, but also alleviates the issue that the amount of information will decrease rapidly with the network going deeper, benefiting from building the straightforward connections between observation x n and its higher-level latent representations { θ ( l ) n } l> 1 .
Besides alleviating “ posterior collapse ”, the characteristics of deep-coupling network structure of dc -ETM also brings us a new view to design the corresponding inference network and training algorithm.
Inference Network : The details of the inference network of dc -ETM can be found in Appendix A.
3.2 Model Property
Sequence-like Generative Process: Taking advantages of the properties of the Poisson distribution, the original generative process of the observed data x n defined in Eq.
(1) can be rewritten as:
[Equation]
where x ( l ) n denotes the augmented observation at layer l , and is generated from the Poisson distribution with a rate of α ( l ) ˆ Φ ( l ) θ ( l ) n .
Then, the observed data x n can be regarded as not only the summation over these augmented vectors { x ( l ) n } Ll =1 , but also equal to the weighted summation over the latent vectors { ˆ Φ ( l ) θ ( l ) n } Ll =1 on the mean, where the weight vector α satisfies the constraint (cid:80)
Ll =
1 α ( l ) = 1 .
Rethinking the generative process of the developed dc -ETM reformulated in Eq.
(4), the set of augmented observation vectors { x ( l ) n } Ll =1 can naturally form an observation sequence [ x ( L ) n , ..., x (1) n ] by sorting these vectors according to their dependencies in the generative process (from deep to shallow).
For each hidden layer (time step) l , the generative process will first incorporate the prior information passing from deeper hidden layers { θ ( t ) n } t>l , and then generate the latent representation θ ( l ) n at the current layer (time step), which not only is supposed to generate the current observation vector x ( l ) n under the Poisson likelihood, but also introduces the information into the shape parameter of the following gamma distributed latent representation θ ( l − 1) n at the next layer (time step).
Thus, the deep-coupling generative process of dc -ETM originally defined in Eq.
(1) can be naturally reinterpreted from the perspective of sequence generation, and its reformulation defined in Eq.
(4) can be also equivalently reformulated as:
[Equation]
providing an intuitive insight for the decomposition of the likelihood function in Sec. 4.1.
Hierarchical Semantic Topics:
The developed dc -ETM can naturally interpret each semantic topic ϕ ( l ) k at layer l by visualizing its projection to the vocabulary space calculated as
{ [ (cid:81) l − 1 t =1 Φ ( t ) ] ϕ ( l ) k } K ( l ) k =1 , and each document can also be roughly seen as a random mixture over
K ( l ) topics with θ ( l ) n being the corresponding topic proportions at layer l as
[Equation]
which can be obtained with the law of total expectation.
Moreover, similar to the underlying idea of the deep learning, the topics learned by dc -ETM tend to be more specific at lower (bottom) layers and those at higher (top) layers are more general, as shown in Fig.
Secondly, in dc -ETM, both words β (0) ∈ R D
× K (0) and hierarchical topics { β ( l ) ∈ R D
× K ( l ) } Ll = are represented as embedding vectors under the same semantic space, contributing to intuitively measuring and visualizing the distance between different topics (words), which has been proven to be effective in capturing the underlying semantic structure as shown in Fig. 5(a).
Policy Gradient-based Training Algorithm 4.1 ELBO of dc -ETM
As a VAE-like NTM, the developed dc -ETM can be trained like usual VAEs by directly maximizing the evidence lower bound (ELBO), specifically as
[Equation]
where the first term is the expected log-likelihood and the other term is the Kullback–Leibler (KL) divergence from the prior p ( θ n ) to the variational posterior q ( θ n | x n ) .
Through introducing the augmented vectors { x ( l ) n } Ll =1 , the log-likelihood of x n in dc -ETM can be equivalently reformulated as
[Equation]
where the function in the second expectation term can be treated as the summation of the set of log-likelihood of { x ( l ) n } Ll =1 .
Due to the hierarchical network structure, the KL divergence term can be factorized as
[Equation]
where q ( θ ( l ) n |− ) is constructed by a Weibull-based inference network described in Appendix A and p ( θ ( l ) n | Φ ( l +1) , θ n ( l +1) ) satisfies a gamma prior in Eq.
(1), and their KL divergence has an analytic expression, benefiting from adopting the Weibull reparameterization technique [20].
Combining the aforementioned derivations, the ELBO of dc -ETM can be equivalently rewritten as
[Equation]
which can be directly optimized with gradient-based methods to update both the encoder parameters Ω and decoder parameters Ψ in dc -ETM.
We emphasize that, after deriving the augmented vectors
{ x ( l ) n } Ll =1 from x n via data augmentation technique [8], the first expectation term in L ( x n ) will be a constant and the ELBO can be directly optimized by maximizing the following loss function
[Equation]
which can be roughly treated as the ELBO of a sequence [ x ( L ) n , ..., x (1) n ] generated from a sequence of latent representations [ θ ( L ) n , ..., θ (1) n ] [31, 32], and naturally meets the sequence-like generative process of dc -ETM as discussed in Sec. 3.2.
4.2 Optimization with Policy Gradient
Similar to RNN-based model, after augmenting { x ( l ) n } Ll =1 from x n , the loss function of dc -ETM defined in Eq.
(11) is equal to the summation of L sub-loss functions, where each sub-loss function
ˆ L ( l ) ( x ( l ) n ) can be equivalently regarded as a separate loss of a subsequence generation model that is only a part of the whole sequential generative model and expected to output x ( l ) n at the final time step l .
Inspired by the great success achieved by RL methods [33, 34, 35, 36] in learning a stable long sequence (Markov decision process) with high quality, we consider the sequence-like generation procedure of a L -layer dc
-ETM
as a Markov decision process with L time steps, and develop a novel training mechanism based on Policy Gradient [35] for dc -ETM, which injects the future rewards obtained from generating the suffix subsequence into each current sub-loss function
ˆ L ( l ) ( x ( l ) n ) .
Specifically, we treat the whole dc -ETM as a stochastic policy network π ( a ( l ) n | s ( l ) n ) expected to generate a fixed-length action sequence [ a ( L ) n , ..., a (1) n ] from the observation x n , defining the state s ( l ) n as { x n , { Φ ( t ) , θ ( t ) n } t>l } and the action a ( l ) n as α ( l ) ˆ Φ ( l ) θ ( l ) n .
For each time step l , given the current state s ( l ) n , the policy network π ( a ( l ) n | s ( l ) n ) will first sample θ ( l ) n from the inference network via
[Equation]
(12) and further obtain the corresponding action as
[Equation]
13) which can be regarded as directly drawing from π ( a ( l ) n | s ( l ) n ) .
The state transition is deterministic after an action has been chosen, indicating that the next state s ( l − 1) n = { x n , { Φ ( t ) , θ ( t ) n } t>l − 1 } if the current state s ( l ) n = { x n , { Φ ( t ) , θ ( t ) n } t>l } and the action a ( l ) n = α ( l )
[Equation]
( l )
[Equation]
( l )
[Equation]
Then we take the separate loss ˆ L ( l ) ( x ( l ) n ) defined in Eq.
(11) as the immediate reward at the the time step l , formulated as
[Equation]
and the action-value function can be formulated as
[Equation]
which indicates the expected accumulative reward starting from state s ( l ) n , taking action a ( l ) n , and then generating the suffix subsequence [ a ( l − 1) n , ..., a (1) n ] with the policy network π ( a ( l ) n | s ( l ) n ) and the discount factor 0 < γ ≤ 1 .
Following [33], the objective function of training dc -ETM with policy gradient can be estimated (on one episode) as
[Equation]
where Ω and Ψ indicate the encoder and decoder parameters in dc -ETM respectively.
We note that the expectation E [ · ] can be approximated by sampling methods based on the Weibull reparameterization, and the objective function can be directly optimized by advanced gradient descent algorithms, like
Adam [37] and RMSprop [38].
We provide the details of PG-based training algorithm in Appendix C.
Experiments
To evaluate the effectiveness of the developed dc -ETM and the corresponding policy gradients (PG) based training algorithm, we make extensive experiments on both quantitative and qualitative aspects.
Considering there are two dc -ETM variants as described in 3.1, we use the suffix − α and − β to distinguish the variants defined in Eq.
(2) and Eq.
(3) respectively, and highlight whether the dc -ETM is trained with the PG based algorithm in Sec.
The implementation is available at https://github.com/yewen99/dc-ETM.
5.1 Datasets and Baselines
Datasets: Four widely used document benchmarks, specifically R8 [39], 20Newsgroups ( 20News ) [40], Reuters Corpus Volume I ( RCV1 ) [41] and World Wide Web Knowledge Base ( WebKB )
[42] are included in the following experiments.
We summarize the statistics of benchmarks in Appendix D and follow the procedure in [21] to preprocess these documents to obtain their BoW representations.
Baselines: We compare the developed dc -ETMs with a series of topic models, which can be roughly divided into two categories: 1) shallow topic models such as LDA [6], AVITM [12] and ETM [24], where LDA is a PTM and the others are NTMs; 2) deep topic models including PGBN [8], WHAI [20] and SawETM [21], where PGBN is a deep PTM and the others are deep NTMs.
We emphasize that WHAI and SawETM are the most relevant strong baselines for comparison, both of which provide hierarchical Weibull-based latent document representations, and SawETM has achieved state-of-the-art performance on unsupervised document modeling and clustering tasks.
Experimental Settings: To make a fair comparison, we set the same network structure for all deep topic models as [256 , 128 , 64 , 32 , 16] from shallow to deep.
For PTMs, we use the default hyperparameter settings in their published papers and accelerate the Gibbs sampling with GPU.
For NTMs, we set the size of their hidden layers as 256, the embedding size as 100 for them incorporating word embeddings, like ETM, SawETM and dc -ETMs, and the mini-batch size as 200.
For optimization, we adopt the same Adam optimizer [43] with a learning rate of 1e-2.
All experiments are performed with an Nvidia RTX 3090 GPU and implemented with PyTorch [44].
5.2 Quantitative Comparisons
To investigate whether the proposed skip-connection structures in both dc -ETM variants can alleviate “ posterior collapse ”, especially at higher layers, we compare them with other popular deep topic models in the first part.
Then, to investigate whether the mitigation of “ posterior collapse ” can improve the quality of latent document representations at higher layers, we conduct more quantitative comparisons in the rest parts.
We report the error bars in Appendix E.
Document Modeling:
In Fig. 2, for each deep topic model, we plot the curve of point log-likelihood ln p ( x n |− ) as a function of iterative epochs conditioned on the t -th-layer reconstruction ˆ Φ ( t ) θ ( t ) n , which can be used to measure the degree of “ posterior collapse ” by the relevance between the data sample x n and its latent representation θ ( t ) n .
From the results, we can see that although SawETM and WHAI achieve a comparable performance with dc -ETMs on the first hidden layer in Fig. 2(a), their reconstruction quality decreases dramatically with the network going deeper in Fig. 2(b) and 2(c), potentially reflecting that little data information can be propagated to higher layers of these traditional deep topic models.
Benefiting from introducing skip connections into the generative process, dc -ETMs can significantly alleviate “ posterior collapse ” at higher hidden layers. [Figure] [Figure] [Figure]
[Table]
[Figure]
[Table]
Perplexity & Topic Diversity:
To make a more comprehensive quantitative comparison, we use the average of heldout-word perplexities (the lower is the better) and topic diversities (the higher is the better) across all hidden layers to measure the document modeling performance and topic quality of these deep topic models with { θ ( t ) n } Tt =1 and { Φ ( t ) n } Tt =1 , respectively.
The experimental settings are consistent with those in [21], and the experimental results have been exhibited in Table 3.
Benefiting from hierarchical network structures, the modeling capability of deep topic models generally outperform those shallow ones.
Thanks to enhancing the connections between the observation and multiple hidden layers with the deep-coupling generative process, the developed dc -ETMs achieve lower perplexity scores and provide higher-quality topics than traditional topic models.
Then, the PG-based training algorithm brings further performance improvement to our dc -ETMs. [Figure]
[Table]
20News R Purity NMI
Purity NMI Purity
NMI
LDA 53.
40.47 AVITM 54.
41.20 ETM 51. PGBN 55. All 53. WHAI 59.
All 57.
SawETM 57.
All 51.
dc -ETM-
α 61.
39.83 All 63.
dc -ETM- β 54.
38.63 All 67.
dc -ETM-
α (Policy) 49.
32.73 All 64.
dc -ETM-
β (Policy) 57.
38.34 All 69.
Document Clustering: To evaluate the quality of the extracted latent document representations on downstream tasks, we consider document clustering, where we use the topic models after training to extract the latent representations of the testing documents and then use k-means to predict the clustering labels.
Using the Purity and Normalized Mutual Information (NMI) as metrics (the higher the better), the results shown in Table 4 demonstrate that concatenating hierarchical latent document representations extracted by traditional deep topic models cannot improve and even hurt the clustering performance, potentially indicating that the latent representations at higher layers are meaningless.
However, distinct from traditional
deep
5_0: game team games hockey baseball play year players season fans 5_1: host nntp posting lines subject organization distribution mit world access 5_2: com article writes apr lines subject organization netcom reply mark 5_3: max israel turkish jews armenian armenians war Israeli jewish armenia 5_4: president national states health american press united cliton year april 5_5: gun people government right law rights guns state fbi weapons 5_6: god jesus bible Christian people believe church truth say know
[Figure]
[Figure]
like 5_
[Figure]
like 5_
[Figure]
lines
[Figure]
writes university like 5_
[Figure]
: lines
[Figure]
com
[Figure]
writes university like 5_4: lines subject organization com article just don university writes like 5_5: lines subject organization com article just don university writes like
[Figure]
: lines subject organization com article just don writes university like
[Figure]
topic models, the concatenation operation on the latent representations of dc -ETMs can significantly improve the performance, which can be attributed to enforcing strong links between the multi-layer representations and the observation with the skip connections in the generation.
5.3 Qualitative Analysis
As discussed in Sec. 3.2, the developed dc -ETM inherits both the characteristics of hierarchical topic structure and semantic topic embeddings.
Then we compare the hierarchical topics of a 5-layer dc -ETM trained on 20News with those learned by SawETM for qualitative analysis.
Topic Visualization: With the visualization techniques [8], we exhibit the 5 th-layer topics learned by dc -ETM and SawETM on 20News in Fig.
3 and Fig.
7, where each topic is interpreted by its top-10 words by sorting the word probabilities by descending order.
Obviously, the topics learned by
SawETM are quite similar, explaining the reason why concatenating its hierarchical latent document representations cannot improve and even hurt the performance on downstream tasks.
On the contrary, the developed dc -ETM can learn meaningful and diverse topics at higher layers, indicating that more data information is passed to higher layers to alleviate “ posterior collapse ”.
We also exhibit a 5-layer topic tree learned by dc -ETM in Fig.
7 to illustrate the topic hierarchy of dc -ETM in Appendix N.
Topic Embedding Visualization: After extracting hierarchical topic trees by dc -ETM, we visualize some of these trees originated from different topic nodes at layer 5 by projecting their semantic embeddings with t-SNE [45].
As shown in Fig.
5(a), we can find that the topics in the same topic tree tend to be closer than others from different trees in the semantic embedding space and similar phenomenon occurs in the words for describing the same root topic, which indicates the hierarchy learn by dc -ETM is of high quality.
Note that we also visualize the topics consisting of similar top words in Fig. 5(b) and 5(c), learned by dc -ETM and SawETM respectively, which demonstrates that the developed dc -ETM can provide more meaningful and discriminative topic and word embeddings.
Conclusion
To provide higher-quality hierarchical latent representations for deep topic modeling, in this paper, with the deep-coupling generative process, we develop a novel dc -ETM, which is constructed by introducing skip connections into the generative process of GBN and also incorporates both topic [Figure] [Figure] [Table]
article writes think [Table] news [Table]
deleted actually just, true, did
[Table]
nd
[Table]
topics
[Table]
[Figure]
embedding and Weibull reparameterization techniques.
Utilizing the property of sequence-like generation process, we design a PG-based training algorithm for dc -ETM to further alleviate the information reduction at higher layers.
We note that the main idea of designing dc -ETM equipped with the PG-based training algorithm can potentially be extended to other deep topic models
Acknowledgments
This research is supported in part by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP)
Funding Initiative.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Appendix A Inference Network of dc -ETM
[Equation]
[Figure]
[Equation]
[Figure]
As a usual VAE-like model, the inference network (encoder) in dc -ETM is designed to approximate the gamma distributed posteriors of latent representations of the corresponding generative model (decoder), specifically p ( θ n | x n ) defined in Eq.
As the gamma distribution, which can generate sparse and non-negative random variables, is not reparameterizable with respect to its shape parameter, we introduce the Weibull distribution [20] to construct our inference network due to its attractive properties as discussed in Appendix B, one of which is that a latent variable x ∼ Weibull ( k, λ ) can be easily reparameterized as
[Equation]
Specifically, following traditional VAEs, we first factorize the variational posterior distribution q ( θ n | x n ) defined by the inference network in a hierarchical manner as follows
[Equation]
which is expected to be flexible enough to well approximate the true posterior distribution p ( θ n | x n ) .
Then, for the design of the network structure, we develop a novel Weibull-based upward-downward inference network, which contains both bottom-up deterministic and top-down stochastic paths, contributing to reducing the noise in the procedure of inferencing stochastic latent variables that are higher in the hierarchy [16, 17].
As the inference network shown in the left part of 6, the bottom-up deterministic path takes the BoW vector x n as input to obtain hierarchical deterministic latent representations { h ( l ) n } Ll =1 as follows:
[Equation]
[Equation]
through specifically defining h n = MLP ( x n ) and h (1) n = MLP ( h n ) with MLP indicating a twolayered fully connected network and ⊕ being the concatenation operation at feature dimension.
Then, each deterministic latent representation h ( l ) n is further transferred into
[Equation]
where Linear is a dense fully connected layer.
Finally, taking Φ ( l +1) θ ( l +1) n as the prior information passing from the deeper layer, the variational posterior q ( θ ( l ) n |− ) can be obtained with the stochastic up-down path as
[Equation]
where Softplus function is applied to ensure positive Weibull shape and scale parameters; ˆ k ( l ) n and ˆ λ ( l ) n incorporate the information passing from h ( l ) n ; and θ ( l ) n can be obtained with the reparameterization technique defined in Eq.
We emphasize that the inference network of our dc -ETM can be naturally treated as an inverse procedure of the generation process of x n , whose generation is conditional on the latent representations { θ ( l ) n } Ll =1 across all hidden layers.
Inverse to the generative network, the inference network can directly inject the data information into the latent representation of each layer through h n and allow all the stochastic latent variables { θ ( l ) n } Ll =1 to have a deterministic dependency on the observation x n , empirically alleviating top stochastic latent variables from being collapsed [17].
B Properties of Weibull distribution • Similar density characteristics with Gamma Distribution
The Weibull distribution has similar characteristics with gamma distribution, i.e. , the density functions of the two distributions are quite similar
[Equation]
• Easily Reparameterization
The latent variable x ∼ Weibull ( k, λ ) can be reparameterized as
[Equation]
24) leading to an expedient and numerically stable gradient calculation.
• Analytic KL-Divergence
The KL-divergence between the Weibull and gamma distributions has an analytic expression formulated as
[Equation]
C PG-based Training Algorithm
Modeling the generation of long sequence with RL-based methods has achieved great success, but still suffers from high variance during model training.
The common ways to conduct policy gradient method for training include: i) employ the score-ratio gradient approximator like REINFORCE [46] to backward the gradients from Q-value Q π ( s ( l ) n , a ( l ) n ) to the parameters of the policy network π ; ii) introduce an additional “critic” network to estimate the Q-value Q π ( s ( l ) n , a ( l ) n ) as ˆ Q π ( s ( l ) n , a ( l ) n ) , then backward the gradients from estimated Q-value ˆ Q π ( s ( l ) n , a ( l ) n ) to the parameters of the policy network π through methods like deep deterministic policy gradient (DDPG) [34].
We emphasize that, both methods above are developed for the RL applications where the Q-value function is not differentiable, e.g., the reward r ( s ( l ) n , a ( l ) n ) at every time step l is given by the non-differentiable black-box environment simulator, or for the applications with hundreds or even thousands time steps, where the direct computation of the gradient from Q π ( s ( l ) n , a ( l ) n )
=
r ( s ( l ) n , a ( l ) n ) + E π (cid:104)(cid:80) l − 1 i =1 γ
i r ( s ( l − i ) n , a ( l − i ) n ) (cid:105) to π is very expensive.
However, the REINFORCE estimator is notorious for its high variance, which would cause the learning of long sequence hard to converge, and the introducing of the “critic” network would also bring extra bias for the gradients and computation burden.
Moving beyond the gradient estimator of REINFORCE method or introducing the critic network, we design a more stable way to model the sequential generative process in deep topic model here.
Specifically, the number of time steps here is the number of layers L of the topic model, e.g., L isin this paper, which is a much smaller number compared to the hundreds of time step in other RL applications.
Besides, we design the r ( s ( l ) n , a ( l ) n ) as a totally differentiable function in Eq.
14, which leads to a differentiable Q-value function Q π ( s ( l ) n , a ( l ) n ) .
More specifically, recall the purpose of policy gradient method is to maximize the Q-value by applying gradient descent to the policy model π , we develop the Policy Gradient based variational inference algorithm for dc -ETM in Algorithm 1.
Algorithm
Set minibatch size m and the number of Layer L ;
Initialize the encoder parameters and decoder parameters Ψ ; for iter do
Randomly select a minibatch of m documents to form a subset X = { x j } 1 ,m ;
Draw random m uniform distribution; for do
; end for
D Datasets
R8 is a subset of 7,674 documents selected form 8 different review groups of the Reutersdataset, and has been split into a training set of 5,485 documents and a testing set of 2,189 ones.
20News dataset consists of 18,774 documents from 20 various new groups and has been split into a training set of 11,314 documents and a testing set of 7,532 ones.
RCV1 is an archive of 804, manually categorized newswire stories made available by Reuters.
WebKB is a dataset that includes web pages from computer science departments of various universities, which has 4,518 web pages that are categorized into 6 imbalanced categories.
E Error Bars
We randomly run 5 seeds for our method in experiments and report the error bar as below.
[Table] [Table]
F Comparison of topic quality
To make an intuitive comparison on the aspect of topic quality, we visualize the 5-layer topics learned by dc -ETM and Sawtooth on 20News dataset as shown in .
Obviously, the topics learned by Sawtooth are quite similar, which potentially explains the reason why concatenating its hierarchical latent document representations cannot improve and even hurt the performance on downstream tasks.
On the contrary, the developed dc -ETM can learn meaningful and diverse topics in higher layers, indicating that more data information is passed to higher layers to alleviate the phenomenon of collapse
[Figure]
G Limitation
The limitation could be the computation burden brought by the extra L − 1 training steps due to the
Policy Gradient training method.
However, since L is a small number, the extra computation burden is affordable.
Besides, compared with the backbone GBN-based deep topic model, our method would not bring extra computation burden during the testing stage, but leads to performance improvements as shown in Section 5.
H Broader Impact
At first, we need to emphasize that this work is developed to mitigate the phenomenon of information decay in deep topic models, which has been widely disclosed in the topic modeling literature but few efforts have been made to address this challenge.
The main difficulty is the need for carefully designing the probabilistic generative process to build the effective connection between the observation and the corresponding latent representations, on the premise of preserving the interpretable hierarchical topic modeling structure, rather than casually introducing skip-connections.
Utilizing the natural hierarchy in deep topic models, we provide a general solution equipped with a novel perspective to incorporate RL-based training algorithm, which brings quite a few contributing ideas to this field.
(1) n  (2) n  (3) n  × (0) (0) T
K D    (3) (3) D K    (3) ˆ  × (0) (0) T
K D    (2) (2) D K    (2)
ˆ  × (0) (0) T
K D    (1) (1) D K    (1) ˆ  × (1) (1) T
K D    (2) (2) D K    (2)  × (3) (3)
D K    (2) (2) T
K D    (1) n x (2) n x (3) n x n x (1) n h (2) n h (3) n h n x
Generative Network: Deep-Coupling topic Embeddings guided GBN Inference Network n h (3) 
Please note that, we take an NTM named SawETM as an example in this paper to illustrate how we develop a dc -ETM, but dc -ETM can be also applied to extend NTMs with similar structures.
Most of exsiting NTMs adopt VAE-liked structures, but none of them attempt to solve the “ posterior collapse ” phenomenon in essence, resulting in that the latent representations at higher layers exhibit similar or meaningless patterns (as shown in the Appendix F).
The significance of developing deep-coupling structure with RL-based training algorithm, which has effectively improved the quality of the latent representations of a deep topic model at higher layers, goes beyond a single specific model.
I The implementation details of dc -ETM variants
To have an intuitive understanding about the implementation details of the projection metrics in dc -ETM, we provide an overview of the network structure of a 3-layer dc -ETM in Fig.
As shown in Fig.
8, the topic matrix Φ ( l ) at layer l can be factorized into the product of two topic embedding matrices.
Then, there are two kinds of choices to obtain the projection matrices, leading to the variants distinguished by the suffix − α and − β .
Specifically, for the variant dc -ETM-
α defined in Eq.
(2), the projection matrix ˆ Φ (3) can be obtained by multiplying Φ (1) Φ (2) Φ (3) , and then the augmented observation vector x (3) n can be generated from the Poisson distribution with a rate of α (3) ˆ Φ (3) θ (3) n .
The other projection matrices ˆ Φ (2) and ˆ Φ (1) can be obtained in a simliar way, where specifically defining ˆ Φ (1) :
=
Φ (1) .
For the variant dc -ETM- β defined in Eq.
(3), the projection matrix ˆ Φ (3) can be obtained by directly multiplying β (0) T and β (3) , which is more efficient than dc -ETM- α .
Thus, in our consideration, with a shorter path for gradient propagation, the short connections in dc -ETM- β can perverse more data information than those in dc -ETM-
α , resulting in more informative latent document representations to achieve better model performance with less “ posterior collapse ”.
We emphasize that, the projection in dc -ETM-
α could be applied to extend any existing deep topic models, while the projection in dc -ETM-
β is only applicable for the NTMs equipped with topic embedding techniques.
J Preliminary of “ posterior collapse ” J.1 Definition of “ posterior collapse ”
For a VAE-based model consisting of a decoder p θ ( x | z ) and an encoder q ϕ ( z | x ) , the definition of posterior collapse is that the posterior of latent variables, denoted as q ϕ ( z | x ) , collapses to it prior p θ ( z ) , which is a non-informative distribution and independent of the data x .
It can be mathematically denoted as that the KL divergence between q ϕ ( z | x ) and p θ ( z ) is close to zero, represented as D KL [ q ϕ ( z | x ) || p ( z )]
≈ 0 .
J.2 How to measure “ posterior collapse ”
As the definition of “ posterior collapse ”, a promising metric to measure “ posterior collapse ” could be the KL-divergence between q ϕ ( z | x ) and p θ ( z ) , where a smaller KL-divergence score indicates that the posterior q ϕ ( z | x ) contains less data information and has a larger tendency to collapse to its non-informative prior.
In Table 5, we compare our method dc -ETM- β (policy) (for brevity, we note it as " dc -ETM") with the previous SOTA NTM named SawETM with the metric of layer-wise KL divergence on 20News, R8, and RCV1 datasets.
As the results shows, the KL divergence of SawETM gradually reduces to zero with the network going deeper, which indicates the occurrence of a serious degree of “ posterior collapse ” at higher layers and limited data information can be captured by these latent variables at higher layers.
On the contrary, benefiting from the efficient skip connection between the observation x n and multiple latent document representations { θ ( l ) n } Ll =1 , dc -ETM can achieve relatively larger KL-divergence scores at higher layers by alleviating “ posterior collapse ”, leading to more expressive latent document representations for downstream tasks.
KL-divergence of each layer
Metric 20News
R
RCV
Layer #i
SawETM dc -ETM SawETM dc -ETM
SawETM
[Table]
J.3 Why “ posterior collapse ” in a VAE-based model
Besides the experimental analysis, to have a theoretical understanding of the reason why “posterior collapse” widely exists in the higher layers of hierarchical VAEs, we try to explain the phenomenon of “posterior collapse” from the perspective of information theory.
For ease of understanding, we use a vanilla L -layer hierarchical VAE with a top-down inference network as an example, where dc -ETMs can be all treated as VAE-based models.
Specifically, following [18], we extend their theoretical explanation for “posterior collapse” in a single-layer VAE to a multi-layer version.
Separating the latent variables as the lower-level variables z ≤ k = { z 1 , ..., z k } and the higher-level ones z >k = { z k +1 , ..., z L } , where k ∈ { 0 , ..., L − 1 } , then the ELBO for this hierarchical VAE can be reformulated as
[Equation]
(26) where q ϕ ( z L | z L +1 ) :
= q ϕ ( z L | x ) , p θ ( z L | z L +1 ) :
= p θ ( z L ) , and the main contribution to the expected log-likelihood term is coming from the lower-level latent variables z ≤ k before the k -th hidden layer [17].
Once the generation capacity of the generative model p θ ( x | z ≤ k ) is powerful enough to reconstruct the observation x well, the variational posteriors of higher-level latent variables z >k will be optimized to be close to their priors, i.e., q ϕ ( z >k | x )
≈ p
θ ( z >k ) , leading the representations learned by VAE at higher layers to be meaningless and cannot provide faithful summaries for x , which is well-known as the phenomenon of “ posterior collapse ” or “ latent variable collapse ” [18].
To find the potential solutions to alleviating “ posterior collapse ”, in the following, we reinterpret this phenomenon from the perspective of information theory by extending the findings in [18] to a hierarchical VAE scenario.
For ease of undertanding, we define the mutual information between the data x and the higher-level latent variables z >k as
[Equation]
, which is induced by the variational posterior q ϕ ( z >k | x ) .
Then KL term in Eq.
(26) can be rewritten as
[Equation]
where q ϕ ( z >k ) =
E p ( x ) [ q ϕ ( z >k | x )] .
By substituting Eq.
(27) into Eq.
(26), due to the non-negativity of mutual information and KL divergence
[Equation]
we can find that maximizing the ELBO is opposite to maximizing the mutual information I q ( x , z >k ) .
When I q ( x , z >k ) is minimized to zero, the variational posterior q ϕ ( z >k | x ) will be independent of the data x , which leads to the phenomenon of “ posterior collapse ”.
Thus, exploiting the property of the deep NTMs and building the skip connection between the x n and θ ( l ) n with two projection variants, as shown in Fig. 8, is promising to force the higher layers of latent variables be more informative by maximizing the mutual information
[Table]
K Evaluations of topic quality under more metrics
For evaluating the quality of the learned topics, we introduce more metrics for comparison between
SawETM and dc -ETM- β (policy) (for brevity, we note it as " dc -ETM") including: topic diversity, topic coherence, and topic quality (a product of the topic diversity and topic coherence).
As shown in
Table 6, Table
[Table]
and Table 8, dc -ETM can achieve comparable performance on the aspect of topic quality at the first layer while significantly outperform SawETM in the higher layers
[Table]
20News
[Table] [Table]
L Layer-wise comparison under the clustering task
We provide an additional evaluation for the clustering tasks on each layer of SawETM and dc -ETMβ (policy) (for brevity, we note it as " dc -ETM").
As shown in Table
[Table]
the higher layer θ ( l ) n of
SawETM can only preserve limited data information for the downstream document clustering task, which could explain why the SawETM cannot obtain a gain of performance improvement even if concatenating all hidden layers, specifically { θ ( l ) n } Ll =
(denoted as "All")
[Table]
M Visualization of topic embeddings learned by different variants
To investigate the effects of dc -ETM variants α , β , and policy , we provide the t-sne visualization of 5th-layer topic embeddings learned by these variants.
Since the semantics of learned topics vary with the model, to avoid cheery-picking and keep fair, we visualize the whole embeddings of 16 topics at the 5th layer.
Note that, the hyper-parameters of the variants for t-SNE visualization are all the same.
Comparing Fig.
9 (SawETM) with Fig.
10 ( dc -ETM- α ), it could be obviously seen that the 5thlayer topics learned by SawETM are similar and meaningless, which means that the introduced skip-connection in dc -ETM-
α is helpful to learn meaningful topics at higher layers.
Comparing Fig.
11 ( dc -ETM- α -policy) with Fig.
10 ( dc -ETM- α ), it could be seen that the learned topics are more distinguishable in dc -ETM-
α -policy, which indicates the effect of the policy gradient training schema.
Comparing Fig.
12 ( dc -ETM- β ) with Fig.
10 ( dc -ETM- α ), it could be seen that the learned topics are more distinguishable in dc -ETM- β , which indicates the effect of the efficient projection method β .
In all, combining the projection method β and the policy gradient training schema, we could acquire a significantly meaningful higher layers’ topic embedding in Fig.
13 than SawETM in Fig.
5_ lines subject organization like writes article just don know think 5_ lines subject organization like writes article com posting host just 5_ lines subject organization posting host nntp don like writes just 5_ lines subject organization like writes article com just know don 5_ lines subject organization writes like article just posting don host 5_ lines subject organization writes like article just posting don host 5_ lines subject organization posting host nntp don like writes just 5_ new government law public time world people used year high 5_ lines subject organization posting host nntp don like writes just 5_ lines subject com organization like just don know people does 5_ lines com subject organization like just don know people does 5_ subject lines organization com writes article university posting host nntp 5_ lines subject organization com posting host reply nntp distribution university 5_ like read world post case does did time make hard 5_ lines subject organization com posting host reply nntp distribution university 5_ subject lines organization com writes article university posting host nntp
SawETM-5-layer
5_ people general like based point subject lines public fact organization 5_ just good seen heard idea type haven wondering bad comes 5_ people general like based point public fact example subject non 5_ lines subject organization just people say general like think don 5_ people general like based point public fact example subject non 5_ help thanks looking info email advance appreciated stanford information paul
_ people general like based point public fact subject example lines 5_ say right don think things course
yes sure know mean 5_ data ftp image graphics software available pub filesfile package 5_ lines organization subject want let writes just article posting host 5_ posting host nntp wpi originator tim lines organization subject limit 5_11 writes article apr deleted sure actually phil path funny
interesting 5_ lines subject organization want let com university data ftp single 5_ data ftp image graphics software available pub files file package 5_ lines subject organization com university want just let writes state 5_ did said got didn away thought went day came couldn dc-ETM- -5-
layer
5_0 drug drugs health medical study doctors disease treatment diet cancer 5_1 solar planet earth space spacecraft temperature moon henry mars surface 5_ ftp pub graphics image format images file free files data 5_ subject lines organization university posting nntp host know thanks new 5_ com writes subject lines organization article sun don posting apr 5_ god jesus christ sin church love bible rutgers christians sex 5_ uiuc people religion evidence claim say christian true cso belief 5_ israel jews jewishisraeli muslimmuslims armenians arab armenian people 5_ car like bike dog year cars dod
speed oil bmw [-
 _-]5_ key chip encryption clipper government algorithm use pgp keys law 5_ like just time way don people going think make good 5_ drive card disk drivers modem work hard use problem faster 5_12 game games team hockey year baseball espn fans season play 5_ windows dos file display run use program files window manager 5_14 writes lines subject organization article posting nntp host university don 5_ koresh people fbi gun government children waco law did rights dc-ETM- -policy-5-layer
[Figure]
gamesbaseballplayers boston season hockey
[Figure]
black brian kent
[Figure]
sandvik
[Figure]
image graphics data bit file available ftp format images files
[Figure]
say people church jesus christ believe faith christian bible
[Figure]
posting host nntp lines organization subject university distribution andrew usa 5_ medicaldiseasehealth key clipper cancer care use drug government 5_6 don like just know time think good way make does 5_ government people fbi koresh gun children batf armenian waco said 5_8 power blue use green used red low equipment monitor sound 5_ drive sale card video price mac disk scsi hard modem 5_ com lines writes subject organization article sun distribution netcom ibm 5_ windows dos file use run program using microsoft server window 5_ war israel jewish muslims islam arab jews israeli muslim peace 5_ lines organization subject
thanks know does mail help like version 5_ space gov nasa car bike hst access digex dod net 5_15 organization university lines writes article subject apr news john state dc-ETM- -5-layer
5_ game teamgames year baseball fans players player play hit 5_1 armenian turkish armenia armenians turks serdar people argic muslim soviet 5_2 image images file software jpeg ftp graphics format data available 5_ god christian people jesus bible homosexual paul church christians homosexuality 5_ posting host nntp subjec t lines organization distribution university state usa 5_5 key clipper chip use number information new encryption public book 5_ don just like think know time people good way really
_ government fbi people koresh children batf gun police waco president 5_8 bit data audio function use color bits imageimages used 5_ drive card mhz bus new scsi price sale cpu ide 5_ com writes article lines organization subject netcom ibm opinions distribution 5_11 dos windows server file mousefonts microsoft network font memory 5_ israel jews jewish war muslims muslim peace people religious world 5_ subject lines organization help know thanks mail does like list 5_ car nasa space gov new miles bike dod mission access 5_ writes article organization lines apr subject university news uiuc michael dc-ETM- -policy-5-layer