Paper:
Sum-Product networks (SPNs) are expressive deep architectures for representing probability distributions, yet allowing exact and efficient inference. SPNs have been successfully applied in several domains, however always as black-box distribution estimators. In this paper, we argue that due to their recursive definition, SPNs can also be naturally employed as hierarchical feature extractors and thus for unsupervised representation learning. Moreover, when converted into Max-Product Networks (MPNs), it is possible to decode such representations back into the original input space. In this way, MPNs can be interpreted as a kind of generative autoencoder, even if they were never trained to reconstruct the input data. We show how these learned representations, if visualized, indeed correspond to “meaningful parts” of the training data. They also yield a large improvement when used in structured prediction tasks. As shown in extensive experiments, SPN and MPN encoding and decoding schemes prove very competitive against the ones employing RBMs and other stacked autoencoder architectures.
1 INTRODUCTION
On a high level, the generative approach to machine learning can be described as follows: Given a set of samples D, drawn (usually i.i.d.) from an unknown distribution p∗ over random variables (RVs) X, recover p∗ from D. To a certain extent, generative learning (GL) can be seen as the “kingclass” paradigm in machine learning. It is well known that an optimal predictor – given an additional loss function – can just be derived from p∗. For example, assuming that Y is a class variable and X are observed features, the classifier with minimal expected 0/1-loss is given by argmaxy p ∗(y,X).
It is therefore not surprising that GL and representation learning (RL) (Bengio et al., 2012) are highly related, as both aim at “formally understanding” data. GL can be described as a “black-box” approach, since we are usually interested in the capability of some model pθ to capture the underlying distribution p∗. In RL, however, one may be interested in interpreting the “inner parts” of pθ as abstract features of the original raw data. Both perspectives can be seen in the seminal RL approaches (Hinton & Salakhutdinov, 2006; Bengio et al., 2006), as the activations of generatively trained models are employed as data representations for initializing deep architectures.
As another simple example, consider a Bayes classifier, which estimates the joint distribution p(Y,X) by using the class-prior p(Y ) and class-conditionals p(X |Y ). In a purist GL view, we estimate p(Y ) and p(X |Y ) to compute p(Y,X) = p(X |Y ) p(Y ) ∝ p(Y |X). In an RL approach, however, we would recognize that the parts of our model p(X |Y ) (or also p(Y |X)) can be interpreted as a kind of soft one-hot encoding for Y , and would use them as features in a discriminative approach. The same argument holds for an unsupervised learning scenario, i.e. when Y is unobserved: we would deal with latent mixture models for which p(X |Y ) are the mixture components. In summary, we
note that any generative model – depending on its structure and semantics – might be a potential feature extractor and thus a potential useful representation learner.
In this paper, we investigate a particular promising candidate for this approach, namely Sum-Product Networks (SPNs) (Poon & Domingos, 2011), recently proposed deep probabilistic networks, admitting exact but tractable inference for several kinds of probabilistic queries. SPNs have been successfully applied to computer vision (Gens & Domingos, 2012; Amer & Todorovic, 2015), speech (Peharz et al., 2014b; Zöhrer et al., 2015) and language modeling (Cheng et al., 2014). In these works, however, SPNs have been used only as black-box distribution estimators.
Here we exploit SPNs for RL. One way to interpret SPNs is as a hierarchically structured generalization of mixture models: they are nested arrangements of factorized distributions (product nodes) and mixture distributions (weighted sum nodes) defining distributions over subsets of X. Due to this peculiarity, representations extracted from an SPN by evaluating the network nodes extend the idea of using mixture components as features, as in the motivating example above, in a recursive way. In Vergari et al. (2016) some initial approaches to encode embeddings via an SPN were proposed, showing how these model can constitute an interesting alternative or addition to other popular generative feature extractors such as RBMs (Hinton & Salakhutdinov, 2006; Marlin et al., 2010). The advantages of employing SPNs for RL are that one can “easily” learn both structure and parameters by leveraging the SPN’s recursive probabilistic semantics (Gens & Domingos, 2013), rather than imposing an a-priori structure or using an ad-hoc weight learning algorithm, as usually done for other deep architectures. Rich hierarchical features can be obtained even by such a simple generative learning scheme. Indeed, in an SPN each node can be seen as a probabilistic part-based feature extractor. Visualizations of the filters learned by SPNs trained on images data confirm that these networks are able to learn meaningful representations at different levels of abstraction.
In this work we provide a way to decode the learned representations back to their original space by employing a Max-Product Network (MPN) (Poon & Domingos, 2011). Our decoding procedure leverages the Most Probable Explanation (MPE) (Darwiche, 2009) inference routine for SPNs and incorporates an imputation mechanism for missing components in a representation to be decoded. To a certain extent, an MPN can be exploited as a kind of generative autoencoder. We continue the work of Vergari et al. (2016) by adding other ways to leverage SPN representations, again for “free”, i.e. without training the network with the aim to reconstruct its input. Additionally, we characterize conditions when MPNs can be considered perfect encoder-decoders under the proposed scheme. As a final contribution, we evaluate the meaningfulness and usefulness of SPN and MPN representations in an extensive set of structured output prediction tasks. Having devised a decoding procedure allows us to explore different learning scenarios, e.g. building embeddings for the input features, for the labels or for both. We demonstrate that these encoding and decoding schemes, “cheaply” obtained by a generative SPN, show surprisingly competitive performances when compared to those extracted from RBMs, probabilistic autoencoders (Germain et al., 2015) and deep autoencoders tailored for label embeddings (Wicker et al., 2016) in all the learning scenarios evaluated.
2 SUM-PRODUCT NETWORKS
Let RVs be denoted by upper-case letters, e.g. X , Y and let corresponding lower-case letters denote their values, e.g. x ∼ X . Similarly, boldface notation denotes sets of RVs and their combined values, e.g. X, Y and x, y. For Y ⊆ X and a sample x, we denote with x|Y the restriction of x to Y. An SPN S over a set of RVs X is a probabilistic model defined via a rooted DAG. The leaves of the graph (the SPN’s inputs) are computationally tractable, possibly unnormalized distributions over a sub-set of X. When n is a leaf of S, let φn denote its associated distribution. The inner nodes compute either weighted sums or products over their children. Let ch(n) be the set of children for a particular node n. For a sum node n and a child c ∈ ch(n), we associate a nonnegative weight wnc with the outgoing sum-edge n→ c. The set of all sum weights in S (the network parameters) is denoted as w. Furthermore, let S⊕ (resp. S⊗) be the set of all sum (resp. product) nodes in S.
Let Sn denote the sub-network rooted at node n and parametrized by wn. Each node n in S defines a probability distribution pwn over its scope by normalizing the output of Sn. Consequently, the distribution of S over all X is defined as the root normalized output. Sn(x|sc(n)), or short-hand Sn(x), indicates the output value of node n when X = x is observed as the network input.
Let sc(∗) denote the scope, a labeling function associating to each node n a subset of X, i.e. sc(n) ⊆ X. For a leaf n, sc(n) is the set of RVs over which φn is defined. The scope of an inner node n is defined as sc(n) = ⋃ c∈ch(n) sc(c). The scope gives rise to some fundamental properties of an SPN: S is complete if ∀n ∈ S⊕ and ∀c1, c2 ∈ ch(n) : sc(c1) = sc(c2). S is decomposable if ∀n ∈ S⊗ and ∀c1, c2 ∈ ch(n), c1 6= c2 : sc(c1) ∩ sc(c2) = ∅ (an example in Figure 1a). While inference in unconstrained SPNs is intractable, marginalization in complete and decomposable SPNs reduces to performing the marginalization task at the leaves and evaluating the inner nodes as usual (Poon & Domingos, 2011; Peharz et al., 2015). An SPN is locally normalized when it holds that ∀n ∈ S⊕ : ∑c∈ch(n) wnc = 1 and all leaves represent normalized distributions. For complete, decomposable and locally normalized SPNs, the distributions of all nodes are already correctly normalized distributions. In the following, we only consider this class of SPNs.
While marginalization can be tackled in time linear in the network size, the problem of finding a Most Probable Explanation (MPE) is generally NP-hard in SPNs (Peharz et al., 2016). Given two sets of RVs Q,O ⊂ X, Q ∪O = X and Q ∩O = ∅, inferring an MPE assignment is defined as finding
x∗|Q = argmax q∼Q p(o,q). (1)
However, MPE can be solved exactly in selective SPNs (Peharz et al., 2014b; 2016), i.e. SPNs where it holds that for each sample x at most one child of each sum node is non-zero. MPE in selective SPNs is solved via MPEAssignment (Poon & Domingos, 2011), which evaluates the network twice. First one builds an MPN M from S by replacing each node n ∈ S⊕ by a max node n ∈Mmax computing maxc∈ch(n) wncMc(x) and each leaf distribution by a maximizing distribution (Peharz et al., 2016) (Figure 1b). One then computes M(x|O) – the MPE probability of the query p(x|O) – by evaluating M bottom-up (Figure 1c). Stage two consists of a top-down traversal of M . Starting from the root, one follows the maximal child branch for each max node and all child branches of a product node. Each partial input configuration determines a unique tree path. The MPE assignment x∗ is obtained by collecting the MPE solutions (w.r.t. Q) of the leaves in the path (Figure 1d). For selective SPNs, the corresponding MPNs compute precisely the same value for each node, since sums and maxes are equivalent when applied to all zeros but one nonnegative value. In the non-selective case, MPNs can be seen as a (lower-bounding) approximation of SPNs, and are thus also an interesting candidate for RL, as showed in the next sections. Furthermore, while MPEAssignment solution for general SPNs is not exact, it is still employable as a reasonable and common approximation (Peharz et al., 2016).
SPNs and MPNs can be interpreted as very peculiar deep Neural Networks (ANNs) that are labeled, constrained and fully probabilistic (Vergari et al., 2016). They are labeled networks because of the scope function, which enables a direct encoding of the input (Bengio et al., 2012). Their topology is constrained because of the completeness and decomposability properties, hence connections are sparse and not dense. Differently from other distribution estimators like NADEs (Larochelle &
Murray, 2011) and MADEs (Germain et al., 2015), they are fully probabilistic: not only the network outputs emit valid probability values, but each inner node as well, due to their recursive definition.
The semantics of SPNs enable the design of simple and yet surprisingly effective structure learning algorithms (Dennis & Ventura, 2012; Peharz et al., 2013; Gens & Domingos, 2013). Many recent attempts and variants (Rooshenas & Lowd, 2014; Adel et al., 2015; Vergari et al., 2015) build upon the currently most prominent algorithm LearnSPN, a greedy top-down SPN learner introduced in (Gens & Domingos, 2013). LearnSPN proceeds by recursively decomposing a given data matrix along its rows (i.e. samples), generating sum nodes and estimating their weights, and its columns (i.e. RVs), generating products. To a certain extent, LearnSPN can be interpreted as a recursive data crawler, extracting peculiar features from a data matrix, which potentially only live in a particular data cluster and/or in a certain subset of RVs. This may be one of the few cases when the structure and parameters of an ANN can be learned without directly optimizing a loss function.
3 LEARNING REPRESENTATIONS WITH SPNS AND MPNS
In this section we discuss how to exploit an SPN S or its corresponding MPN M for RL, after structure and parameters are generatively learned over X, following Vergari et al. (2016). We are interested in encoding each sample xi ∼ X into a continuous vector representation ei in a new d-dimensional space, i.e. an embedding ei ∈ EX ⊆ Rd, where ei = fS(xi) (SPNs) or ei = fM (xi) (MPNs). We usually refer to SPNs, since most of the time similar consideration hold also for MPNs.
For ANNs, the common approach is to use the hidden neuron activations of the upper layers as the learned representations for f . As argued above, SPN nodes are particular interpretable due to their clear probabilistic semantics. Given an SPN S and a set of nodes N = {nj}dj=1 ⊂ S, we construct our embedding as eij = Snj (x|sc(nj)) = pwnj (x|sc(nj)), where a reasonable selection criterion for N is given below. Each value represents the probability to see that sample according to a marginal distribution over a node scope. Thus, the so-constructed embedding is a point in the geometric space induced by a collection of proper probability densities.
SPN nodes can also be seen as part-based filters operating over sub-spaces given by the node scopes. Sum nodes can be interpreted as filters built by weighted averages over filters sharing the same scope, and product nodes can be seen as compositions of filters over non-overlapping scopes. From the perspective of the internal mechanisms of LearnSPN-like algorithms, each filter captures a different aspect of sub-population and sub-space of the data. Thereby, the scope information induces a hierarchy of filters at different levels of abstraction.
To confirm this interpretation, we visualize the features extracted from nodes in an SPN learned on image samples (Vergari et al., 2016). For ANNs, the feature filtered by a hidden neuron can be visualized as the image in the input space that maximally activates that neuron (Erhan et al., 2009). In SPNs this corresponds to solving MPE for the sub-SPN rooted at a particular node, and restricted to the node’s scope. As stated in Section 2, we employ MPEAssignment as an approximation to this generally hard problem. Figure 2 shows some of the MPE solutions/filter activations for an SPN trained on a binarized version of MNIST (Larochelle et al., 2007) (see Appendix C for details). Note that they resemble part-based features at different levels of complexity: from small blobs (Figure 2a) to shape contours (Figures 2b and 2c), to full digits comprising background parts (Figure 2d). The missing background pixels, visualized in a checkerboard pattern, is due to those pixels being out of the scope for those nodes. This pattern locality is an SPN peculiarity: although also fully connected ANNs typically show locality (e.g. edge filters), the locality information is explicitly encoded in SPNs via the node scopes. This suggests that the scope information alone may already be able to convey a meaningful representation of “object parts”, e.g. see the ‘O’ shapes in Figure 2. Also, note that filters appear qualitatively different from most classical ANNs, which motivates to combine SPN features with those from other deep architectures, an approach worth further investigation.
While in classical deep ANNs the layer depth is usually associated with the level of abstraction of its filters (Erhan et al., 2009; Zeiler & Fergus, 2014; Yosinski et al., 2014), note that this does not easily translate to SPNs. First, even rather simple models might yield extremely deep networks, when translated into SPNs. For example, when representing a hidden Markov model (HMM) as SPN Peharz et al. (2014b), the SPN’s depth grows linearly in the length of the HMM. Thus, representations learned by SPNs are not easily arranged in a meaningful layered hierarchy, due to their constrained
topology and how they are learned. Moreover, LearnSPN-like algorithms can introduce nodes with very different scope information at the same level of depth. This also occurs when compiling SPNs into a minimal layered structure (Vergari et al., 2016).
Therefore, we suggest that rather the scope length |sc(n)| of a node n should be associated with its level of abstraction. The filter activations in Figure 2 give confirming evidence for our conjecture. Thus, when the aim is to compress data into an abstract representation of at most d dimensions, one reasonable filter criterion for SPN/MPN representations would be to collect the d nodes with largest scopes. Clearly, the smaller we choose d, the smaller will be the theoretically achievable quality of the reconstructed data. In our experiments, we leverage SPN representations and decoding schemes by adopting full embeddings, comprising all nodes activations (all colored values in Figure 1c), and inner embeddings, dropping out the leaf information (only orange values in Figure 1c), according to the observation that the number of leaves is overabundant w.r.t. inner nodes in SPNs built by LearnSPN (Vergari et al., 2016). Note that, in both cases, the embedding size d is adaptively induced by the data, when building the SPN, without the need to fix or tune it beforehand.
4 DECODING REPRESENTATIONS
Now we tackle the task to revert SPN representations back to the input space, i.e. to find an inverse transformation g : EX → X such that xi ≈ x̂i = g(f(xi)). Being able to decode representations extends the ways one can exploit SPNs for RL to new learning scenarios for predictive tasks. For example, if one were to learn a classifier from features X to labels Y, he could train the classifier to predict the label embeddings EY rather than Y directly. Then, the predicted embeddings could be turned into the actual outputs by applying g for decoding. By disentangling dependencies over Y in the new space EY, one can obtain better predictors. Following this principle, label embeddings have been greatly employed in RL for structured output prediction. One common approach is to compress labels into a lower dimensional space, then a regressor is trained to predict such embeddings and the predictions are decoded (decompressed) by an inverse linear transformation Bhatia et al. (2015); Akata et al. (2013). The advantage of the decoding scheme we propose is that g does not need additional training to be learned, rather it is provided by an already learned SPN turned into MPN.
Let a perfect encoder-decoder be a pair (f, g) such that, for each x ∼ X, its reconstruction is the exact same sample, i.e. g(f(x)) = x. In our analysis, we focus on MPNs and characterize when they can be used as perfect encoder-decoders. In practice, autoencoders, for which f and g are learned from data, are usually not trained to be perfect encoder-decoders, as they often might learn trivial representation, such as the identity function. This seems not to be an issue for MPNs, since the learning phase is decoupled from the decoding one. We will also empirically confirm it in Section 5.
Given an MPNM , the encoder function fM is given by collecting activations as illutrated in Section 3. Concerning the decoder, we propose a procedure for gM that mimics the MPEassignment algorithm as presented in Section 2. Recall that MPEAssingment finds a solution in the input space (top-down phase) after probabilities, i.e. node activations, are evaluated (bottom-up phase). Consider Eq. 1 in the case in which Q = ∅ and sample xi is fully observed. If all the activations from the bottom-up phase are collected into an embedding ei, its components will exactly determine the top-down descending
phase, i.e. which branch to take when encountering a max node. As a consequence, the set of leaves in the traversed tree path completely depend on ei. This is also true if ei components are not determined from a bottom-up phase but come from the “outside”, e.g. they are predicted. In order to completely define gM , each leaf node encoding φn reached in the top-down phase has to provide itself a decoder function gφn , operating over its scope. Similarly to MPEAssignment, a fully decoded embedding is then constructed by collecting the reconstructions at the leaves according to each gφn decoder.
In practice, we are interested in decoding embeddings that have been predicted by some learned model, i.e. the decoding phase is not applied to the embeddings obtained by directly evaluating a network. Nevertheless, it is important to determine under which circumstances these models behave as perfect encoder-decoders when transforming each instance to a new representation and back. Proposition 1. If for an MPN M over X there exist a perfect encoder-decoder for each leaf distribution φn and it holds for each max node n ∈ M that there is only one child node c ∈ ch(n) for which Mn(x) = wncMc(x), given x ∼ X, then M is a perfect encoder-decoder.
Proof. It is easy to demonstrate this by inductive reasoning. If M comprised only a leaf node, then it would be a perfect encoder-decoder by definition. If it were composed by a product node over child encoder-decoder MPNs, then each input could be reconstructed perfectly by the composition of the reconstruction of the child MPNs. Lastly, if it were composed by a max node over child perfect encoder-decoder MPNs Mc, c = 1 . . . k, then it would also be a perfect encoder-decoder since for each possible input, only one child component Mc∗ would output a value s.t. Mn = wnc∗Mc∗ .
From Proposition 1 it follows immediately that deterministic MPNs can be perfect encoder-decoders. Proposition 2. An MPN M constructed from a selective SPN (Peharz et al., 2014a) S is a perfect encoder-decoder, provided that the leaves have perfect encoder-decoder functions.
Thus, to complete our decoding procedure, we still have to cope with the leaf decoder functions. We define the decoded state for a leaf n as the configuration over its scope that minimizes some distance D over the leaf activation value and its encoded representation: x̃|sc(n) = argminu∼sc(n)D(φn(u)||fMn(x)). In our experiments we will employ simple L1 distance |φn(u) − fMn(x)|. Unfortunately, decoding is ambiguous for most interesting leaf distributions, such as Gaussians. However, this approach works well for discrete data used in our experiments, as long as the state probabilities are mutually distinct. In future work, we will explore techniques to disambiguate decoding the leaves, e.g. by duplicating and splitting Gaussians. In Section 5, we empirically evaluate how good are the decoding schemes depicted here, since it is worth investigating how close to perfect encoder-decoders MPNs learned on real datasets can be.
In order to apply the proposed decoding procedure, a full embedding comprising all the node activations is required. In some real cases (e.g. data compression), only an incomplete embedding, comprising only activations from a subset of the network nodes, is available. For certain incomplete embeddings a full decoding, however, is still possible.
A decodable incomplete embedding e is an embedding such that for each missing activationMn(x) 6∈ e corresponding to a node n ∈M, all the activations ec =Mc(x) ∀c ∈ ch(n) are in e. For such an incomplete embedding, it is sufficient to evaluate the MPN by propagating the embedding activations bottom-up, evaluating parent nodes after their children. The missing embedding components are then reconstructed and the decoding phase for the now full embedding can proceed as before. If even this child information is missing, such a reconstruction is not possible in general. We argue that in such a case, the missing node activations can be reasonable imputed by their most probable value. When encountering a node nj , whose corresponding embedding value ej is not available, ej is estimated as maxu∼sc(nj)Mnj (u) by employing MPEassignment on the sub-networks rooted at nj . Since the MPE activations can be precomputed for all nodes, the complexity of the whole procedure is still linear in the size of M . The pseudocode for the complete decoding procedure is listed in Appendix A.
In our experiments we evaluate the effectiveness and robustness of the decoding procedure both for complete (full) and incomplete predicted embeddings. In particular, for structured output prediction, we employ inner embeddings (cf. Section 3), where leaf values are imputed using MPEassignment as stated above. Moreover, we investigate its resilience when imputing missing at random embedding components either by just replacing them by their MPE value or by additionally evaluating the MPN bottom-up after the missing leaf activations have been imputed.
5 EXPERIMENTS
The research questions we are validating are: i) how good are learned MPNs at reconstructing their input when full/inner embeddings are decoded? ii) how meaningful are the representations learned by SPNs and MPNs and how useful is to predict these embeddings instead of the raw targets and then decoding them? iii) how resilient to missing components are the proposed decoding schemes?
Structured output prediction tasks like Multi-label Classification (MLC) offer a good experimental ground to answer the above questions. In MLC one is interested in predicting the target labels associated to a sample x ∼ X and represented as binary arrays: y ∼ Y. Since there is no unique way to assess a classifier performance in MLC, we measure the JACCARD, HAMMING and EXACT MATCH scores, as metrics highly employed in the MLC literature and whose maximization equals to focus on different sets of probabilistic dependencies (Dembczyński et al., 2012).
For all experiments we use 10 standard benchmark datasets for MLC. To fairly compare all the algorithms in our experiments, we employ the binarized versions of all datasets already processed by Di Mauro et al. (2016) and divided in 5 folds. Detailed dataset statistics are reported in Appendix B.
We learn both the structure and weights of our SPN, and hence MPN, models on X andY separately for each fold by employing LearnSPN-b (Vergari et al., 2015), a variant of LearnSPN (see Appendix C)1. Structural statistics, e.g. the number of inner nodes, for all the models are reported in Appendix D. Please refer to Tables 3 and 4 to determine the extracted embedding sizes.
5.1 RECONSTRUCTION PERFORMANCES
We want to determine how close to perfect encoder/decoders are MPNs learned from real data and equipped with our decoding schemes. In particular, we evaluate their decoding performances when the leaf activations are available (full embeddings), and the decoder employed is the L1 distance, or when they are missing (inner embeddings) and therefore their MPE state is used.
First we turn each learned SPN into an MPN. Then, each model is asked to reconstruct both the training and test samples. Detailed results are reported in Tables 5 and 6, Appendix E.2. It can be observed that the L1 leaf decoder proves to be a very reasonable approximation for binary RVs, scoring very high reconstructions for all the three measures. For the models over Y, the MPE approximation scores surprisingly good reconstructions scoring > 80% EXACT MATCH on half datasets. In general, if the network is small enough, e.g., MPNs learned on the Flags dataset or on Y alone, it behaves as a perfect encoder-decoder for full embeddings. This demonstrates the efficacy of the proposed decoding schemes and shows how the presence of tied max node children activations impacts non-deterministic MPNs learned from data. We investigate if these potentially perfect reconstructions lead to banal representations in the following experiments.
5.2 STRUCTURED OUTPUT PREDICTION PERFORMANCES
We now focus on leveraging the representations learned by SPNs and MPNs in an unsupervised way for structured output prediction. In a fully supervised scenario one wants to build a classifier on the input RVs X to predict the output RVs Y directly (X→ Y). Instead, we can first encode both the input RVs X and/or the target RVs Y into different embedding spaces, EX, EY, and build a predictive model on top of them. In order to do so, we explore different settings: we learn a classifier on the input embeddings instead of the raw features (EX → Y); alternatively, one can first train a regressor on the original input X to predict label embeddings (X → EY), then decoding such predictions back to the original label space; finally, the same regressor can be trained on the input embeddings instead (EX → EY) and its predictions decoded as above. As a proxy measure to assess the meaningfulness of the learned representations, we are considering their prediction performances. Given a predictive model, its improvement in performance in one of the above settings over the raw input/output case, X→ Y, determines how good the representations employed are. To highlight the ability of these representations to disentangle the dependencies underlying the RVs, we always train a simple linear model in all the settings. In particular, we employ an L2-regularized logistic regressor, LR, (resp. a ridge regressor, RR) to predict each RV
1All the code employed for the experiments and visualizations will be made available
in Y (resp. component in EY) independently. Therefore, the most natural baseline to measure the aforementioned representation meaningfulness is to employ the same L2-logistic regressor to the X→ Y setting. We now introduce other models as either encoders or encoder/decoders to plug into our unsupervised settings. The aim is to compare SPN/MPN representations against theirs w.r.t. the aforementioned baseline. Therefore we select them as generative models for which inference can be exactly and tractably computed. For the EX → Y setting, we compare to RBMs (Smolensky, 1986) as highly expressive generative models, for which, while the joint likelihood is intractable for RBMs, exact conditionals of the latent RVs can be computed exactly and have been proven to be very predictive features (Larochelle & Bengio, 2008; Marlin et al., 2010; Larochelle et al., 2010). To evaluate different embedding sizes, we consider RBMs having 500, 1000 and 5000 hidden units (h). A natural competitor for all settings are MADEs (Germain et al., 2015), because they are deep autoencoders which are also tractable probabilistic models. We employ MADEs comprising 3 layers and 500 and 1000 (resp. 200 and 500) hidden units per layer for the EX → Y (resp. X→ EY) setting. Additionally, we add to the comparison MANIAC (Wicker et al., 2016) a non-probabilistic autoencoder model tailored to MLC. In MANIAC, stacked autoencoders are trained to reconstruct Y by compressing each label into a new representation, which, in turn, is used to train a base model exactly as in our X→ EY setting. We employ architectures up to 4 hidden layers with different compression factors. Finally, we employ a max-margin CRF Finley & Joachims (2008), CRFSSVM, in the X→ Y setting that considers a dependency structure on the label space in the form of a Chow-Liu tree. In this way we are able to frame the performances of all the models in the unsupervised setting against a fully supervised and discriminative method on the same datasets.
In Appendix E.1 we report all the choices made to learn and tune the involved models. For SPNs, and hence MPNs, we do not need to define a handcrafted structure a priori like for all the competitors above. Consequently, for RBMs, MADEs, MANIAC it is needed to learn and cross-validate several models with different capacities to obtain properly sized embeddings. On the other hand, the size of embeddings extracted from SPNs/MPNs is adaptively determined by data, as stated in Section 3. The learned embedding sizes are reported in Tables 3 and 4 in Appendix D.
5.2.1 RESULTS AND DISCUSSION
Detailed average fold metrics and their average ranks for all datasets are reported in Tables 9, 7, 8 in Appendix E.3.1. In Table 1, instead, we report the aggregated scores over all datasets d ∈ D for each method f in the form of the average relative improvement w.r.t. the LR baseline: 1 |D| ∑ d∈D scoref (d)−scoreLR(d) scoreLR(d)
· 100. The best models for each setting and score are in bold, the higher their improvement, the better.
In summary, SPN and MPN embeddings proved to be highly competitive and even superior to all other models in the three settings and for all the scores. Even the fully supervised and discriminative CRFSSVM performance are comparable to the best SPN/MPN JACCARD (resp. HAMMING) score in the EX → EY (resp. X → EY) setting, while reporting a largely worse EXACT MATCH improvement than our models in the EX → EY setting. In particular, the setting EX → Y has proven to be hard for many models. This likely indicates that the dependencies on the X might not contribute much to the Y prediction (Dembczyński et al., 2012). Representations from SPNs, even with smaller embeddings than RBMs and MADEs (see Table 3), yield the largest improvements. In the X → EY setting, disentangling the relationships among the Y gives all models a performance boost. This is not the case for MADEs on some datasets, probably due to their reconstruction power being traded off to their generalization power as generative models. MPNs, on the other hand, consistently exploit the label representation space and do not provide overfitted reconstructions. This answers our question about the meaningfulness of MPN representations suggesting that their tendency to be perfect encoder/decoders does not damage their representation performances.
Concerning two decoding schemes we proposed, operating on incomplete (inner) embeddings not only performs comparably to the full case, but also scores the best results on some datasets for JACCARD and EXACT MATCH scores. This aspect can be seen in the EX → EY setting as well.
Additionally, to better understand the role of our decoding procedures in the X→ EY and EX → EY settings, we run a new set of experiments in which the decoding phase for EY is performed by a nearest neighbor (k = 5) model on the basis of the training labelled embeddings. In these settings, therefore, we can add to the comparison even RBM-encoded representations. Even in this scenario, MPN embeddings are the best or very competitive. As expected, the non-linear kNN predictor performs better than our full decoding on several datasets for the JACCARD and EXACT MATCH scores, but less well for the inner variant. This is likely due to the smaller inner embedding sizes and highlights the goodness of the proposed decoding approach in presence of missing values and, more in general, for maximizing the HAMMING score.
All in all, with these structured output prediction tasks we gathered empirical confirmation of the meaningfulness and practical usefulness of SPN and MPN embeddings. The reported large improvements over the three scores cannot be due to SPN/MPN larger embedding sizes. In fact, their sizes are always comparable or smaller than RBM, MADE, and MANIAC ones since the latter max capacities have been chosen after SPNs have been learned (Tables 3 and 4, Appendix D). It is also not possible to state that these representation higher predictive performances are correlated to the SPN ability to better model the data distributions, at least we look at the model likelihoods. Indeed, MADE log-likelihoods have proven to be higher that SPN ones on many datasets and comparable on the rest. We argue that the reason behind these results lies in the hierarchical part-based representations SPNs provide. Each embedding component is responsible for capturing only the significant feature portions according to its corresponding node scope, as shown in Section 3. The meaningfulness of these components as features has to be found in the structure learning performed by LearnSPN-b (Section 2): while its hierarchical co-clustering chooses to split the data into sub-populations in an unsupervised way to determine a reasonable distribution estimation, it highlights meaningful ways to discriminate among them.
5.3 RESILIENCE TO MISSING COMPONENTS
Lastly, we evaluate the resilience of the decoding procedure proposed when label embedding components are missing at random in the X → EY setting. We want to compare the two imputation schemes presented in Section 4: either employing MPEAssignment to retrieve the most probable activation or evaluating the MPN bottom-up to compute the missing predicted components.
For all datasets, for each label embedding that has been predicted, we remove at random a percentage of components varying from 0 (full embedding) to 90%, by increments of 10%. If leaves activations are missing, their MPE activation is considered. After the full embedding has been reconstructed, the decoding phase proceeds as before. Figure 3 shows how the two strategies perform differently for the EXACT MATCH score. The re-evaluation scheme is much more resilient one among the two, being able to maintain comparable scores to the full embedding case up to 30% missing components, then decaying less faster than the MPE based one. The proposed decoding scheme is therefore proved to be not only surprisingly effective but also quite robust. Similar, but less prominent, behaviors are reported for the JACCARD and HAMMING scores in the Appendix.
6 CONCLUSION
In this work we investigated SPNs and MPNs under a RL lens. We suggested an interpretation of MPNs as generative autoencoders by providing a decoding procedure that leverages approximate MPE inference. We characterize when these networks can lead to perfect reconstructions of their inputs, linking this property to determinism. When empirically evaluated in an extensive comparison for MLC, SPN and MPN representations ranked as one of the most predictive features and MPN reconstructions proved to be surprisingly effective. Encouraged by these results, we plan to explore new learning schemes directly exploiting these models learned representatons, and not optimizing their likelihood scores only. For instance, a differentiable procedure for MPE inference would allow SPNs and MPNs to be trained directly to reconstruct or denoise their input, bridging the gap even more between these networks, autoencoders and other ANNs and opening the path to hybridize them.
0.0 0.2 0.4 0.6 0.8 0.10 0.15 0.20 0.25
Arts
0.0 0.2 0.4 0.6 0.8 0.45 0.50 0.55 0.60 MPE eval
Business
0.0 0.2 0.4 0.6 0.8 0.10 0.15 0.20 0.25 0.30
Emotions
0.0 0.2 0.4 0.6 0.8 0.10 0.12 0.14 0.16 0.18
Flags
0.0 0.2 0.4 0.6 0.8 0.20 0.30 0.40 0.50
Health
0.0 0.2 0.4 0.6 0.8 0.16 0.18 0.20 0.22
Human
0.0 0.2 0.4 0.6 0.8
0.22
0.24
0.26 Plants
0.0 0.2 0.4 0.6 0.8 0.10 0.20 0.30 0.40 0.50 0.60
Scene
0.0 0.2 0.4 0.6 0.8 0.04 0.06 0.08 0.10 0.12 0.14
Yeast
Figure 3: Average test EXACT MATCH scores (y axis) obtained by imputing different percentages of missing random embedding components (x axis) for the X → EY setting on all datasets by employing MPE inference (orange crosses) or the bottom-up evaluation imputation schemes (blue squares). Results for Cal dataset are not reported since they are all zeros (see Table 9).
A DECODING ALGORITHM
Algorithm 1 lists the pseudocode for our decoding procedure as illustrated in Section 4.
Algorithm 1 decodeEmbedding(M , e, a)
1: Input: an MPN M over X, an embedding e ∈ Rd and a map a : Me ⊆M→ {1, . . . , d} 2: Output: a sample x̃ ∼ X decoded from e, according to M 3: x̃← 0|X| 4: Qx root(M) . top-down traversal of M by using a queueQ 5: while not empty(Q) do 6: nx Q . process current node 7: if n ∈Mmax then . max node 8: cmax ← argmaxc∈ch(n) wncvc such that vc ← ea(c) if c ∈Me else maxu∼sc(c)Mc(u) 9: Qx cmax 10: else if n ∈M⊗ then . product node 11: ∀c ∈ ch(n) : Qx c 12: else . leaf node 13: if n ∈Me then 14: x̃sc(n) ← argminu∼(sc(n))D(φn(u)||ea(n)) 15: else . MPEAssignment (inner embedding) 16: x̃|sc(n) ← argmaxu∼sc(n)Mn(u) 17: return x̃
B DATASETS
The 10 datasets employed come from the freely accessible MULAN2, MEKA3, and LABIC4 repositories. They are real world standard benchmarks for MLC from text, image, sound and biological domains. Subsets of them have been also used in Dembczyński et al. (2012); Antonucci et al. (2013); Kong et al. (2013). They have been binarized as in (Di Mauro et al., 2016) by implementing the Label-Attribute Interdependence Maximization (LAIM) (Cano et al., 2016) discretization method5.
Table 2 reports the information about the adopted datasets, where N , M and L represent the number of attributes, instances, and possible labels respectively. They are divided into five standard folds. Furthermore, for each dataset D = {xi,yi}Mi=1 the following statistics are also reported: label cardinality: card(D) = 1M ∑M i=1 ∑L j=1 y i j , label density: dens(D) = card(D)L and distinct labels: dist(D) = |{y|∃(xi,y) ∈ D}|.
C LEARNING SPNS
To learn the structure and weights of our SPNs (and hence MPNs), we employ LearnSPN-b Vergari et al. (2015), a variant of LearnSPN. LearnSPN-b splits the data matrix slices always into two, while performing row clustering or checking for RVs independence. With the purpose of slowing down the greedy hierarchical clustering processes, it has proven to obtain simpler and deeper networks without limiting their expressiveness as density estimators. Based on the datasets statistics reported above in Appendix B, we define the same ranges for LearnSPN-b hyperparameters both when we learn our SPNs for the X and the Y. We set the G-test independence test threshold to 5, we limit the minimum number of instances in a slice to split to 10 and we performed a grid search for the best leaf distribution Laplace smoothing value in {0.1, 0.2, 0.5, 1.0, 2.0}. We perform all computations in the log space to avoid numerical issues.
For the SPN learned on the binarized version of MNIST in Section 3 we set the G-test independence test threshold to 20 and the instance threshold to 50 in order to reduce the network size. We then applied the same grid search as above for the leaf Laplace smoothing coefficient.
D SPN MODEL STATISTICS
Statistics for the reference SPN models learned with LearnSPN-b on the X RVs only are reported in Table 3. Their average (and standard deviations) values over the dataset folds provide information about the network topology and quality: how many nodes are in there (edges + 1), how are they divided into leaves and sum and products and their max depth (as the longest path from the root). The same statistics are reported for the SPNs over RVs Y, then turned in MPNs, in Table 4.
The length of the embeddings extracted from such models is the number of inner nodes from Table 3 for the inner embeddings over X. For the embeddings over RVs Y, their length in the full setting shall be considered as the number of all nodes from Table 4.
E MORE EXPERIMENT DETAILS AND RESULTS
E.1 TRAINING DETAILS
E.1.1 LEARNING LINEAR PREDICTORS
We learn to predict each target feature independently from the others, both when we employ the L2-regularized logistic regressor (LR) to predict RV Y directly and when we use a ridge regressor (RR) to predict the label embeddings.
To select the best value for the regularization parameter we will perform a grid search for LR in the space {10−4, 10−3, 10−2, 10−1, 1} and for RR in the space {10−4, 10−3, 10−2, 10−1, 1, 10, 102}6 for each experiment.
E.1.2 LEARNING RBMS
Concerning RBMs, we train them on the X alone (or on the Y alone for the kNN experiments) by using the Persistent Constrastive Divergence (PCD) Marlin et al. (2010) algorithm, leveraging the implementation available in scikit-learn. For the weight learning hyperparameters we run a grid search for the learning rate in {0.1, 0.01}, the batch size in {20, 100} and let the number of epochs range in {10, 20, 30} since no early stopping criterion was available. We then select the best models according to their pseudo-log likelihoods. To generate embeddings from RBMs, we evaluate the conditional probabilities of the hidden units given each sample. To make the comparison fairer we transform these values in the log domain in the same way we do for our SPN and MPN representations.
E.1.3 LEARNING MADES
For MADEs, following the experimentation reported in (Germain et al., 2015), we employ adadelta to schedule the learning rate during training and fix its decay rate at 0.95; we set the max number of worsening iterations on the validation set to 30 as for RBMs and we employed a batch size of 100 samples. We initialize the weights by employing an SVD-based init scheme.
6We leverage the python implementations for LR and RR from the scikit-learn package (http:// scikit-learn.org/). Note that in scikit-learn the grid parameter for LR has to be interpreted as an inverse regularization coefficient.
Other hyperparameters are optimized by a log-likelihood-wise grid search. The gradient dumping coefficient is searched in {10−5, 10−7, 10−9}, and we employ once the shuffling of mask and orders. Both ReLus and softplus functions are explored as the non-linearities employed for each hidden neuron. We employ a MADE openly available implementation, ported to python37.
We learn architectures of three hidden layers comprising 500 and 1000 (resp. 200 and 500) hidden neurons each for the X (resp. Y). For each reference model, we extract EX embeddings by evaluating all the hidden layer activations (d = 1500 and d = 3000); for the EY case, however, only the last hidden layer embeddings are actually exploited for the prediction (d = 200 and d = 500).
E.1.4 LEARNING MANIAC MODELS
Following the experiments in Wicker et al. (2016), we perform a grid search for the following hyperparameters: the number of layers is chosen in {2, 3, 4} and the compression factor β ∈ {0.7, 0.8, 0.9}. We employ the Java implementation freely available in MEKA. For the RF version of MANIAC we build a random forest comprising 100 trees as it has been used in Wicker et al. (2016) (see Appendix E.3.2 for the results of such a model).
We were not able to properly learn MANIAC for one dataset, Cal, for all measures, as a numerical error in MEKA prevented the model evaluation, thereby we removed it in the result Table.
We were also not able to train MANIAC on the EX → Y and hence EX → EY settings because the learned representations were not available through MEKA.
E.2 RECONSTRUCTION ERRORS
In this Section we provide the detailed results for the reconstructions of the input for our SPNs turned into MPNs for each train and test portion of each dataset, averaged by fold. Table 5 (resp. Table 6) reports the results for architectures trained on the X (resp. Y) and asked to reconstruct their inputs w.r.t these RVs.
E.3 OTHER RESULTS FOR MLC
E.3.1 JACCARD HAMMING AND EXACT MATCH MEASURES
In this Section we report the additional results for the JACCARD and HAMMING measures in Table 7 and Table 8 respectively. Figures 4 and 5 report the resilience of the decoding scheme for missing at random embedding components for the JACCARD and HAMMING measures, respectively. We employ a euclidean 5-nearest neighbor classifier to perform the decoding step on all our models. These results are reported in the table last rows.
7https://github.com/arranger1044/MADE.
E.3.2 MORE MANIAC RESULTS
In addition to the ridge regressor (RR) employed as the base model in our previous experiments, we also evaluate a much complex regressor as a random forest (RF) in conjunction with MANIAC, as
suggested in (Wicker et al., 2016). The rationale behind this is that a linear model, such as RR, could be at disadvantage on a compressed representation space, like those learned by MANIAC. Results for the JACCARD, HAMMING and EXACT MATCH scores are reported in Table 10 along with our previous results of MPN embeddings employing RR, for the X→ EY. The performance of a linear models on our embeddings is favorably comparable to that of a non-linear one on MANIAC embeddings, proving the efficacy of MPN as feature extractors.
E X →
E Y
5 -N N MADEhX=500,hY=200 24.21 44.51 0.00 18.21 15.92 40.30 22.09 23.44 63.06 9.14 4.7 MADEhX=500,hY=500 25.11 48.76 0.00 18.38 10.75 44.87 23.08 25.69 58.57 10.21 3.8 MADEhX=1000,hY=200 24.88 44.58 0.00 20.23 13.87 41.65 22.05 25.28 63.23 9.39 4.1 MADEhX=1000,hY=500 25.53 49.79 0.00 16.35 14.43 44.85 24.53 24.66 59.45 9.51 3.8 SPNinner → MPNfull 35.98 57.79 0.00 25.46 7.24 50.13 28.75 34.25 63.60 14.81 1.6 SPNinner → MPNinner 31.93 56.03 0.00 23.44 6.25 47.58 25.94 30.06 61.36 13.16 2.7
0.0 0.2 0.4 0.6 0.8 0.15 0.20 0.25 0.30 0.35
Arts
0.0 0.2 0.4 0.6 0.8 0.60 0.65 0.70 0.75 MPE eval
Business
0.0 0.2 0.4 0.6 0.8 0.20 0.21 0.21 0.22
Cal500
0.0 0.2 0.4 0.6 0.8 0.20 0.30 0.40 0.50 0.60
Emotions
0.0 0.2 0.4 0.6 0.8 0.52 0.54 0.56 0.58
Flags
0.0 0.2 0.4 0.6 0.8 0.30 0.40 0.50 0.60
Health
0.0 0.2 0.4 0.6 0.8 0.22
0.24
0.26 Human
0.0 0.2 0.4 0.6 0.8 0.26 0.27 0.28 0.29 0.30
Plants
0.0 0.2 0.4 0.6 0.8 0.20 0.30 0.40 0.50 0.60 0.70
Scene
0.0 0.2 0.4 0.6 0.8 0.35 0.40 0.45 0.50
Yeast
Figure 4: Average test JACCARD scores (y axis) obtained by imputing different percentages of missing random embedding components (x axis) for the X→ EY setting on all datasets by employing MPE inference (orange crosses) or the bottom-up evaluation imputation schemes (blue squares).
Arts
Business
Cal500
Yeast

Review:
The authors propose and evaluate using SPN's to generate embeddings of input and output variables, and using MPN to decode output embeddings to output variables. The advantage of predicting label embeddings is to decouple dependencies in the predicted space. The authors show experimentally that using SPN based embeddings is better than those produced by RBM's.
This paper is fairly dense and a bit hard to read. After the discussion, the main contributions of the authors are:
1. They propose the scheme of learning SPN's over Y and then using MPN's to decode the output, or just SPNs to embed X.
2. They propose how to decode MPN's with partial data.
3. They perform some analysis of when their scheme will lead to perfect encoding/decodings.
4. They run many, many experiments comparing various ways of using their proposed method to make predictions on multi-label classification datasets.
My main concerns with this paper are as follows:
- The point of this paper is about using generative models for representation learning. In their experiments, the main task is discriminative; e.g. predict multiple Y from X. The only discriminative baseline is a L2 regularized logistic regression, which does not have any structure on the output; it'd be nice to see how a discriminative structured prediction method would do, such as CRF or belief propagation. 
- The many experiments suggest that their encoder/decoder scheme is working better than the alternatives; can you please give more details on the relative computation complexity of each method?
- One thing I'm still having trouble understanding is *why* this method works better than MADE and the other alternatives. Is it learning a better model of the distribution of Y? Is it better at separating out correlations in the output into individual nodes?  Does it have larger representations? 
- I think the experiments are overkill and if anything, they way they are presented detract from the paper. There's already far too many numbers and graphs presented to be easy to understand.  If I have to dig through hundreds of numbers to figure out if your claim is correct, the paper is not clear enough. And, I said this before in my comments, please do not refer to Q1, Q2, etc. -- these shortcuts let you make the paper more dense with fewer words but at the cost of readability.
I *think* I convinced myself that your method works...I would love to see a table that shows, for each condition: (A) a baseline X->Y, (B) one *average* result across datasets for your method, and (C) one *average* result from a reasonable best competitor method. Please show for both the exact match and hamming losses, as that will demonstrate the gap between independent linear prediction and structured prediction. That would still be plenty of numbers but would make it much easier for the reader to verify your claims and you can put everything else in the Appendix.  E.g. something like:
Input | Predicted Output | Decoder | Hamming | Exact Match
----
X | P(Y) | CRF | xx.xx | xx.xx   (this is your baseline)
SPN E_X | P(Y) | n/a | xx.xx | xx.xx 
X | SPN E_Y | MPN | xx.xx | xx.xx  (given X, predict E_Y, then decode it with an MPN)
Does a presentation like that make sense? It's just really hard and time-consuming for me as a reviewer to verify your results, the way you've laid them out currently.