Making sense of a dataset in an automatic and unsupervised fashion is a challenging problem in statistics and AI. Classical approaches for density estimation are usually not flexible enough to deal with the uncertainty inherent to real-world data: they are often restricted to fixed latent interaction models and homogeneous likelihoods; they are sensitive to missing, corrupt and anomalous data; moreover, their expressiveness generally comes at the price of intractable inference. As a result, supervision from statisticians is usually needed to find the right model for the data. However, as domain experts do not necessarily have to be experts in statistics, we propose Automatic Bayesian Density Analysis (ABDA) to make density estimation accessible at large. ABDA automates the selection of adequate likelihood models from arbitrarily rich dictionaries while modeling their interactions via a deep latent structure adaptively learned from data as a sum-product network. ABDA casts uncertainty estimation at these local and global levels into a joint Bayesian inference problem, providing robust and yet tractable inference. Extensive empirical evidence shows that ABDA is a suitable tool for automatic exploratory analysis of heterogeneous tabular data, allowing for missing value estimation, statistical data type and likelihood discovery, anomaly detection and dependency structure mining, on top of providing accurate density estimation. arXiv:1807.09306v2 [stat.ML] 3 Oct 2018 criterion. Differently from the approaches mentioned above, here we address model selection for a fully unsupervised task, with the aim of assisting domain experts in exploratory data analysis, providing them with a probabilistic framework to perform efficient inference and gain useful insights from the data in an automatic way. A suitable unsupervised learning approach to find out what is in the data is density estimation (DE). To perform DE on data in a tabular format, if there is no statistician around, a practitioner would first try to heuristically infer the statistical types of the data features, e.g., real, positive and numerical and nominal data types [33] . Based on this, she would need to make assumptions about their distribution, i.e., selecting a parametric likelihood model suitable for each marginal, e.g., Gaussian for real, Gamma for positive, Poisson for numerical and Categorical for nominal data. Then, she could start investigating the global interactions among them, i.e., determining the statistical dependencies across features. In the process, she would also likely deal with missing values and reason whether the data may be corrupted or contain anomalies. Unfortunately, classical approaches for DE, even if ubiquitous in ML applications, are still far from delivering automatic tools suitable for performing all these steps on real-world data. First, general-purpose density estimators usually assume the statistical data types and the likelihood models to be known a priori and homogeneous across random variables (RVs) or mixture model components. Indeed, the standard approach is still to generally treat all continuous data as (mixtures of) Gaussian RVs and discrete data as categorical variables. Second, they either assume "shallow" dependency structures that might be too simplistic to capture real-world statistical dependencies or fix "deeper" latent structures which cannot be easily learned. As a result, they lack enough flexibility to deal with fine grain statistical dependencies, and to be robust to corrupt data and outliers, especially when a maximum likelihood learning approach is used. A Bayesian treatment of DE, on the other hand, often struggles to scale up to high-dimensional data and approximate inference routines are needed [15] . CrossCat [21] shows a clear example of this trade-off. Even though CrossCat models sub-populations in the data with contextspecific latent variables, this latent structure is limited to a two-layer hierarchy, and even so inference routines have to be approximated for efficiency. Moreover, it is still limited to fixed and homogeneous statistical data types-and therefore, likelihood models. The latent variable matrix factorization model (ISLV) introduced in [33] is the first attempt to overcome this limitation by modeling uncertainty over the statistical data types of the features. However, like CrossCat, it can only perform inference natively in the transductive case, i.e., to data available during training. While ISLV allows domain experts to infer the data type of a feature, this is still represented by a single ad-hoc likelihood function, assuming all samples to be generated by the same likelihood family. Recently, Mixed Sum-Product Networks (MSPNs) [23] have been proposed as deep models for heterogeneous data able to perform tractable inference also in the inductive scenario, i.e. on completely unseen data. Indeed, MSPNs can exploit context specific independences to learn latent variable hierarchies that are deeper than CrossCat. However, MSPNs assume piecewise-linear approximations as likelihood models for both continuous and discrete RVs, which are not as interpretable as parametric distributions and also require continuous RVs to undergo a delicate discretization process. As a result, MSPNs are highly prone to overfitting, and despite several regularization techniques are employed, learning them via maximum likelihood results into a lack of robustness. In this paper, we leverage the above models' advantages while addressing their shortcomings by proposing Automatic Bayesian Density Analysis (ABDA). Specifically, ABDA: i) captures statistical dependencies in the data at different granularity through a hierarchical co-clustering of the data (partitioning both features and samples) by a sum-product network (SPN), automatically learning its rich latent structure as MSPNs do; ii) however, differently from MSPNs, it automates the selection of an adequate likelihood model for each data partition from As SPNs provide the hierarchical latent backbone of ABDA, we will now briefly review them. Please refer to [25] for more details. Upper-case letters X denote RVs and lower-case letters their values, i.e., x ∼ X. Similarly, we denote sets of RVs as X, and their combined values as x. Representation. An SPN S over a random vector X = {X 1 , . . . , X D } is a probabilistic model defined via a directed acyclic graph. Each leaf node L represents a probability distribution function over some subset of X-also called its scope-and inner nodes represent either weighted sums (S) or products (P). The scope of an inner node N is recursively defined as the union of the scopes of its children nodes, here defined by ch(S). A sum node S encodes a mixture model S S (x) = N∈ch(S) ω S,N S N (x) over sub-SPNs rooted at its children ch(S). We require that the all children of a sum node share the same variable scope-this condition is referred to as completeness [27] . The weights of a sum S, drawn from the standard simplex (i.e., ω S,N ≥ 0, N∈ch(S) ω S,N = 1), are denoted as Ω S . A product node P defines the factorization S P (x) = N∈ch(P) S N (x), over its children distributions defined over disjoint scopes-this condition is referred to as decomposability [27] . The parameters of S are the set of sum weights Ω = {Ω S } S∈S and the set of all leaf distribution parameters {H L } L∈S . An example of an SPN structure is shown in Fig. 1b . Complete and decomposable SPNs are not only highly expressive deep models-successfully employed in several ML domains [26, 22, 28, 35] to capture highly complex dependencies in real-world data-but allow also evaluating exactly complete evidence, marginal and conditional probabilities in linear time w.r.t. their representation size [7] . Latent variable representation.
In addition, REF aims to also infer the parametric likelihood model of the data and is shown to be more robust in working with missing, corrupted or anomalous data.
Automatic Bayesian Density Analysis
ArXiv
ArXiv
Computer Science Mathematics