We present an approach to model-based hi erarchical clustering by formulating an ob jective function based on a Bayesian anal ysis. This model organizes the data into a cluster hierarchy while specifying a complex feature-set partitioning that is a key compo nent of our model. Features can have either a unique distribution in every cluster or a com mon distribution over some (or even all) of the clusters. The cluster subsets over which these features have such a common distribu tion correspond to the nodes (clusters) of the tree representing the hierarchy. We apply this general model to the problem of docu ment clustering for which we use a multino mial likelihood function and Dirichlet priors. Our algorithm consists of a two-stage pro cess wherein we first perform a flat clustering followed by a modified hierarchical agglom erative merging process that includes deter mining the features that will have common distributions over the merged clusters. The regularization induced by using the marginal likelihood automatically determines the op timal model structure including number of clusters, the depth of the tree and the subset of features to be modeled as having a com mon distribution at each node. We present experimental results on both synthetic data and a real document collection. . In addition a model-based HAC algorithm based on a multinomial mixture model has been developed [9] . In the rest of the paper our refer ences to HAC will be to the version of HAC used in a likelihood setting as described above. In particular we will be concentrating on multinomial mixture models. Other hierarchical clustering algorithms in the litera ture include [8] , which describes a scheme to charac terize text collections hierarchically based on a deter ministic annealing algorithm. In this model, besides the latent variables used for clustering the documents at the base of the hierarchy, additional latent vari ables are used to define intermediate nodes. These additional latent variables (called abstraction nodes) model the conditional probabilities of the words. Reg ularization is achieved by maximizing the likelihood on a separate validation data set. As will be seen, our model differs considerably from the work described in [8] . First, only a part of the feature set is modeled at the intermediate nodes in the hierarchy (the choice of these features is a part of the model selection). This can dramatically reduce the number of adjustable pa rameters in the model, especially in the context of very high-dimensional data. Second, regularization is achieved by integrating over the parameter values in a Bayesian fashion. This allows the model to utilize the entire data-set in the model building (i.e. no held out validation set is required). Further, the two-stage algorithm described in this paper uses only the suffi cient statistics computed in the first stage during the merging process and therefore can be computationally less expensive. The probabilistic model developed in this paper orga-
Among these, REF proposed a method in the context of Bayesian analysis.
Model-Based Hierarchical Clustering
ArXiv
ArXiv
Mathematics Computer Science