We introduce the Pitman Yor Diffusion Tree (PYDT) for hierarchical clustering, a generalization of the Dirichlet Diffusion Tree (Neal, 2001 ) which removes the restriction to binary branching structure. The generative process is described and shown to result in an exchangeable distribution over data points. We prove some theoretical properties of the model and then present two inference methods: a collapsed MCMC sampler which allows us to model uncertainty over tree structures, and a computationally efficient greedy Bayesian EM search algorithm. Both algorithms use message passing on the tree structure. The utility of the model and algorithms is demonstrated on synthetic and real world data, both continuous and binary. Tree structures play an important role in machine learning and statistics. Learning a tree structure over data points gives a straightforward picture of how objects of interest are related. Trees are easily interpreted and intuitive to understand. Sometimes we may know that there is a true underlying hierarchy: for example species in the tree of life or duplicates of genes in the human genome, known as paralogs. Typical mixture models, such as Dirichlet Process mixture models, have independent parameters for each component. We might expect for example that certain clusters are similar, for example are sub-groups of some large group. By learning this hierarchical similarity structure, the model can share statistical strength between components to make better estimates of parameters using less data. Classical hierarchical clustering algorithms employ a bottom up "agglomerative" approach (Duda et al., 2001 ) which hides the statistical assumptions being made. Heller and Ghahramani (2005) use a principled probabilistic model in lieu of a distance metric but simply view the hierarchy as a tree consistent mixture over partitions of the data. If instead a full generative model for both the tree structure and the data is used (Williams, 2000; Neal, 2003b; Teh et al., 2008; Blei et al., 2010) Bayesian inference machinery can be used to compute posterior distributions over the tree structures themselves. Such models can also be used to learn hierarchies over latent variables (Rai and Daumé III, 2008) . Both heuristic and generative probabilistic approaches to learning hierarchies have focused on learning binary trees. Although computationally convenient this restriction may be undesirable: where appropriate, arbitrary trees provide a more interpretable, clean summary of the data. Some recent work has aimed to address this. Blundell et al. (2010) extend Heller and Ghahramani (2005) by removing the restriction to binary trees. However, as for Heller and Ghahramani (2005) the lack of a generative process prohibits modeling uncertainty over tree structures. Williams (2000) allows nonbinary trees by having each node independently pick a parent in the layer above, but requires one to pre-specify the number of layers and number of nodes in each layer. Blei et al. (2010) use the nested Chinese restaurant process to define probability distributions over tree structures. Each data point is drawn from a mixture over the parameters on the path from the root to the data point, which is appropriate for mixed membership models but not standard clustering. An alternative to the PYDT to obtain unbounded trees is given by Adams et al. (2010) . They use a nested stick-breaking process to construct the tree, which is then endowed with a diffusion process. Data live at internal nodes of the tree, rather than at leaves as in the PYDT. We introduce the Pitman Yor Diffusion Tree (PYDT), a generalization of the Dirichlet Diffusion Tree (Neal, 2001) to trees with arbitrary branching structure. While allowing atoms in the divergence function of the
A generalization of the Dirichlet diffusion tree to arbitrary branching degrees is achieved by the Pitman-Yor diffusion tree REF .
Pitman-Yor Diffusion Trees
UAI
Mathematics Computer Science