Hierarchical clustering is a fundamental task often used to discover meaningful structures in data, such as phylogenetic trees, taxonomies of concepts, subtypes of cancer, and cascades of particle decays in particle physics. When multiple hierarchical clusterings of the data are possible, it is useful to represent uncertainty in the clustering through various probabilistic quantities, such as the distribution over tree structures and the marginal probabilities of subtrees. Existing approaches represent uncertainty for a range of models; however, they only provideapproximate inference. This paper presents dynamic-programming algorithms and proofs for exact inference in hierarchical clustering at small but useful scales. We are able to compute the partition function, MAP hierarchical clustering , and marginal probabilities of sub-hierarchies and clusters. Our method supports a wide range of hierarchical clustering models and only requires a cluster compatibility function. Rather than scaling with the number of hierarchical clusterings of N elements (( 2N-3)!! ), our approach runs in time and space proportional to the significantly smaller powerset of N. Despite still being large, there are many important applications at the practically-computable range of N < 20. We demonstrate the advantages of exact inference on synthetic data of interest to Dasgupta's cost as well as on two real world applications, in particle physics at the Large Hadron Collider at CERN and in cancer genomics  .
Hierarchical clustering is a fundamental task often used to discover meaningful structures in data, such as phylogenetic trees, taxonomies of concepts, subtypes of cancer, and cascades of particle decays in particle physics. Typically approximate algorithms are used for inference due to the combinatorial number of possible hierarchical clusterings. In contrast to existing methods, we present novel dynamic-programming algorithms for exact inference in hierarchical clustering based on a novel trellis data structure, and we prove that we can exactly compute the partition function, maximum likelihood hierarchy , and marginal probabilities of sub-hierarchies and clusters. Our algorithms scale in time and space proportional to the powerset of N elements which is super-exponentially more efficient than explicitly considering each of the ( 2N-3)!! possible hierarchies. Also, for larger datasets where our exact algorithms become infeasible, we introduce an approximate algorithm based on a sparse trellis that compares well to other benchmarks. Exact methods are relevant to data analyses in particle physics and for finding correlations among gene expression in cancer genomics , and we give examples in both areas, where our algorithms outperform greedy and beam search baselines. In addition, we consider Dasgupta's cost with synthetic data .
D
When multiple hierarchical clusterings of the data are possible, it is useful to represent uncertainty in the clustering through various probabilistic quantities, such as the distribution over tree structures and the marginal probabilities of subtrees. Existing approaches represent uncertainty for a range of models; however, they only provide
D
approximate
R
inference. This paper presents
Typically approximate algorithms are used for inference due to the combinatorial number of possible hierarchical clusterings. In contrast to existing methods, we present novel
D
and proofs
R
at small but useful scales. We are able to
based on a novel trellis data structure, and we prove that we can exactly
R
MAP hierarchical clustering
maximum likelihood hierarchy
R
method supports a wide range of hierarchical clustering models and only requires a cluster compatibility function. Rather than scaling with the number of hierarchical clusterings of N elements ((
algorithms scale in time and space proportional to the powerset of N elements which is super-exponentially more efficient than explicitly considering each of the (
R
), our approach runs in time and space proportional to the significantly smaller powerset of N. Despite still being large, there are many important applications at the practically-computable range of N < 20. We demonstrate the advantages of exact inference on synthetic data of interest to Dasgupta's cost as well as on two real world applications,
possible hierarchies. Also, for larger datasets where our exact algorithms become infeasible, we introduce an approximate algorithm based on a sparse trellis that compares well to other benchmarks. Exact methods are relevant to data analyses
R
at the Large Hadron Collider at CERN and
and for finding correlations among gene expression
A
, and we give examples in both areas, where our algorithms outperform greedy and beam search baselines. In addition, we consider Dasgupta's cost with synthetic data