Consensus clustering
Consensus clustering is a method of aggregating (potentially conflicting) results from multiple clustering algorithms. Also called cluster ensembles or aggregation of clustering (or partitions), it refers to the situation in which a number of different (input) clusterings have been obtained for a particular dataset and it is desired to find a single (consensus) clustering which is a better fit in some sense than the existing clusterings. Consensus clustering is thus the problem of reconciling clustering information about the same data set coming from different sources or from different runs of the same algorithm. When cast as an optimization  problem, consensus clustering is known as median partition, and has been shown to be NP-complete, even when the number of input clusterings is three. Consensus clustering for unsupervised learning is analogous to ensemble learning in supervised learning. Justification for using consensus clustering. There are potential shortcomings for all existing clustering techniques. This may cause interpretation of results to become difficult, especially when there is no knowledge about the number of clusters. Clustering methods are also very sensitive to the initial clustering settings, which can cause non-significant data to be amplified in non-reiterative methods. An extremely important issue  in cluster analysis is the validation of the clustering results, that is, how to gain confidence about the significance of the clusters provided by the clustering technique (cluster numbers and cluster assignments). Lacking an external objective criterion (the equivalent of a known class label in supervised analysis), this validation becomes somewhat elusive. Iterative descent clustering methods, such as the SOM and k-means clustering circumvent some of the shortcomings of hierarchical clustering by providing for univocally defined clusters and cluster boundaries. Consensus clustering provides a method that represents the consensus across multiple runs of a clustering algorithm, to determine the  number of clusters in the data, and to assess the stability of the discovered clusters. The method can also be used to represent the consensus over multiple runs of a clustering algorithm with random restart (such as K-means, model-based Bayesian clustering, SOM, etc.), so as to account for its sensitivity to the initial conditions. It can provide data for a visualization tool to inspect cluster number, membership, and boundaries. However, they lack the intuitive and visual appeal of hierarchical clustering dendrograms, and the number of clusters must be chosen a priori. The Monti consensus clustering algorithm. The Monti  consensus clustering algorithm is one of the most popular consensus clustering algorithms and is used to determine the number of clusters, formula_1. Given a dataset of formula_2 total number of points to cluster, this algorithm works by resampling and clustering the data, for each formula_1 and a formula_4 consensus matrix is calculated, where each element represents the fraction of times two samples clustered together. A perfectly stable matrix would consist entirely of zeros and ones, representing all sample pairs always clustering together or not together over all resampling iterations. The relative stability of the consensus matrices can be used to  infer the optimal formula_1. More specifically, given a set of points to cluster, formula_6, let formula_7 be the list of formula_8 perturbed (resampled) datasets of the original dataset formula_9, and let formula_10 denote the formula_4 connectivity matrix resulting from applying a clustering algorithm to the dataset formula_12. The entries of formula_10 are defined as follows: formula_14 Let formula_15 be the formula_4 identicator matrix where the formula_17-th entry is equal to 1 if points formula_18 and formula_19 are in the same perturbed dataset formula_12, and 0 otherwise. The indicator matrix is used to keep track of which samples were selected  during each resampling iteration for the normalisation step. The consensus matrix formula_21 is defined as the normalised sum of all connectivity matrices of all the perturbed datasets and a different one is calculated for every formula_1. formula_23 That is the entry formula_17 in the consensus matrix is the number of times points formula_18 and formula_19 were clustered together divided by the total number of times they were selected together. The matrix is symmetric and each element is defined within the range formula_27. A consensus matrix is calculated for each formula_1 to be tested, and the stability of each matrix, that  is how far the matrix is towards a matrix of perfect stability (just zeros and ones) is used to determine the optimal formula_1. One way of quantifying the stability of the formula_1th consensus matrix is examining its CDF curve (see below). Over-interpretation potential of the Monti consensus clustering algorithm. Monti consensus clustering can be a powerful tool for identifying clusters, but it needs to be applied with caution as shown by Şenbabaoğlu "et al." It has been shown that the Monti consensus clustering algorithm is able to claim apparent stability of chance partitioning of null datasets drawn from a  unimodal distribution, and thus has the potential to lead to over-interpretation of cluster stability in a real study. If clusters are not well separated, consensus clustering could lead one to conclude apparent structure when there is none, or declare cluster stability when it is subtle. Identifying false positive clusters is a common problem throughout cluster research, and has been addressed by methods such as SigClust and the GAP-statistic. However, these methods rely on certain assumptions for the null model that may not always be appropriate. Şenbabaoğlu "et al" demonstrated the original delta K metric to decide formula_1 in  the Monti algorithm performed poorly, and proposed a new superior metric for measuring the stability of consensus matrices using their CDF curves. In the CDF curve of a consensus matrix, the lower left portion represents sample pairs rarely clustered together, the upper right portion represents those almost always clustered together, whereas the middle segment represent those with ambiguous assignments in different clustering runs. The proportion of ambiguous clustering (PAC) score measure quantifies this middle segment; and is defined as the fraction of sample pairs with consensus indices falling in the interval (u, u) ∈ [0, 1] where u is a  value close to 0 and u is a value close to 1 (for instance u=0.1 and u=0.9). A low value of PAC indicates a flat middle segment, and a low rate of discordant assignments across permuted clustering runs. One can therefore infer the optimal number of clusters by the formula_1 value having the lowest PAC. Hard ensemble clustering. This approach by "Strehl" and "Ghosh" introduces the problem of combining multiple partitionings of a set of objects into a single consolidated clustering without accessing the features or algorithms that determined these partitionings. They discuss three approaches towards solving this problem to  obtain high quality consensus functions. Their techniques have low computational costs and this makes it feasible to evaluate each of the techniques discussed below and arrive at the best solution by comparing the results against the objective function. Soft clustering ensembles. "Punera" and "Ghosh" extended the idea of hard clustering ensembles to the soft clustering scenario. Each instance in a soft ensemble is represented by a concatenation of "r" posterior membership probability distributions obtained from the constituent clustering algorithms. We can define a distance measure between two instances using the Kullback–Leibler (KL) divergence, which calculates the "distance" between two  probability distributions.