Separating two probability distributions from a mixture model that is made up of the combinations of the two is essential to a wide range of applications. For example, in information retrieval (IR), there often exists a mixture distribution consisting of a relevance distribution that we need to estimate and an irrelevance distribution that we hope to get rid of. Recently, a distribution separation method (DSM) was proposed to approximate the relevance distribution, by separating a seed irrelevance distribution from the mixture distribution. It was successfully applied to an IR task, namely pseudo-relevance feedback (PRF), where the query expansion model is often a mixture term distribution. Although initially developed in the context of IR, DSM is indeed a general mathematical formulation for probability distribution separation. Thus, it is important to further generalize its basic analysis and to explore its connections to other related methods. In this article, we first extend DSM's theoretical analysis, which was originally based on the Pearson correlation coefficient, to entropy-related measures, including the KL-divergence (Kullback-Leibler divergence), the symmetrized KL-divergence and the JS-divergence (Jensen-Shannon divergence). Second, we investigate the distribution separation idea in a well-known method, namely the mixture model feedback (MMF) approach. We prove that MMF also complies with the linear combination assumption, and then, DSM's linear separation algorithm can largely simplify the EM algorithm in MMF. These theoretical analyses, as well as further empirical evaluation results demonstrate the advantages of our DSM approach. Entropy 2016, 18, 105 2 of 21 rather than the true relevance distribution that is supposed to be derivable from the truly relevant documents only. Recent research on negative relevance feedback has attempted to make use of irrelevant documents to improve the retrieval performance [3] [4] [5] . By assuming that a set of seed irrelevant documents is available, a distribution separation method (DSM) has been proposed in our earlier work [6] . Essentially, given a mixture distribution and a seed irrelevance distribution, DSM aims to derive an approximation of the true relevance distribution, in other words to separate the irrelevance distribution from the mixture one. It has been shown in [6] that, compared to the direct removal of irrelevant documents, separating the irrelevance distribution from the mixture distribution is theoretically more general and practically has led to a better performance. The formulation of DSM was based on two assumptions, namely the linear combination assumption and the minimum correlation assumption. The former assumes that the mixture term distribution is a linear combination of the relevance and irrelevance distributions, while the latter assumes that the relevance distribution should have a minimum correlation with the irrelevance distribution. DSM provided a lower bound analysis for the linear combination coefficient, based on which the desired relevance distribution can be estimated. It was also proven that the lower bound of the linear combination coefficient corresponds to the condition of the minimum Pearson correlation coefficient between DSM's output relevance distribution and the input seed irrelevance distribution. Although initially developed in the context of IR, DSM is indeed a general mathematical formulation for probability distribution separation. The separation algorithm and analysis of DSM are not restricted to query term distributions or any other distributions for IR tasks. It is thus important to further investigate its theoretical properties and make it become more general. In this article, we propose to generalize DSM's theoretical analysis, which was originally based on the Pearson correlation coefficient, to entropy-related measurements, specifically the KL-divergence and two variants. In addition, we investigate the distribution separation idea in a widely-used method, i.e., the mixture model feedback (MMF) approach [7] . Theoretical analysis has shown that the linear separation algorithm in DSM can be applied to simplify the EM-algorithm in MMF. The specific descriptions of the above contributions are as follows. First, we explore the effect of DSM on the KL-divergence between the DSM's estimated relevance distribution and the seed irrelevance distribution. In Section 3, we prove that DSM's lower bound analysis can also be extended to KL-divergence, and the minimum correlation coefficient corresponds to the maximum KL-divergence. We further prove that the decreasing correlation coefficient also leads to the maximum symmetrized KL-divergence, as well as the maximum JS-divergence between DSM's output distribution and the seed irrelevance distribution. These extended analyses enrich DSM's own theoretical properties. Second, we investigate the relationship between DSM and the mixture model feedback (MMF) approach [7] . In Section 4, we show that the linear combination assumption is valid in MMF, and the EM-based iterative algorithm of MMF is essentially a distribution separation process. Thus, its iterative steps can be largely simplified by the linear separation algorithm (see Equation (2)) developed in DSM. Furthermore, compared to MMF with an empirically-assigned combination coefficient, DSM's combination coefficient is analytically derived and is adaptive for each query. The experimental results in terms of the retrieval performance and running time costs have demonstrated the advantages of our DSM approach.
We also generalize DSM's theoretical analysis by providing additional KL-divergence analysis in REF .
Generalized Analysis of a Distribution Separation Method
Entropy
Entropy
Mathematics Computer Science