NMF Ensembles? Not for Text Summarization! (2020)

Abstract: Non-negative Matrix Factorization (NMF) has been used for text analytics with promising results. Instability of results arising due to stochastic variations during initialization makes a case for use of ensemble technology. However, our extensive empirical investigation indicates otherwise. In this paper, we establish that ensemble summary for single document using NMF is no better than the best base model summary.

## introduction non-negative matrix factorization (nmf) has demonstrated promise in text analytic tasks like topic modeling @xcite @xcite , document summarization @xcite and document clustering @xcite . the method finds favour due to the presence of non-negative elements in resultant factor matrices, which enhance intuitive understanding of the underlying latent semantic structure of the text @xcite . recent applications of ensemble methods for nmf based topic modeling has shown considerable promise @xcite @xcite . these observations drive our motivation for exploring nmf ensembles for document summarization task. ## nmf ensembles for text summarization ensemble methods are employed in supervised, semi-supervised and unsupervised learning settings. in all scenarios, they comprise two phases. in the first phase, diverse base models are generated. diversity in models is recognized to be the key factor for improvement @xcite , which is commonly sourced from variations in base algorithm, algorithmic parameters or data itself. in second phase, multiple base models are combined using a consensus (aka integration) function. wide variety of choices for creating diversity and combining base models gives rise to numerous possibilities for creating ensembles @xcite . ## performance evaluation in this section, we present extensive experimentation foot_2 carried out to investigate the performance of nmf ensembles for extractive summarization. first, we evaluate the performance based on combining methods described in sec. 2 and the sizes of ensemble. next, we study the effect of diversity in base models on ensemble performance. finally, we test the statistical significance of our results. all experiments are performed on duc2001 foot_3 dataset consisting of 308 documents and duc2002 1 data-set. we report macro averaged rouge recall scores (r-1: rouge-1, r-2: rouge-2, r-l: rouge-l) @xcite . in interest of brevity, all results are reported as performance gain (+) or loss (-) over nmf-tr scores as baseline method. ## discussion and conclusion extensive empirical investigation shows that leveraging stochastic variations due to random initialization of nmf factor matrices for extractive document summarization is not straight-forward. we experimented with different nmf solvers available in @xcite and found no change in results. in absence of any concrete explanation for degraded performance, we forward two plausible reasons. first, apparently simple combining methods fail to tease apart the differences in term-topic and topic-sentence strengths in the latent space. possible future investigation in this direction include projection of these matrices in higher dimension, and drawing from the cluster ensemble research to design more sophisticated combining methods. second reason is related to the maxim of sentence ranking and selection for extractive document summarization. combining scores from base models alters sentence ranking, and probably less important sentence get pulled up in the summary. this is most likely to happen with the lowest ranked sentence in the summary. a single bad sentence in the summary can lower down the score substantially. achieving stable ranks in ensemble technology could be another direction of research.