Approximate unsupervised summary optimisation for selections of ROUGE (2016)

Abstract: Approximate summary optimisation for selections of ROUGE It is standard to measure automatic summariser performance using the ROUGE metric. Unfortunately, ROUGE is not appropriate for unsupervised summarisation approaches. On the other hand, we show that it is possible to optimise approximately for ROUGE-n by using a document-weighted ROUGE objective. Doing so results in state-of-the-art summariser performance for single and multiple document summaries for both English and French. This is despite a non-correlation of the documentweighted ROUGE metric with human judgments, unlike the original ROUGE metric. These findings suggest a theoretical approximation link between the two metrics.

## introduction a standard metric in automatic multi-and single-document summarisation is rouge. certain variants of the metric have been shown to correlate well with human judgments @xcite . in particular, rouge-2 is the rouge variant shown to correlate best with human judgments. while it is impossible to directly optimise for these variants in unsupervised summarisation systems, a state-of-the-art unsupervised summariser optimises for a sort of document-weighted coverage version of rouge-2 @xcite . however, optimisation of this coverage version does not seem to imply optimisation of the original rouge metric. moreover, we show in this paper that the document-weighted metric scores in themselves do not exhibit any correlation with human judgments as certain selections of rouge do. despite this fact, state-of-the-art rouge scores for systems optimising for the document-weighted metric are achieved by @xcite . we therefore take up study of this seemingly contradictory situation from an empirical point of view : we want to obtain more comprehensive empirical evidence for or against the correlation between document-weighted optimisation and corresponding rouge scores. in this paper, we consider the direct consequences of optimising for separate document-weighted coverage versions of variants of rouge that have been shown to correlate most with human judgments. our results follow that of @xcite , showing that optimising for the document-weighted coverage version of the metric generally yields improvements in the corresponding rouge variant. the results are surprising given the non-correlation with human judgments. we carry out our experiments over both french and english data, for both single-and multi-document summarisation. in doing so, we introduce a new single-document summarisation dataset for french, within the legal text domain, and provide the first results for this dataset with a state-of-the-art summariser. ## rouge let g be an n-gram and r and s be multiset representations of reference and system summaries, respectively. we define the intersection a ∩ b of two multisets a, b as a multiset containing all multiples of elements shared between a and b. the rouge-n score of a system summary s with respect to r is defined as follows, where sums are over n-gram types (and not tokens) @xcite in extractive summarisation, it is impossible to optimise for rouge-n using just the input document as in unsupervised summarisation. indeed, replacing the reference summary by the input document makes all sentences of the the same length equally beneficial to the summary : since replacing references by documents for unsupervised rouge optimisation is worthless when conducting actual summarization, we must look for alternatives. one such alternative is documentweighted coverage. ## data we carry out experiments over both english and french data for single-and multi-document summarisation. single-document summarisation datasets (echr). for single-document summarisation, our datasets consist of judgment-summary pairs scraped from the european court of human rights case-law website, hudoc, 3 the english judgments (echr_en) of which were first used by @xcite . the english test set consists of 138 pairs. we adopt the same summary budget length as in their work, namely 805 words. we introduce the french counterpart of this dataset, produced in precisely the same manner as for english. the french data (echr_fr) was divided into training, test, and development sets respectively comprising 1295, 154 and 146 judgment-summary pairs each. the average summary size of the training set is 840 words. as in @xcite we adopt this average for our system summary budget for our test set input. ## conclusions we cannot optimise for rouge in an unsupervised fashion. however, we have shown in this paper that it is possible to optimise approximately for rouge by using a document-weighted rouge objective. doing so results in state-of-the-art summariser performance. this is despite a non-correlation of the document-weighted rouge metric with human judgments, unlike the original rouge metric. these empirical results should further verified on complementary datasets. future study will attempt to make explicit the theoretical ties between the two metrics from which the empirical results shown here should stem.