Fill in the BLANC : Human-free quality estimation of document summaries (2020)

Abstract: We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our approach achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document’s text. We present evidence that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements. And unlike ROUGE, the BLANC method does not require human-written reference summaries, allowing for fully human-free summary quality estimation.

## introduction two most widely used methods for measuring the quality of a summary are rouge @xcite and human evaluation @xcite . the rouge family of methods are well-defined and reproducible. however, these methods typically require a human-written reference summaries for comparison, completely disregarding the original document text. even if one assumes that a reference summary is available and of optimal quality, the rouge method is limited to measuring a mechanical overlap of text tokens with little regard to semantics. this deficiency may be partially addressable through measurement of the similarity not of text tokens but named entities or other preprocessed features @xcite @xcite @xcite or embeddings @xcite @xcite . in the latter work @xcite the references are not human-written but unsupervisedly constructed from selected salient sentences. an overlap can be measured as well between summary and document text @xcite . human evaluation of summary quality is far more meaningful and powerful than rouge, but it is far less reproducible. summary quality estimation is a cognitively demanding and highly subjective task. humans are also vulnerable to biases, such as the preference for phrases and sentences copied directly from the document text into summaries @xcite . improving human evaluation may require prompting labelers to pay higher attention @xcite , as well as splitting quality scores into multiple dimensions such as fluency, informativeness, and factual correctness @xcite . even if humans can be trained to be more reliable, reproducible estimators of summary quality, they will forever remain a slow, expensive, limiting resource. one possible route to a better automatic method for summary quality estimation is to train a model on document summaries annotated with human quality scores @xcite @xcite . such a model could be used to evaluate summaries without further human involvement. but even if such a model could achieve high agreement with human labelers, its performance would only be as objective and reproducible as the summary quality scores generated by one particular group of humans on a particular group of documents. such a model may not generalize beyond the domain and style of the training samples unless they are a massive, representative sample of all documents of interest. a more fundamental approach to the problem is to estimate how "helpful" a summary is for the task of understanding a text. for example this might be achieved through a series of question-answers @xcite @xcite . however, with this approach one must choose from a vast set of questions one might ask of a text, presupposing knowledge of the document itself and seriously limiting its reproducibility. in the following section we suggest a new approach that is fundamentally justifiable as an estimator of summary quality, as well as being conceptually simple and reproducible. ## basic validation of blanc measurement as part of the validation of these new measures we performed experiments to determine how a substitution of an obviously bad summary affects the measure. one example is a summary generated by selecting random words from the text. the random words summary is generated with the same length as the original summary. our original summaries are generated for randomly selected daily news by three different methods: by microsoft's abstractive uniml model @xcite , by semi-abstractive summarization model (based on @xcite ), and by extractive lexrank model (based on @xcite ). the summaries generated by these models are not flawless and vary widely in overall quality when evaluated by human labelers. in another validation experiment, we generate a "random sentences summary", which is constructed from the sentences of a document. for this example, we apply blanc-help with the "no-copy-pair" guard introduced above. but we use the second version of the guard rule, because it is less exclusive of text sentences overall, and we also compensate for the length of the summary by replacing the copysentence of the summary with another sentence, rather than simply removing the copy-sentence. blanc-help results for both examples (in comparison to the measure of the original summaries) are shown in figure 5 . we can see that the blanc value for the real generated summary is almost always higher than the value for the randomsentences summary. this confirms that the measure takes into account the context as well as the informativeness of the summary to assess the quality. selecting only summaries with exactly three sentences, we can observe how blanc-help deteriorates if we spoil some of the sentences of the summary. we replace one, two or all three sentences with random words, keeping the same length of the resulting randomized summary as the original summary. we also take care to run on each possible choice of replacement sentences twice, and average the resulting blanc-help. the result is shown up in figure 6 . ## comparison with human evaluation scores the blanc measures do not require any "goldlabeled" data: no human-written summaries nor human-annotated quality scores are needed. theoretically, the measures should reflect how fluent, informative, and factually correct a summary is, simply because only fluent, informative, correct summaries are helpful to the underlying language model. we now turn to the question of whether the blanc measures correlate with summary quality scores assigned by human readers. human scoring is fallible; a correlation with human scores should not be considered as a full validation of our measures, but rather as an independent confirmation that the measures are sensible. for purposes unrelated to this study, we have undertaken a series of human evaluations of many generated summaries of approximately similar length. as mentioned in the previous section, the summaries were generated by microsoft's abstractive uniml model @xcite , by semiabstractive model @xcite , and by extractive lexrank model @xcite . the summaries from the latter two sources were "equalized" in length to the uniml, so that at least on average the summaries from all three generation sources would be equal, and also so that most summaries would not differ significantly in length. altogether, we assembled 555 summary-text pairs for human scoring, with the texts taken from the cnn / daily mail dataset @xcite . we hired 10 annotators through odetta.ai and trained them to assess the overall quality of each summary on a 5-point scale: 0 = very bad, 1 = bad, 2 = ok, 3 = good or 4 = very good. the annotators worked independently from each other and had access to only one summary-text pair at a time. the task was performed through the online text annotation tool lighttag (lighttag.io). the values of the correlations are illustrated in figure 7 . the green step-function shows the value of correlation of an annotator score (with id ranging from 1 to 10) with the averaged score of the 9 other annotators. the number of samples used for the correlation is 555 -the summaries generated by the three models. the red and blue lines show correlations of blanc-help and rouge correspondingly with the averaged score of all 10 annotators. the rouge here is calculated using the google-research package (github.com/googleresearch/google-research/tree/master/rouge) as f1 value of "rougel" (lower blue line on the plot) and f1 value of "rougelsum" (upper blue line). the latter is the 'summary-level lcs', with summaries split to sentences and using a union longest common subsequence @xcite the yellow line in the figure shows how a simplest combination of blanc-help and rouge correlates with the annotators. the "blanchelp + rouge-lsum" is literally a simple sum of blanc-help and the rouge-lsum. as usual a blending of two different models produces better results, though it is not our purpose here to fit human scores, and we do not fit the weights in the sum. (for example, using a @xmath0. all shown correlations have p-values of order 10 -6 and lower. we observe that both blanchelp and rouge correlate with annotators as good as or better than about 30% of annotators. in figure 8 we present correlations with human scores on summaries generated for 100 typical daily news documents. the summaries were generated by the same three models; there were 300 summary-text pairs for scoring, again by 10 annotators. since there are no "gold-labeled" summaries for these news documents, there is no rouge score in the figure. as we see in all these examples, the humanhuman agreement is not impressive. we have observed from yet another evaluation dataset that if the texts and the generated summaries are challenging with very low inter-annotator agreement, the correlation of our measure with human scores is similarly diminished (with borderline p-values). the values assigned to humans scores (0,1,2,3,4) are not a perfect translation of the human perception of the corresponding labels ("very bad", "bad", "ok", "good", "very good"). from multiple evaluations unrelated to this study we know that when an evaluation is repeated, human annotators are far more likely to substitute "ok" and "good" with each other than other tags. when we obtain an averaged human score, a weighting of the values (0, 1, 2, 3, 4) with @xmath1.0, 3.0, 1.0, 1.0, 2.0) may be more inline with human perception, but the changes to results we presented here are not essential, of order 1%. bertscore @xcite , similarly to rouge, requires reference summaries, but is using overlaps of bert embeddings rather than strings. in figure 7 the bertscore f1 would be at 0.35 -close to blanc, bertscore precision at 0.16, and bertscore recall at impressive 0.47 (calculated using python package bert-score). a few simple observations may serve as an evidence that our measure deals with the length of a summary more reasonably than either humans or rouge. in simple correlation with a consensus score of annotators is not an easy criterion for judging the usefulness of the measure. when annotators are tasked with scoring several different qualities of a summary, their final score for the overall quality should be more grounded, because more attention has been spent on the summary and the text. in figure 9 we show values of correlations obtained from such evaluation. the data used here are the same as the data for figure 8 : summaries generated on randomly selected daily news documents. for this illustration, however, we split our 10 annotators into a small group of 3 and an "others" group of the remaining 7. there are 120 ways to chose the split (on the x-axis). the circle markers show human-human correlation, i.e. the correlation between the average score of the small group and the average score of the "others" group. the plus markers show blanc-human correlation, i.e. a correlation of the blanc with the "others" group of annotators. hence we see how well the blanc measure performs against the team of 3 annotators in correlating with the "others". for simplicity of the presentation, each type of correlation was sorted independently. if a correlation is unreliable (p-@xmath3.05) then the marker is not shown. we see that blanc can be competitive to a team of three human annotators on all summary qualities, especially on the 'overall' and fluency. ## conclusion in this paper we present blanc, a new family of objective and reproducible measures of summary quality. blanc does not require human-written reference summaries; it is based on how helpful the summary for understanding the document. by comparison, it is difficult to suspend disbelief when considering a method like rouge that does not inspect the document itself when estimating the quality of a summary. it is notable that rouge scores are often cited even for headline generation @xcite @xcite where it is hard to imagine that any single headline could be regarded as the best possible headline for a document. one may argue that rouge requires less processing power, unless we recall that applying it re-quires the processing power of a human who must write the reference summary for rouge. in future research we will consider variations of blanc and, for convenience, provide a public package blanc.