Recently, there has been growing interest in using question-answering (QA) models to evaluate the content quality of summaries. While previous work has shown initial promising results in this direction, their experimentation has been limited , leading to a poor understanding of the utility of QA in evaluating summary content . In this work, we perform an extensive evaluation of a QA-based metric for summary content quality , calculating its performance with today's state-of-the-art models as well as estimating its potential upper-bound performance. We analyze a proposed metric, QAEval , which is more widely applicable than previous work. We show that QAEval already achieves state-of-the-art performance at scoring summarization systems, beating all other metrics including the gold-standard Pyramid Method, while its performance on individual summaries is at best competitive to other automatic metrics . Through a careful analysis of each component of QAEval, we identify the performance bottlenecks and estimate that with human-level performance , QAEval's summary-level results have the potential to approach that of the  Pyramid Method.
A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings . In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval . QAEval out-performs current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models . Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.
R
Recently, there has been growing interest in using question-answering (QA) models to evaluate
A desirable property of a reference-based evaluation metric that measures
R
summaries. While previous work has shown initial promising results in this direction, their experimentation has been limited , leading to a poor understanding of the utility of QA in evaluating summary content
a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics such as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings
R
perform an extensive evaluation of a QA-based metric for summary content quality , calculating its performance with today's state-of-the-art models as well as estimating its potential upper-bound performance. We analyze a
propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our
R
, which is more widely applicable than previous work. We show that QAEval already achieves
. QAEval out-performs current
R
performance at scoring summarization systems, beating all other metrics including the gold-standard Pyramid Method, while its performance on individual summaries is at best competitive to other automatic metrics
metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models
R
the
its
R
with human-level performance , QAEval's summary-level results have the potential to approach
its potential upper-bound performance surpasses all other automatic metrics, approaching
A
gold-standard