ROUGE is the de facto criterion for summarization research. However, its two major drawbackslimit the research and application of automated summarization systems . First, ROUGE favors lexical similarity instead of semantic similarity , making it especially unfit for abstractive summarization . Second, ROUGE cannot function without a reference summary, which is expensive or impossible to obtain in many cases . Therefore , we introduce a new end-to-end metric system for summary quality assessment by leveraging the semantic similarities of words and/or sentences in deep learning. Models trained in our framework can evaluate a summary directly against the input document , without the need of a reference summary . The proposed approach exhibits very promising results on gold-standard datasets and suggests its great potential to future summarization research. The scores from our models have correlation coefficients up to 0.54 with human evaluations on machine generated summaries in TAC2010. Its performance is also very close to ROUGE metrics' .
Canonical automatic summary evaluation metrics, such as ROUGE, suffer from two drawbacks . First, semantic similarity and linguistic quality are not captured well . Second, a reference summary, which is expensive or impossible to obtain in many cases , is needed. Existing efforts to address the two drawbacks are done separately and have limitations. To holistically address them , we introduce an end-to-end approach for summary quality assessment by leveraging sentence or document embedding and introducing two negative sampling approaches to create training data for this supervised approach . The proposed approach exhibits promising results on several summarization datasets of various domains including news, legislative bills, scientific papers, and patents. When rating machine-generated summaries in TAC2010, our approach outperforms ROUGE in terms of linguistic quality, and achieves a correlation coefficient of up to 0.5702 with human evaluations in terms of modified pyramid scores. We hope our approach can facilitate summarization research or applications when reference summaries are infeasible or costly to obtain, or when linguistic quality is a focus .
R
ROUGE is the de facto criterion for summarization research. However, its two major drawbackslimit the research and application of automated summarization systems
Canonical automatic summary evaluation metrics, such as ROUGE, suffer from two drawbacks
R
ROUGE favors lexical similarity instead of semantic similarity , making it especially unfit for abstractive summarization
semantic similarity and linguistic quality are not captured well
D
ROUGE cannot function without
R
. Therefore
, is needed. Existing efforts to address the two drawbacks are done separately and have limitations. To holistically address them
R
a new
an
R
metric system
approach
R
the semantic similarities of words and/or sentences in deep learning. Models trained in our framework can evaluate a summary directly against the input document , without the need of a reference summary
sentence or document embedding and introducing two negative sampling approaches to create training data for this supervised approach
D
very
R
gold-standard datasets and suggests its great potential to future summarization research. The scores from our models have correlation coefficients up to 0.54
several summarization datasets of various domains including news, legislative bills, scientific papers, and patents. When rating machine-generated summaries in TAC2010, our approach outperforms ROUGE in terms of linguistic quality, and achieves a correlation coefficient of up to 0.5702
R
on machine generated summaries in TAC2010. Its performance is also very close to ROUGE metrics'
in terms of modified pyramid scores. We hope our approach can facilitate summarization research or applications when reference summaries are infeasible or costly to obtain, or when linguistic quality is a focus