Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram cooccurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
It has been shown that the automatic evaluation using the ROUGE framework REF is highly correlated with human evaluation.
Automatic Evaluation Of Summaries Using N-Gram Co-Occurrence Statistics
Human Language Technology Conference And Meeting Of The North American Association For Computational Linguistics
Computer Science