  The goal of a summary is to concisely state the most important information in
a document. With this principle in mind, we introduce new reference-free
summary evaluation metrics that use a pretrained language model to estimate the
information shared between a document and its summary. These metrics are a
modern take on the Shannon Game, a method for summary quality scoring proposed
decades ago, where we replace human annotators with language models. We also
view these metrics as an extension of BLANC, a recently proposed approach to
summary quality measurement based on the performance of a language model with
and without the help of a summary. Using GPT-2, we empirically verify that the
introduced metrics correlate with human judgement based on coverage, overall
quality, and five summary dimensions.
