  We propose a new reference-free summary quality evaluation measure, with
emphasis on the faithfulness. The measure is designed to find and count all
possible minute inconsistencies of the summary with respect to the source
document. The proposed ESTIME, Estimator of Summary-to-Text Inconsistency by
Mismatched Embeddings, correlates with expert scores in summary-level SummEval
dataset stronger than other common evaluation measures not only in Consistency
but also in Fluency. We also introduce a method of generating subtle factual
errors in human summaries. We show that ESTIME is more sensitive to subtle
errors than other common evaluation measures.
