ESTIME : Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings (2021)

Abstract: We propose a new reference-free summary quality evaluation measure, with emphasis on the faithfulness. The measure is based on finding and counting all probable potential inconsistencies of the summary with respect to the source document. The proposed ESTIME, Estimator of Summary-to-Text Inconsistency by Mismatched Embeddings, correlates with expert scores in summary-level SummEval dataset stronger than other common evaluation measures not only in Consistency but also in Fluency. We also introduce a method of generating subtle factual errors in human summaries. We show that ESTIME is more sensitive to subtle errors than other common evaluation measures.

## introduction summarization must preserve the factual consistency of the summary with the text. human annotation of factual consistency can be accompanied with detailed classification of factual errors, thus giving a hope that the annotation scores are reasonably objective @xcite @xcite @xcite . factual consistency of a summary is one of several summary qualities; for the purpose of human annotation these qualities can be specified in different ways @xcite @xcite @xcite . summarization models nowadays create satisfactorily fluent, coherent and informative summaries, but the factual consistency suffers from hallucinations, entity swaps and other errors. some factual errors are easily noticeable; other factual errors could be hardly noticeable even for annotators @xcite -which is arguably even worse. existing summary evaluation measures are based on several approaches, which may be sensitive to some qualities more than to others. a questionanswering based evaluation estimates how helpful is the summary in answering questions about the source text @xcite @xcite @xcite . a text reconstruction approach estimates how helpful is the summary in guessing parts of the source text @xcite . evaluation measures that use some kind of text similarity can estimate how similar is the summary to special human-written reference summaries @xcite @xcite , or, more realistically, how similar is the summary to the source text @xcite . in order to assess how well an evaluation measure works for factual consistency, it is necessary either to have a dataset of human-annotated imperfect machine-generated summaries @xcite , or to have a dataset of artificially introduced factual errors in originally factually correct human-written summaries @xcite . in this paper we focus on presenting a new evaluation measure with emphasis on factual consistency. our contribution: ## methods the motivations for our estimator: 1. any location in a summary has a context that loosely corresponds to a context in one or more locations in the text. 2. in the most similar context, the summary would normally use the same word that was used in the text. 3. summary generation models produce very few new (not from the text) words per summary. 4. transformer-made token embeddings are highly contextual @xcite . in order to estimate the consistency of a summary with the text, we attempt to count all the summary tokens that could be potentially related to a factual error. to this end, we check embeddings of all the tokens of the summary that have one or more occurrences in the text. for each embedding we find its match: the most similar embedding in the text. if the corresponding tokens are not the same, we add up such mismatch into our score of inconsistency. our goal is not an error correction or precise location of errors, but a score estimating the summary consistency quality. the algorithm is simple: in the summary, obtain embeddings only for the tokens that occur at least once in the text. to obtain an embedding of a token, mask the token, run a token-prediction transformer model on the context surrounding the token, and take the embedding of the token from a hidden layer. 2. for each of the obtained embeddings of the summary tokens, find the most similar embedding in the text. if the corresponding tokens do not coincide, count this as a potential inconsistency. the total number of such inconsistencies is our score, estime. we measure 'similarity' of embeddings by their scalar product. thus, estime score is the number n a of 'alarms': here h is heaviside function; the summary is a sequence of tokens t i , each having embedding e i ; the text t is a sequence of tokens t α , having embeddings e α . the summation in equation 1 is over all the summary tokens t i that exist in the text t . the count n a gets added +1 whenever the best match to e i from the embeddings of unequal tokens e β exceeds the best match from the embeddings of occurrences of the same token t α = t i in the text. the tokens are obtained by the tokenizer corresponding to the token-prediction transformer model. notice that we do not verify the summary tokens that do not occur in the text. such tokens still can influence the context used for embeddings of other tokens. the algorithm is asymmetric with respect to the summary and the text: it is supposed to estimate summary-to-text inconsistency. this approach is different from matching embeddings for sake of measuring similarity (e.g. similarity between a summary and a reference summary in bertscore @xcite ), and from using a model trained to replace wrong tokens with correct ones @xcite . the embeddings are taken using the pretrained bert model @xcite bert-largeuncased-whole-word-masking of transformers library @xcite . while there is no crucial difference with other varieties of bert, albert and roberta, this model showed a better overall performance, and we used it for evaluations in the next sections. for the sake of faster processing, we do not run the model separately for each token, but at a single run obtain embeddings for all tokens separated by the distance of 8 tokens. this means that the context for each masked token is a little muddied by masking of a few other tokens, but the distance of 8 tokens is large enough for the effect to be negligible. the results of the next sections are obtained with input size of 450 tokens (close to max bert input length). finally, when input window does not touch the beginning or end of the text, we do not mask the tokens too close to the edge of the window: no masking within the margin of 50 tokens at the edges of the input window. the algorithm is simple, but for convenience we provide the code foot_1 . in the next sections we present results for the versions estime-12 and estime-24, corresponding to the embeddings from the middle (12th layer) and from the top (24th layer) of the large bert; as explained later we also consider estime-21. ## performance on human summaries with generated subtle errors machine-generated summaries, even by abstractive summarization models, generally follow the source text by frequently reproducing large spans from it. human summaries are more varied in describing the source text, and it is interesting how useful can be estime for evaluating them. fundamentally, we are asking how flexible are the embeddings in understanding the context. in order to answer this question, we made random selection of 2000 text-summary pairs from cnn/daily mail dataset @xcite . for each human-written summary we then added the same summary modified by generated factual errors. we thus made 4000 text-summary pairs. we assigned the 'golden' scores as 1 to each clean summary, and 0 to each summary with errors. our 'subtle errors' generation method is simple, heuristic-free and easily reproducible. in order to generate an error, we randomly select a wholeword token in the summary, mask and predict it by an lm model (we used bert-base-cased). we then select the top predicted candidate that is not equal to the real token, and substitute it for the real token. the resulting subtle errors are similar to real machine-generated mishaps and hallucinations, with the fluency preserved. the evaluation task is now more difficult: the summaries are human-written, and the errors are subtle. without labeling of the generated errors, we cannot be confident of always having real factual errors: large part of the generated errors are indeed truly factual errors, but the rest disturb coherence or fluency, or make synonyms. for purposes of a preliminary simple evaluation here, and to ensure high probability of having true errors, we generated 3 random errors in each '@xmath0. in ## conclusion we introduced estime: estimator of summary-totext inconsistency by mismatched embeddings, -a measure of summary quality with emphasis on measuring factual inconsistency between the summary and the text. the fact that this simple measure correlates with human-labeled consistency and fluency much better than more complex measures tells us about the current state of summary evaluation, and about the power of contextual embeddings. we also introduced a method for generating subtle errors; the method has a potential for creating consistent and realistic benchmark datasets for factual consistency. in the near future we intend to release such fully labeled flexible dataset.