H a R i M ^+ : Evaluating Summary Quality with Hallucination Risk (2022)

Abstract: One of the challenges of developing a summarization model arises from the difficulty in measuring the factual inconsistency of the generated text. In this study, we reinterpret the decoder overconfidence-regularizing objective suggested in (Miao et al., 2021) as a hallucination risk measurement to better estimate the quality of generated summaries. We propose a reference-free metric, HaRiM+, which only requires an off-the-shelf summarization model to compute the hallucination risk based on token likelihoods. Deploying it requires no additional training of models or ad-hoc modules, which usually need alignment to human judgments. For summary-quality estimation, HaRiM+ records state-of-the-art correlation to human judgment on three summary-quality annotation sets: FRANK, QAGS, and SummEval. We hope that our work, which merits the use of summarization models, facilitates the progress of both automated evaluation and generation of summary.

## introduction although recent state-of-the-art summarization models have achieved remarkable performances @xcite @xcite , appropriate metrics for measuring faithfulness of the generated summaries are still needed. the practice of measuring performance in the summarization task heavily relies on the n-gram matching based metric, rouge @xcite . reportedly, rouge barely satisfies more than indicating lexical similarity @xcite and does not consider semantic dimensions of the generation, which current research needs of. there have been numerous attempts to come up with faithfulness evaluation metrics @xcite . neural-based metrics have demonstrated good performances in estimating the factual consistency of a summaryarticle pair with semantic entailment @xcite , questionanswering framework @xcite @xcite , and text generation @xcite . most of the model-asa-metric approach generally requires fine-tuning or complicated pipelines. consequently, evaluating generated texts with recent model-as-a-metric methods has become cumbersome. with the increased demand for faithful generation models, it has come to a lot of attention on reformulating training objectives for purported for this @xcite @xcite . we focus on the training objective suggested in @xcite , which directly targets hallucination problems in generating sentences given a source context. @xcite suggest that an overconfident decoder causes hallucination since the model excessively pays attention to the previously generated tokens over the source context which is in line with @xcite . in this paper, we reinterpret the decoder overconfidence regularization term from @xcite as hallucination risk and recompose the objective to be practical for summary quality evaluation in various aspects. unlike other recent metrics @xcite , our metric, harim + , detects hallucination in summary texts and evaluate their quality with the help of log-likelihood of summarization models. also, harim + does not require complicated pipelines, further training, or modification of the generation model in use. we conduct experiments to verify the effectiveness of our metric on several summary quality estimation benchmarks. we test harim + on frank, annotation sets from qags, and summeval, which provides multiple aspects of summary-quality judgements accompanied by summarization system outputs. through quantitative and qualitative experiments, we demonstrate the robust performance of our metric harim + , present the analysis of its inductive bias, and potential ex-tension. ## method we describe the logic behind margin-based tokenlevel objective @xcite , and reinterpret it as hallucination risk. we then propose modifications to re-formulate the original objective to be feasible for evaluating text quality. ## results in the followings, we report (1) metric to human judgement correlation in kendall's τ rank coefficient, and (2) qualitative examples that reveals inductive bias of the hallucination risk (harim + ) we proposed. comparisons with reported values of several other works are attached to appendix (table b .1). ## conclusion in this study, we propose harim + as a new summarization metric, which exploits the power of the summarization model for evaluation accompanied with the hallucination risk into consideration. for 1 ) model weight change (2) applying harim + . all results are averaged over language pairs from data supported by each model (i.e. bart-large+cnn+para averages the results of only 'to english' language pairs). note that τ we use here is wmt-variant suggested in @xcite . for fair comparison, in (1), only 'to english' pairs are used. for mbart @xcite we used mbart50-many-to-many model, for prism @xcite , we used m39v1 model.