Evaluating and Improving Factuality in Multimodal Abstractive Summarization (2022)

Abstract: Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for vision-and-language summarization. We propose CLIPBERTSCORE, a simple weighted combination of CLIPScore and BERTScore to leverage the robustness and strong factuality detection performance between image-summary and document-summary, respectively. Next, due to the lack of meta-evaluation benchmarks to evaluate the quality of multimodal factuality metrics, we collect human judgments of factuality with respect to documents and images. We show that this simple combination of two metrics in the zero-shot setting achieves higher correlations than existing factuality metrics for document summarization, outperforms an existing multimodal summarization metric, and performs competitively with strong multimodal factuality metrics specifically fine-tuned for the task. Our thorough analysis demonstrates the robustness and high correlation of CLIPBERTSCORE and its components on four factuality metric-evaluation benchmarks. Finally, we demonstrate two practical downstream applications of our CLIPBERTSCORE metric: for selecting important images to focus on during training, and as a reward for reinforcement learning to improve factuality of multimodal summary generation w.r.t automatic and human evaluation.

## introduction multimodal abstractive summarization is the task of generating an abridged text that contains the most important information of the source inputs from various modalities. this challenging task builds upon the success of document summarization, where the input is only text documents. for document summarization, there has been tremendous progress in improving the quality of the summaries with the help of large pre-trained models @xcite @xcite . however, one crucial problem for such models is hallucination, where the model generates contents that are not present or entailed by the document @xcite . while there have been significant advancements in developing metrics that correlate highly with the human judgment of factuality @xcite @xcite , these metrics only measure factuality between the document and the summary. the lack of judgment between other modalities, such as vision, and the summary makes such metrics not suitable for multimodal settings. we demonstrate this with the example in figure 1 . the given summary captures less relevant information (cutting the nail) from the document, but it is still considered factual to the document. however, the image shows the main point of the document (finding the place where the nail separates from the quick), making the summary not factual with respect to the image. current factuality metrics do not account for the image and thus cannot correctly assess factuality for multimodal summaries. in this work, we introduce a metric that judges factuality of the summary with respect to each input modality. focusing on the vision-and-language summarization, we propose clipbertscore, a simple and robust automatic factuality evaluation metric for multimodal summaries that combines two successful metrics: clipscore @xcite , which shows strong performance in detecting hallucinations between image and text, and bertscore @xcite , which correlates well with the human judgment of factuality for document summarization @xcite . next, due to the lack of corpora containing ground-truth human factuality judgments to eval-uate multimodal factuality metrics via correlation with human evaluation, we propose a multimodal factuality meta-evaluation (mufame) benchmark by collecting human annotation for four summarization systems and the reference summary on wikihow, foot_1 a large collection of how-to articles containing rich images relevant to the document. we find that our simple clipbertscore metric, which combines two off-the-shelf metrics in the zero-shot setting, achieves higher correlation with human judgment over existing text-based factuality metrics, outperforms current multimodal summarization metric mmae @xcite , and performs competitively with a metric trained with a triplet loss specifically for this task. next, we perform a detailed analysis of clip-bertscore by evaluating the correlation of the metric and each of its modules on four additional factuality metric-evaluation benchmarks. we first propose the wikihow factuality (wikihowfact) task, derived from the visual goal-step inference task @xcite . this ranking experiment measures how well the metric can select the correct summary from four choices given the correct document and image. since hallucinations are also present for image-captioning task @xcite , we also evaluate the correlations on two captioning tasks focusing on hallucinations, foil @xcite and bison @xcite , and one document summarization factuality benchmark frank @xcite . across all these tasks, we demonstrate the robustness of clipbertscore and its components, as they achieve the highest correlations compared to strong baselines in each of its respective settings. lastly, we present two practical applications for improving the factuality of downstream multimodal summarization models using clip-bertscore: (1) selecting the most important images as visual guidance @xcite , and (2) using the metric as a reward for self-critical sequence training @xcite . our results indicate that both applications improve the factuality of the generated summaries across two multimodal summarization datasets, as measured by three factuality metrics and human evaluation. to summarize, our contributions are: ## clipbertscore clipbertscore consists of two parts that tackle the image-summary and document-summary factuality judgments, respectively. we show an illustration of the computation in figure 1 . image-summary. we use a variant of clip-score @xcite for evaluating the factuality between images and the summary. this metric is based on clip @xcite , a crossmodal model trained on 400m image and caption pairs with infonce (oord et al., 2018) contrastive loss. hessel et al. (2021) finds that using this offthe-shelf model as a metric achieves the highest human correlation on the foil @xcite benchmark that explores how well the metric can detect hallucinations present in the captions. thus, it serves as a fitting candidate for factuality evaluation between the image and the summary. we use clip-s, which calculates the cosine similarity between the image embedding v and the text embedding of the summary sentence t. to adapt to multimodal summarization, where we have multiple images and multi-sentence summaries, foot_2 we take the average of the scores of all image and sentence pairs. formally, given a list of image embeddings v and summary sentence embeddings t from clip's image and text encoder, respectively: ## metric meta-evaluations next, after defining the multimodal factuality metric clipbertscore, we want to evaluate the quality of this new metric by checking whether it correlates with human judgments, similar to what has been done for textual factuality metrics @xcite @xcite . as there is no human annotations of factuality for multimodal summarization, we first propose a multimodal factuality meta-evaluation (mufame) benchmark derived from wikihow to test the correlations of clip-bertscore with human judgments of factuality. ## additional factuality metric-evaluation benchmarks we evaluate clipbertscore and its components on additional factuality metric-evaluation benchmarks, focusing on how robust the metrics performs across a variety of tasks and domains. ## downstream applications finally, we present two useful downstream applications for improving factuality of multimodal summarization models: first by using the metric as a reference image selection to guide the model in attending important images, and second by using it as a reward for self-critical sequence training. for both applications, we train strong baseline models by adapting clip-bart @xcite for multimodal summarization. specifically, we extract visual features with clip and use a projection layer to transform the dimension of the visual representation to the correct dimension of bart @xcite . then, the projected features are concatenated with the text features from the original encoder as the joint input representation for bart. see appendix d for more details. ## conclusion in this work, we present clipbertscore, an automatic metric for evaluating factuality for multimodal abstractive summarization. through metaevaluation with mufame and additional factuality benchmarks, we show clipbertscore and its modules correlate well with the human judgment of factuality with respect to the document, image and combined. clipbertscore is robust across the different image and text domains and achieves competitive correlation in the zero-shot setting with more complex metrics. we hope this work provides a meta-evaluation for evaluating future multimodal factuality metrics with mufame, a strong baseline metric clipbertscore to compare against, and two methods to improve the factuality of multimodal abstractive summarization models.