REFINESUMM : Self-Refining MLLM for Generating a Multimodal Summarization Dataset (2024)

Abstract: Multimodal Large Language Models (MLLMs) excel at synthesizing key information from diverse sources. However, generating accurate and faithful multimodal summaries is challenging, primarily due to the lack of appropriate multimodal datasets for fine-tuning that meaningfully integrate textual and visual modalities. To address this gap, we present a new dataset designed specifically for image-text multimodal summarization, harnessing the capabilities of state-of-the-art MLLMs. We generate summaries from Wikipedia sections and corresponding images and evaluate them across text-based, visual and multimodal dimensions, employing reference-free metrics. To refine the dataset, we: (1) Filter the MLLM-generated summaries by training a critic model on human annotations and using its predictions to remove low-quality summaries; (2) Fine-tune the MLLM with the filtered high-quality summaries; (3) Use the fine-tuned model in turn to regenerate the summaries. This self-refinement process significantly improves summary quality, as measured by human judgements and automatic multimodal metrics, resulting in a valuable dataset for multimodal summarization research. The dataset is publicly available at https://github.com/amazon-science/refinesumm.

## introduction in the age of information overload, efficiently extracting and summarizing key points from diverse sources is crucial. large language models (llms) have become powerful tools, generating humanlike text with impressive fluency. however, their limitations in faithfully capturing information become evident when faced with diverse, heterogeneous multimodal inputs like text and images. this poses a significant challenge for the task of multimodal summarization, which requires condensing rich information from multiple modalities into the italian wall lizard, also known as the ruin lizard, is a species of lizard native to various countries including bosnia and herzegovina, croatia, france, italy, serbia, montenegro, slovenia, and switzerland. the image shows insects..... the image features a green and brown lizard with long legs, sitting on a rock. this lizard is an italian wall lizard, which is native to various countries in europe and has been introduced to other regions. the lizard is a habitat generalist and has a diverse diet, allowing it to thrive in different environments. figure 1 : the summary generated from zero-shot manner (snowflake icon) has incorrect information (in red) like insects and is not able to ground that information with the text which talks about lizard whereas the summary generated by our self-refined model (fire icon) has image information (in green) that has been coherently combined with the information from the input article. multimodal large language models (mllms) utilize the strengths of llms to excel in handling multimodal data and offer a new research paradigm that can potentially address these challenges. @xcite highlight the challenges faced by multimodal summarization such as identifying key information from both modalities and using it to generate coherent and faithful summaries. generating content that faithfully reflects the original information is especially challenging when dealing with complex, heterogeneous multimodal inputs @xcite . to illustrate this point, figure 1 shows a summary generated by a multimodal llm that is not fully faithful to the input. high-quality datasets for multimodal summariza-tion that demonstrate faithful and informative summaries could guide the models towards improving fidelity of their generations. such datasets should comprehensively assess cross-modal capabilities, ensuring the coherent and faithful integration of information from both textual and visual sources in the summaries. however, such datasets are not available today; existing multimodal datasets @xcite are often constructed by taking the first sentence as the summary, leading to short summaries where the image plays a limited role. this underscores the need for improved datasets that truly challenge the crossmodal capabilities of such models while ensuring the meaningful integration of both modalities in the summarization process. to address this need, we present a new benchmark dataset for image-text multimodal summarization, featuring a key innovation: a self-refinement process for creating high-quality summaries. in this process, we first evaluate the summaries generated by an mllm along textual, visual, and multimodal dimensions and then filter them systematically with the help of a set of classifiers trained on data obtained from human annotations. these classifiers serve as a critic that predicts human judgments from automatic metrics. the high-quality summaries that pass the critic-based filter are then used to fine-tune the mllm. this fine-tuned, selfrefined mllm then generates the summaries that constitute our dataset. textual and visual faithfulness of the generated summaries, as measured by automatic evaluation metrics improves after refining the mllm generator. human evaluation results show that the quality of summaries generated by our approach along dimensions such as informativeness and correctness with respect to both text and image improves after refinement. our dataset creation method is cost-effective, using minimal human annotation for training the critic model to screen the summaries. in essence, we present a cost-effective approach for constructing a high-quality multimodal summarization dataset to benefit further research and development in this field. overall, our main contributions are as follows: ## experimental setup we now describe the evaluation metrics that we use for automatic reference-free evaluation of the generated summaries in ## results through our experiments, we aim to answer the following questions: ## conclusion we have introduced a self-refinement pipeline utilizing a multimodal llm, which we apply to create a new dataset for multimodal summarization of image-text pairs. within this pipeline, we train a model that serves as a critic that predicts human judgments from automated metrics. we use this multi-dimensional critic to identify high-quality summaries. our analysis shows that the proposed self-refinement approach yields improvements in the quality of generated summaries. this improvement is evident in improved faithfulness to both individual modalities (text and image) and better fusion of multimodal information, as demonstrated by human evaluation. furthermore, we show a case study based on examples of summaries before and after self-refinement which suggests that selfrefinement can lead to a better balance of modalities in the generated summaries. thus, this simple and cost-effective dataset generation procedure offers considerable potential for tasks where human article section image zero-shot summary self-refined summary the huisne is a 164.5 km (102.2 mi) long river in france. it is a left tributary of the river sarthe, which it meets in le mans. its source is near the town of pervenchères, in the orne department. the huisne flows through the following departments and towns: orne: saint-jouin-de-blavou, rémalard, condé-sur-huisne, le theil eure-et-loir: nogent-le-rotrou sarthe: la ferté-bernard, montfortle-gesnois, le mans. both village and church are said to have been founded by the archbishop of canterbury, st dunstan, in 960 ce, and there are legends surrounding his connection with the village. dunstan is supposed to have become an ironworker and run a small forge next to the church. ... the anglican church is dedicated to dunstan. in 1389 much of the village and most of the church were destroyed by fire; the latter was struck by lightning in the 17th century. the church was subsequently rebuilt in the fifteenth century. the church is in a mostly perpendicular style and has a squat, shingled broach spire ¨. inside the church there are a number of graves made of iron for the families of mayfield's ironmasters. there are three other religious buildings in mayfield: the roman catholic church of st thomas of canterbury (in station road), coggins mill church (a free church, also in station road), and mayfield baptist chapel (on south street).