Self-Supervised Multimodal Opinion Summarization (2021)

Abstract: Recently, opinion summarization, which is the generation of a summary from multiple reviews, has been conducted in a self-supervised manner by considering a sampled review as a pseudo summary. However, non-text data such as image and metadata related to reviews have been considered less often. To use the abundant information contained in non-text data, we propose a self-supervised multimodal opinion summarization framework called MultimodalSum. Our framework obtains a representation of each modality using a separate encoder for each modality, and the text decoder generates a summary. To resolve the inherent heterogeneity of multimodal data, we propose a multimodal training pipeline. We first pretrain the text encoder–decoder based solely on text modality data. Subsequently, we pretrain the non-text modality encoders by considering the pretrained text decoder as a pivot for the homogeneous representation of multimodal data. Finally, to fuse multimodal representations, we train the entire framework in an end-to-end manner. We demonstrate the superiority of MultimodalSum by conducting experiments on Yelp and Amazon datasets.

## introduction opinion summarization is the task of automatically generating summaries from multiple documents containing users' thoughts on businesses or products. this summarization of users' opinions can provide information that helps other users with their decision-making on consumption. unlike conventional single-document or multiple-document summarization, where we can obtain the prevalent annotated summaries @xcite @xcite @xcite , opinion summarization is challenging; it is difficult to find summarized opinions of users. accordingly, studies used an unsupervised approach for opinion summarization @xcite @xcite @xcite . recent studies @xcite ; amplayo and lapata, 2020; @xcite used a self-supervised learning framework that creates a synthetic pair of source reviews and a pseudo summary by sampling a review text from a training corpus and considering it as a pseudo summary, as in figure 1a . users' opinions are based on their perception of a specific entity and perceptions originate from various characteristics of the entity; therefore, opinion summarization can use such characteristics. for instance, yelp provides users food or menu images and various metadata about restaurants, as in figure 1b . this non-text information influences the review text generation process of users @xcite . therefore, using this additional information can help in opinion summarization, especially under unsupervised settings @xcite . furthermore, the training process of generating a review text (a pseudo summary) based on the images and metadata for self-supervised learning is consistent with the ac-tual process of writing a review text by a user. this study proposes a self-supervised multimodal opinion summarization framework called multimodalsum by extending the existing selfsupervised opinion summarization framework, as shown in figure 1 . our framework receives source reviews, images, and a table on the specific business or product as input and generates a pseudo summary as output. note that images and the table are not aligned with an individual review in the framework, but they correspond to the specific entity. we adopt the encoder-decoder framework and build multiple encoders representing each input modality. however, a fundamental challenge lies in the heterogeneous data of various modalities @xcite . to address this challenge, we propose a multimodal training pipeline. the pipeline regards the text modality as a pivot modality. therefore, we pretrain the text modality encoder and decoder for a specific business or product via the self-supervised opinion summarization framework. subsequently, we pretrain modality encoders for images and a table to generate review texts belonging to the same business or product using the pretrained text decoder. when pretraining the non-text modality encoders, the pretrained text decoder is frozen so that the image and table modality encoders obtain homogeneous representations with the pretrained text encoder. finally, after pretraining input modalities, we train the entire model in an end-to-end manner to combine multimodal information. our contributions can be summarized as follows: the extractive approach selects the most meaningful texts from input opinion documents, and the abstractive approach generates summarized texts that are not shown in the input documents. most previ-ous works on unsupervised opinion summarization have focused on extractive approaches. clusteringbased approaches @xcite @xcite were used to cluster opinions regarding the same aspect and extract the text representing each cluster. graph-based approaches @xcite @xcite were used to construct a graph-where nodes were sentences, and edges were similarities between sentences-and extract the sentences based on their centrality. although some abstractive approaches were not based on neural networks @xcite @xcite , neural network-based approaches have been gaining attention recently. @xcite generated an abstractive summary from a denoising autoencoder-based model. more recent abstractive approaches have focused on self-supervised learning. bražinskas and titov (2020) randomly selected n review texts for each entity and constructed n synthetic pairs by sequentially regarding one review text as a pseudo summary and the others as source reviews. amplayo and lapata (2020) sampled a review text as a pseudo summary and generated various noisy versions of it as source reviews. @xcite selected review texts similar to the sampled pseudo summary as source reviews, based on tf-idf cosine similarity. we construct synthetic pairs based on bražinskas and titov (2020) and extend the self-supervised opinion summarization to a multimodal version. multimodal text summarization has been mainly studied in a supervised manner. text summaries were created by using other modality data as additional input @xcite , and some studies provided not only a text summary but also other modality information as output @xcite @xcite @xcite . furthermore, most studies summarized a single sentence or document. although @xcite summarized multiple documents, they used non-subjective documents. our study is the first unsupervised multimodal text summarization work that summarizes multiple subjective documents. ## problem formulation the goal of the self-supervised multimodal opinion summarization is to generate a pseudo sum-mary from multimodal data. following existing self-supervised opinion summarization studies, we consider a review text selected from an entire review corpus as a pseudo summary. we extend the formulation of @xcite to a multimodal version. let @xmath0, r 2 , ..., r n } denote the set of reviews about an entity (e.g., a business or product). each review, r j , consists of review text, d j , and review rating, s j , that represents the overall sentiment of the review text. we denote images uploaded by a user or provided by a company for the entity as @xmath1, i 2 , ..., i m } and a table containing abundant metadata about the entity as t . here, t consists of several fields, and each field contains its own name and value. we set j-th review text d j as the pseudo summary and let it be generated from r -j , i, and t , where r -@xmath2, ..., r j-1 , r j+1 , ..., r n } denotes source reviews. to help the model summarize what stands out overall in the review corpus, we calculate the loss for all n cases of selecting d j from r, and train the model using the average loss. during testing, we generate a summary from r, i, and t . ## model framework the proposed model framework, multimodalsum, is designed with an encoder-decoder structure, as in figure 1b . to address the heterogeneity of three input modalities, we configure each modality encoder to effectively process data in each modality. we set a text decoder to generate summary text by synthesizing encoded representations from the three modality encoders. details are described in the following subsections. ## model training pipeline to effectively train the model framework, we set a model training pipeline, which consists of three steps, as in figure 2 . the first step is text modality pretraining, in which a model learns unsupervised summarization capabilities using only text modality data. next, during the pretraining for other modalities, an encoder for each modality is trained using the text modality decoder learned in the previous step as a pivot. the main purpose of this step is that other modalities have representations whose distribution is similar to that of the text modality. in the last step, the entire model framework is trained using all the modality data. details of each step can be found in the next subsections. ## results we evaluated our model framework and model training pipeline. in particular, we evaluated the summarization quality compared to other baseline models in terms of automatic and human evaluation, and conducted ablation studies. ## conclusions we proposed the first self-supervised multimodal opinion summarization framework. our framework can reflect text, images, and metadata together as an extension of the existing self-supervised opinion summarization framework. to resolve the heterogeneity of multimodal data, we also proposed a multimodal training pipeline. we verified the effectiveness of our multimodal framework and training pipeline with various experiments on real review datasets. self-supervised multimodal opinion summarization can be used in various ways in the future, such as providing a multimodal summary or enabling a multimodal retrieval. by retrieving reviews related to a specific image or metadata, controlled opinion summarization will be possible.