Multimodal Event Transformer for Image-guided Story Ending Generation (2023)

Abstract: Image-guided story ending generation (IgSEG) is to generate a story ending based on given story plots and ending image. Existing methods focus on cross-modal feature fusion but overlook reasoning and mining implicit information from story plots and ending image. To tackle this drawback, we propose a multimodal event transformer, an event-based reasoning framework for IgSEG. Specifically, we construct visual and semantic event graphs from story plots and ending image, and leverage event-based reasoning to reason and mine implicit information in a single modality. Next, we connect visual and semantic event graphs and utilize cross-modal fusion to integrate different-modality features. In addition, we propose a multimodal injector to adaptive pass essential information to decoder. Besides, we present an incoherence detection to enhance the understanding context of a story plot and the robustness of graph modeling for our model. Experimental results show that our method achieves state-of-the-art performance for the image-guided story ending generation.

## introduction story ending generation @xcite aims to generate a reasonable ending for a given story plot. it requires deep models to integrate powerful language understanding capability, which is crucial for artificial intelligence. many efforts @xcite @xcite have been proposed and achieved promising results since neural models designed for comprehending natural language allow them to understand story plots and reason reasonable story endings. with the advance of automatic story generation, it has attracted outstanding attention in multimodality research @xcite @xcite . however, since story plots and story ending usually correspond to different content, the context with information bottleneck is not enough to deduce an informative story ending, i.e., generated endings tend to be inane and generic. to address this issue, @xcite propose an imageguided story ending generation (igseg) task that combines story plots and ending image to generate a coherent, specific and informative story ending. igseg demands not only introducing information from the ending image to story plots for story ending generation but also reasoning and mining implicit information from story plots and ending image, respectively. as shown in figure 1 , for story plots, "party" can be inferred from "big backyard barbeque" and "invited all friends", and "all friends", "all sat around" and "caught up with" can deduce "had a great time". for the ending image, "dim indoor" and "bright lights" can infer "very late in the night". existing methods @xcite focus on cross-modal feature fusion but overlook reasoning and mining implicit information from story plots and ending images. nonetheless, to effectively conduct cross-modal feature fusion, it is necessary to reason and mine more implicit information from single-modality data. an event is a fine-grained semantic unit, which refers to a text span composed of a predicate and its arguments @xcite . recently, event-centric reasoning displays excellent capability for context understanding and subsequent event prediction @xcite . in this work, we propose a multimodal event transformer (met) to mine implicit information to improve cross-modal fusion. for story plots, we leverage semantic role labeling (srl) parser @xcite to extract events from story plots and then construct them into a semantic event graph. for an ending image, we utilize scene graph parser @xcite to capture visual concepts and their relation to construct visual event graphs. since edges contain relationships between nodes in visual and semantic event graphs, we employ relational graph convolutional networks (rgcn) @xcite to encode event graphs to infer implicit information. for cross-modal feature fusion, most recent works @xcite adopt attention-based neural network models to implicitly integrate multi-modal features. however, due to the complexity of cross-modal features and the existence of dependency between single-modal features, it is often difficult for these models to complement cross-modal features. to tackle the issue, we propose cross-modal fusion to integrate differentmodality features. specifically, we merge visual and semantic event graphs and use rgcn to fuse cross-modal features for feature complement. moreover, since features from different modalities suffer from domain inconsistency, previous methods @xcite directly concatenate them and pass them to the decoder, which is not a crafted manner. to appropriately combine features from different modalities, we design a multimodal injector to integrate relevant features into the decoder. in addition, we propose an incoherence detection to enhance the context understanding for a story plot and the robustness of graph modeling for our model. in experiments, we conduct extensive evaluations on two datasets (i.e., vist-e @xcite and lsmdc-e @xcite ). experimental results show that our method outperforms strong competitors and achieves state-of-the-art performance. in addition, we conduct further analysis and human evaluation to demonstrate the effectiveness of our method. ## method this section will elaborate on our method for image-guided story ending generation, including event graph construction, event-based reasoning, cross-modal fusion, multimodal injector and story ending generation. the details of our method are shown in figure 2 . lastly, details about objectives and training are elaborated. ## conclusion in this work, we propose a multimodal event transformer, a framework for image-guided story ending generation. our method includes event graph construction, event-based reasoning, cross-model fusion, multimodal injector and story ending generation. different from previous work, our method not only focuses on cross-modal information fusion but also on reasoning and mining implicit information from single-modality data. in addition, we propose an incoherence detection to enhance the understanding context of a story plot and robustness of graph modeling for our model. in the experiments, results show that our method delivers state-of-the-art performance.