MEEL : Multi-Modal Event Evolution Learning (2024)

Abstract: Multi-modal Event Reasoning (MMER) endeavors to endow machines with the ability to comprehend intricate event relations across diverse data modalities. MMER is fundamental and underlies a wide broad of applications. Despite extensive instruction fine-tuning, current multi-modal large language models still fall short in such ability. The disparity stems from that existing models are insufficient to capture underlying principles governing event evolution in various scenarios. In this paper, we introduce Multi-Modal Event Evolution Learning (MEEL) to enable the model to grasp the event evolution mechanism yielding advanced MMER ability. Specifically, we commence with the design of event diversification to gather seed events from a rich spectrum of scenarios. Subsequently, we employ ChatGPT to generate evolving graphs for these seed events. We propose an instruction encapsulation process that formulates the evolving graphs into instruction-tuning data, aligning the comprehension of event reasoning to humans. Finally, we observe that models trained in this way are still struggling to fully comprehend event evolution. In such a case, we propose the guiding discrimination strategy, in which models are trained to discriminate the improper evolution direction. We collect and curate a benchmark M-EV2 for MMER. Extensive experiments on M-EV2 validate the effectiveness of our approach, showcasing competitive performance in open-source multi-modal LLMs.

## introduction events are instances or occurrences that are the fundamental semantic units. events are not independent, and they are usually interconnected by the following relations: causality, temporality, and intention. multi-modal event reasoning (mmer) is to comprehend these events and their the evolution scenario of a hurricane event. the buildings and facilities were damaged. local authorities conduct damage assessments to determine the extent of the destruction. residents and businesses affected by the hurricane begin the process of filing insurance. government agencies assign work to restore infrastructure and services. a hurricane or severe weather event caused significant damage to the buildings. local authorities issue evacuation orders for coastal or low-lying areas. the moist air rises from the ocean surface, it releases heat into the storm system. relations in both visual and textual modalities, and finally pave a path to better understanding the true world. mmer is expected to serve as the underpinning for various multi-modal applications, including visual storytelling @xcite , visual event prediction @xcite , event-related vqa @xcite , mm knowledge graph construction @xcite , and video generation @xcite . such intricate tasks require an understanding of the event evolution mechanism across diverse scenarios. with the deepening of research on multi-modal instruction tuning, multi-modal large language models (mllm) have been able to handle various multi-modal tasks effectively @xcite @xcite @xcite . these models master some abilities of mm event reasoning implicitly during training in diversified sorts of tasks. among all the task categories, the perception tasks such as referring expression comprehension, referring expression generation, and grounded image captioning @xcite @xcite enable the model to comprehend the entities of the events in the image and text. the cognitive tasks, namely image caption and vqa @xcite , endow the model with the semantic understanding capability of events. however, the models trained by these tasks are unable to perceive event evolution because of the static nature of all modality inputs. existing visual instruction-tuning methods only consist of questions for few clips of the entire event scenario. as shown in figure 1 , current methods only model the queried events with the green event and ignore the rest of the scenario. they lack a vision of a broad spectrum of other events in the evolving context. such contextual absence impedes models from learning abundant evolution knowledge resulting in poor performances in mmer. to address this issue, we propose multi-modal event evolution learning (meel) for endowing the model to understand the event evolution to enhance the ability of mmer, leading to improved performances on downstream tasks. specifically, we first design the event scenario diversification to acquire various events from abundant scenarios. then, we employ chatgpt to generate the evolving graphs of these seed events. the aim is to use these graphs to train the model to understand the rich knowledge of the evolution of events. to accomplish this goal, we propose the instruction encapsulation process to adapt the evolving graphs into instruction-tuning data to train the model. in this way, the training allows the model to comprehend more event evolutional knowledge of the scenario leading to better performance of mmer. however, allowing the model to learn only the evolving graphs is insufficient. without acknowledging the incorrect evolving events, the model would improperly forward the process, resulting in the hallucination of event reasoning. to mitigate, we perform the guiding discrimination. the model requires judging the incorrect evolution. we design various negative mining strategies to harvest incorrect events. then, we train the model to classify the right event. we also adapt the guiding discrimination into instruction tuning. after obtaining all the data, we finetune the llava @xcite model after its stage-1 pre-taining with lora @xcite to get our model. to validate the effectiveness of meel, we curate a benchmark m-ev 2 for multi-modal evaluation of event reasoning. m-ev 2 is collected or curated from nine existing datasets covering visual storytelling @xcite , visual event prediction @xcite , and event-related vqa @xcite . overall, m-ev 2 is a challenging task demanding the model to be capable of reasoning for diverse inter-event relations, like causality, temporality, and intent. it consists of two reasoning paradigms: close and open reasoning. we conducted extensive experiments on m-ev 2 and compare meel against some strong mllm baselines. our results demonstrate that meel does enhance the mmer ability of the model yielding significant improvements in downstream tasks. we conclude our contributions as: learning (meel). it aims to train the model to comprehend the intricate event evolution of diversified scenarios. our method may shed light on other mm event reasoning research. ## multi-modal event evolution learning we strive to enhance a multi-modal large language model's capability in multi-modal event reasoning (mmer) to boost performance on downstream tasks. the key is to enable the model to comprehend event evolution. as shown in figure 1 , current multi-modal sft data only model the target events with the green event and ignore the rest of the scenario. they lack a vision of a broad spectrum of other events in the evolving context. such contextual absence impedes models from learning abundant evolution knowledge resulting in poor performances in mmer. the intuitive motivation is to endow the model with the knowledge of all events in the whole scenario. to do that, we propose multi-modal event evolution learning (meel). we leverage chatgpt to obtain the evolution graph via our event graph evolution mechanism, which starts from diversified seed events. the evolving graphs contain the entire event semantics of a whole scenario. then we transform the evolving graphs into instruction-tuning data to train our model. note that instruction tuning is one of the feasible ways to learn the knowledge of event-evolving graphs. one can also leverage other methods such as in-context learning based on our data. we find that only when trained on the instruction-tuning data does the model turn to generate hallucinations. therefore, we further add guiding discrimination loss to require the model to distinguish the correct events. this section is organized as follows: section 2.1 details the mmer task. the main purpose of meel is to enhance the comprehension of event evolution. we initiate with an event diversification step to generate a diverse mix of seed events of various scenarios (section 2.2). then we construct the event-evolving graphs through a novel method named event graph evolution (section 2.3). our next objective is to leverage these event-evolving graphs for model instruction tuning training (section 2.4). finally, we incorporate a guiding discrimination training strategy to refine evolution pathways and reduce reasoning errors (section 2.5). meel's comprehensive framework is graphically represented in figure 2 . ## conclusion we propose the multi-modal event evolution learning for mmer. we design the event graph evolution process based on the diversified seed events. we then encapsulate the evolving graphs into instruction-tuning data. we introduce the guiding discrimination training paradigm to further improve the learning of evolution. we conduct experiments on our collected and curated m-ev 2 benchmark for mmer. results show the effectiveness of meel and it achieves competitive performance among open-source baselines.