Multi-modal Action Chain Abductive Reasoning (2023)

Abstract: Abductive Reasoning , has long been considered to be at the core ability of humans, which enables us to infer the most plausible explanation of incomplete known phenomena in daily life. However, such critical reasoning capability is rarely investigated for contemporary AI systems under such limited observations. To facilitate this research community, this paper sheds new light on Abductive Reasoning by studying a new vision-language task, M ulti-modal A ction chain abductive R easoning ( MAR ), together with a large-scale Abductive Reasoning dataset: Given an incomplete set of language described events, MAR aims to imagine the most plausible event by spatio-temporal grounding in past video and then infer the hypothesis of subsequent action chain that can best explain the language premise. To solve this task, we propose a strong baseline model that realizes MAR from two perspectives: (i) we first introduce the transformer, which learns to encode the observation to imagine the plausible event with explicitly interpretable event grounding in the video based on the commonsense knowledge recognition ability. (ii) To complete the assumption of a follow-up action chain, we design a novel symbolic module that can complete strict derivation of the progressive action chain layer by layer. We conducted extensive experiments on the proposed dataset, and the experimental study shows that the proposed model significantly outperforms existing video-language models in terms of effectiveness on our newly created MAR dataset.

## introduction abductive reasoning typically begins with an incomplete observation or several observations and then proceeds to the likeliest possible explanation for the set @xcite . given an event observation (o), humans can find some related information in the recollection and easily trace the complete process with strong reasoning ability as a hypothesis (h) to explain the observation. for instance, when we observe the o: "the man in a t-shirt chocked on food is vomiting into the toilet" and remember that he used to devour food in the kitchen, we could infer the complete event chain about that man as hypothesis h: the man devoured the food in the @xmath0. this ability enables us to perform better than machines in high-level reasoning and would be the most precious capacity for modern ai. therefore, it is important to enhance such abductive reasoning capacities of ai models, i.e., complete process as explanation. motivated by the aforementioned abductive reasoning scenario, we present a novel visionlanguage task, called multi-modal action chain abductive reasoning (mar), which is illustrated in figure 1 . specifically, given a set of languagedescribed observations, the mar task targets to precisely localize the target event in the past video (visual recollection simulation of human) about the language-described person and rigorously reason out the subsequent action chain (subsequent events inference), to explain the observation. different from the previous abductive reasoning task @xcite focusing on the unimodal and partial reasoning, our new task has the following characteristics: (i) mar needs to locate the target event from the complex video information to explain the textual observation; (ii) mar requires rigorous recovery of the complete action chain. these characteristics introduce two challenges to the mar task: (1) heterogeneous information alignment. to realize the event grounding, aligning the cross-modal information is necessary. however, unlike the highly concise language description, the videos in real scenes usually contain complex and redundant information, including multiple people with different appearances, actions, scenes, etc. only a small amount of information in videos aligns with the text-described observation. precisely extracting information from the complicated video information to align is difficult but necessary for the ai system. (2) action chain reasoning. rigorous action chain reasoning is an interlocking and progressive process. if one step of reasoning is wrong, the correctness of subsequent steps cannot be guaranteed. therefore, for action chain reasoning, it is highly required to correctly learn the logical relationship between actions and correctly select from multiple next-step actions in each step of reasoning. we contribute a carefully annotated large-scale dataset, to-mar, based on our collected data to facilitate the challenges solved for the mar task. it contains 14, 201 cross-modal examples based on the videos manually collected from the tv show and the existing dataset @xcite . to address the mar task challenges, these examples have targeted manual annotations: (i) commonsense knowledge annotation for assisted alignment. we provide the full annotation of commonsense knowledge for every textual observation related person in the large-scale videos, including the character's appearance, clothing, actions, sentiment, etc. (ii) rigorous annotation for action chains. expert annotators with strong logical ability are asked to annotate the language-described observations and the action chains, ensuring the accuracy and rigor of logical annotations. based on the constructed dataset, we propose an end-to-end neural-symbolic model via commonsense knowledge for multi-modal action chain abductive reasoning (novel). there are two key targeted designs: (i) knowledge-guided alignment. we adopt the multi-task learning paradigm to synchronize the recognition learning of commonsense knowledge. based on such knowledge recognition ability, our novel can minimize the interference of the inferred event and past video, thereby more easily learning to generate explicit event grounding conditioned on textual observations. (ii) graph-aware symbolic reasoning. motivated by the powerful reasoning ability of the symbolic network @xcite , we design the targeted symbolic reasoning module based on the traditional graph theory. specifically, we store the learned action association graph in the training process. during inference, we determine the intermediate action chain between the textual observation and the grounded video event with dijkstra's algorithm @xcite . our contributions are three-fold: multi-modal spatio-temporal grounding. our mar task is related to the multi-modal spatiotemporal grounding task, which aims to detect target visual information described by the sentence from the video. it is an important task in the visual understanding domain @xcite @xcite . for video grounding research direction, most researchers focus their research on temporal grounding task @xcite . however, spatio-temporal grounding and spatial grounding @xcite @xcite @xcite have received less attention. @xcite uses the graph neural network to model the spatio-temporal relationship between objects to align text descriptions for object localization. in addition, to evaluating the model performance, this paper proposes a complete large-scale dataset. @xcite designs an end-to-end multi-modal grounding model based on the transformer. it outperforms all previous models without pre-training. later, @xcite makes a targeted design to fit the pre-trained parameters and achieves a great improvement in accuracy. neural-symbolic reasoning. compared with the pure neural network @xcite @xcite @xcite , neural-symbolic models have stronger inference and perception capabilities. @xcite is an earlier paper exploring this direction. it stitches symbolic models behind the multi-modal neural network to reason on the information the network perceives. in this neat way, the model achieves excellent results. @xcite combines the laws of physics with deep learning to make models capable of fitting complex physical processes. in this way, the model can effectively predict the motion trends of objects in the physical world. similarly, @xcite uses physical laws such as collision to design a symbolic model to process the information perceived by the neural network, which can effectively predict the future motion of objects such as balls or sliders. @xcite applies neural network and symbolic model to highlevel and low-level visual relation detection, respectively, and achieves good performance through the cooperation of the two. abductive reasoning. there is a limited amount of existing research work on abductive reasoning ai systems @xcite . previous abductive reasoning tasks @xcite require ai systems to provide a unimodal and partial explanation, which may lack some key information. ## dataset description to advance research on abductive reasoning, we propose the multi-modal action chain abductive reasoning task (mar) and contribute a large-scale text-video dataset for the mar task (to-mar). to complete the mar task, the ai system needs to after that, the action chain @xmath1, where the n a is the number of actions in chain a. ## experiments we evaluate the effectiveness of the proposed novel on to-mar dataset, followed by a discussion of novel's property with controlled studies. ## conclusion in this paper, we propose a new task, multi-modal action chain abductive reasoning, to promote the development of the abductive field. this cross-modal task targets to reason out a more complete explanation (explanation event grounding and sequential action chain inference) than the previous abductive reasoning tasks. furthermore, we propose a largescale dataset (to-mar) and a neural-symbolic model via commonsense knowledge (novel) for our new task as a strong baseline. extensive experiments on the to-mar dataset and the to-mstg dataset demonstrate the effectiveness of our novel model.