ART : rule b A sed futu R e-inference deduc T ion (2023)

Abstract: Deductive reasoning is a crucial cognitive ability of humanity, allowing us to derive valid conclusions from premises and observations. However, existing works mainly focus on language-based premises and generally neglect deductive reasoning from visual observations. In this work, we introduce rule bAsed futuRe-inference deducTion (ART), which aims at deducing the correct future event based on the visual phenomenon (a video) and the rule-based premises, along with an explanation of the reasoning process. To advance this field, we construct a large-scale densely annotated dataset (Video-ART), where the premises, future event candidates, the reasoning process explanation, and auxiliary commonsense knowledge (e.g., actions and appearance) are annotated by annotators. Upon Video-ART, we develop a strong baseline named ARTNet. In essence, guided by commonsense knowledge, ARTNet learns to identify the target video character and perceives its visual clues related to the future event. Then, ARTNet rigorously applies the given premises to conduct reasoning from the identified information to future events, through a non-parametric rule reasoning network and a reasoning-path review module. Empirical studies validate the rationality of ARTNet in deductive reasoning upon visual observations and the effectiveness over existing works.

## introduction deductive reasoning is a systematic method that rigorously follows a set of explicitly given constraints (i.e., rules) to deduce valid conclusions from empirical facts through logical inferences @xcite . it represents a corner-first author. mengzeli@zju.edu.cn reasoning process for : + candidate future events: event 1: the lady with yellow hair is putting the book on the desk into her bag. event 2: the woman in green dress is putting down her bag. event 3: the lady who opened the door is walking to the lamp and turning on it. observed phenomena stone of human psychological functioning, serving as an indispensable aspect of our daily cognitive processes. for example, human beings possess the capability to utilize the given rule set (r) to deduce future events (f) through the interpretation of observed phenomena (o). to illustrate: despite deductive reasoning being acknowledged as a fundamental cognitive competency of humanity @xcite , there is a scarcity of investigations in designing ai systems that are capable of executing deductive reasoning in the multi-modal field. to advance the research, we simulate the deductive reasoning of human beings and propose a rule based future-inference deduction task (art). overall, in aligning with the established deductive reasoning studies within the nlp community @xcite , art should select the correct textual future events (correct conclusions) from potential candidates, rigorously based on the given language rule set and the observed phenomena. as depicted in figure 1 , the observations in art are illustrated by videos, which is grounded in the fact that visual information has a profound impact on the human brain, accounting for an impressive 83% of all inputs @xcite . upon the video observations, art reasons on a crucial ingredient of videos, i.e., human actions, and endeavors to derive the most accurate future event, constrained by the given rule set and the video observation. in addition to deducing future events, art should provide explanations of the reasoning process by generating the rule chain. in formal terms, our art task is close to the well-established videolanguage inference task @xcite , which assesses the accuracy of language descriptions in relation to input videos. analyzing the difference between the two tasks can facilitate the advancement of our new field. compared with the video-language inference, our art task has the following characteristics: (1) the task demands a transparent explanation of the reasoning process; (2) in accordance with deductive reasoning @xcite , the reasoning processes should be constrained by the rule sets (regardless of the perceived irrationality of provided rules) that are explicitly assigned to each individual sample, while relying on implicitly learned rule constraints that might come from other samples could cause mistakes. to promote multi-modal deductive reasoning research and meet the demands of the art task, we introduce a new dataset, named video-art, consisting of 23, 895 samples. careful annotation was performed by annotators and verifiers with strong logical reasoning skills, who mainly focused on two key aspects: (1) they targeted to design the rule sets and the candidate future events that are closely associated with the visual information presented; (2) the annotators provided the correct future events and a rule-based explanation of the reasoning process. in addition, to enhance the ai system's deduction from visual semantics in highly unstructured videos, which are composed of densely arranged pixels, we have carefully annotated the commonsense knowledge of the target objects, including their appearance and related actions. to lay the groundwork for future research, we propose a strong baseline for the rule based future-inference deduction, named artnet. art-net mainly consists of three components, i.e., knowledge-guided target perception (ktp), nonparametric rule reasoning network (rrn), and reasoning path review (rpr). ktp learns to identify the target character and corresponding visual clues related to the upcoming event through multi-task learning and commonsense knowledge annotations such as actions and appearance. inspired by traditional graph-theoretic algorithms, rrn performs layer-by-layer reasoning through a purpose-built non-parametric rule reasoning network, uncovering the reasoning paths from the identified visual clues to potential future events. rrn offers two advantages for the art task over traditional models: (1) rrn provides explanations of its rule-based reasoning process. (2) rrn avoids rote memorization of rules within the training data and ensures the rigorous application of the sample-specific rule set. furthermore, the rpr module validates the semantic consistency between the rule reasoning paths uncovered by rrn, the video observations, and the future event descriptions. overall, the main contributions of this work are three-fold: video-language inference. as the development of the deep learning @xcite @xcite , the visual and language related tasks attracts more and more attentions @xcite @xcite @xcite . the video-language inference task aims at judging the correctness of the textual conclusion, based on the video information and the language description @xcite @xcite @xcite . @xcite proposes a carefully labeled dataset for this task and introduces a strong baseline to further develop this field. based on this dataset, @xcite designs a new model based on the graph network and validate it. ## deductive reasoning dataset our rule based future-inference deduction task (art) requires the ai system to (1) select the correct future event from the candidate events by reasoning on the rule set and the observation (a ## method we propose a new task, rule based futureinference deduction (art), and design a targeted model named artnet. according to the task characteristics, we contribute the non-parametric rule reasoning module for artnet. in addition to the key reasoning module, the knowledge-guided perception module and the rechecking module are introduced to assist in the completion of the art task. task formulation. given an observation (a video) v, a rule set (multiple rules) @xmath0, and candidate future events @xmath1, the art task aims to reason out the correct future event and explain the reasoning process based on the rules. we define the model with the parameter θ for the art task as m. then, the training optimization function δ(.) of m is represented as: ## experiments we experiment with our artnet model on our proposed video-art dataset to verify the model's effectiveness for the rule based futureinference deduction task (art). all experimental environments are deployed in hikvision ( https://www.hikvision.com/en/ ). ## conclusion we study the deductive reasoning process in humans and propose a video-text deductive reasoning task, art, which is an early exploration of deductive reasoning in the field of multi-modal. to promote this new task development, we propose a new dataset, video-art, and a strong baseline called artnet. experiments prove the artnet effectiveness.