Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues (2022)

Abstract: It is a common practice for recent works in vision language cross-modal reasoning to adopt a binary or multi-choice classification formulation taking as input a set of source image(s) and textual query. In this work, we take a sober look at such an “unconditional” formulation in the sense that no prior knowledge is specified with respect to the source image(s). Inspired by the designs of both visual commonsense reasoning and natural language inference tasks, we propose a new task termed “Premise-based Multi-modal Reasoning” (PMR) where a textual premise is the background presumption on each source image. The PMR dataset contains 15,360 manually annotated samples which are created by a multi-phase crowd-sourcing process. With selected high-quality movie screenshots and human-curated premise templates from 6 pre-defined categories, we ask crowd-source workers to write one true hypothesis and three distractors (4 choices) given the premise and image through a cross-check procedure.

## introduction cross-modal reasoning between image and text has been recognized as a fundamental and longstanding task in both academia and industry, which has recently attracted intensive attention from both natural language processing and computer vision communities @xcite @xcite . researchers try to teach machines to perceive, understand and reason with both visual and textual clues, which mimics human cognitive process @xcite . the canonical form for the cross-modal reasoning tasks usually take the source image(s) as the input and request the inference model to perform a multi-choice classification according to the specified textual query, including visual question answering @xcite , visual commonsense reasoning @xcite , visual entailment @xcite and image-text grounding @xcite , etc. we argue the default cross-modal setting is "static" or "unconditional" in the sense that no prior presumption or belief is attached to the input images @xcite . the lack of prior presumption would be insignificant while answering factoid questions according to the images, e.g. "how many dogs are lying on the grass?" @xcite or "is the bowl to the right of the green apple?" @xcite . however it may not be the case for more sophisticated cross-modal reasoning that involves human-like cognition and commonsense, e.g. "what may happen to @xcite ?", "what will @xcite say to @xcite do to @xcite ". the answers to these questions are ambiguous by merely looking at the source images, instead we offer an extra textual premise, which serves as the prior belief to the source images, such as "to @xcite , nothing could be more somber", " @xcite is the son of [personb]". we name the proposed task as "premise-based multi-modal reasoning" (pmr) 1 . in pmr, the inference model should be able to reason with both textual (from the premise) and visual (from images) clues. the motivations for pmr are two-folds. 1) from the social psychology perspective, the design of pmr is inspired by the abc theory @xcite , which represents a widely accepted framework for how one's feelings and behavioral patterns are connected. as illustrated in figure 1 , the theory ## dataset and task overview we describe the proposed pmr task with an example in figure 2 i&ii. given a source image and a textual premise, the inference model should perceive and understand the image in combination with the premise so as to choose the exclusive correct action among the four hypothetical candidates. the premise would serve as the background knowledge or domain-specific commonsense for the given image. in the running example, the model should be able to recognize what [person2] wears from the image and infer whether [person4] would give his seat to [person2] under the premise "[per-son4] is very friendly". the corrected answer is 'c' according to the visual and textual clues. in total, we collect about 15k instances for pmr. we list the statistics for pmr in ## data collection we collect pmr dataset in a multi-step crowdsourcing pipeline, including 1) image and premise creation, 2) annotator recruitment and instruction, 3) cross-checking annotation. ## experiments to set up the benchmark, we introduce multiple text-only and pretrained multi-modal baseline models as well as human performance on pmr. ## related work multimodal commonsense reasoning a series of tasks and datasets have been proposed for crossmodal commonsense reasoning. vcr @xcite is the most related work to the proposed pmr. it requires machines to understand a image and answer a multi-choice question. specifically, questions are like 'what is going to happen next', 'infer the relationship between @xcite smiling', as well as the rationale why the answer is true. in the video understanding regime, vlep @xcite ) is a dataset for future event prediction from videos. given a video with aligned dialogue, and two possible future events, the ai system is required to choose the more likely event from two provided options. in the world of cross-modal com- premises: 1. [person3] is indifferent. 2. [person3] is helpful. actions: a t : [person2] raises his hand , which is hurt badly , but [person3] just looks at it coldly and says nothing. d1: [person2] raises his left hand , which is hurt badly , but [person3] just looks at it coldly and says nothing. af: when [person2] raises his hand , [person3] sees that it is hurt and offers to bandage it. d2: when [person2] raises his right hand , [person3] sees that it is hurt and offers to bandage it. premises: 1. the situation is critical, but people are lucky. 2.this is telling a nightmarish story. actions: a t : the water fills the room , but [person2] who has black hair will be saved by a passing boat. d1: the water fills the room , but [person2] who has golden hair will be saved by a passing boat. af: the water fills the room and [person2] who has black hair will drown. d2: the water fills the room and [person2] who has golden hair will drown. premises: [person1] just got a new spouse. actions: a t : [person1] who is laughing happily will announce his engagement. d1: [person1] who is sobbing bitterly will announce his engagement. af: [person1] in a suit is sharing his happiness of being single with others. d2: [person1] in a t-shirt is sharing his happiness of being single with others. premise: [person1] is thought to be a/an outgoing man. actions: a t : [person1] will chat with the strange girl on his left and ask her to have a drink. d1: [person1] will chat with the strange girl in front of him and ask her to have a drink. af: [person1] will keep silent at the wine party. d2: [person1] will keep silent at the dock. monsense graph, visualcomet @xcite is a repository of visual commonsense graphs that consists of 1.4 million textual descriptions of visual commonsense inferences. the proposed pmr is different from the above-mentioned dataset as we request the models to perceive and understand the source image with the supervision from a specified textual premise, and reason conditionally with joint visual and textual clues. natural language inference early methods for textual nli mainly relied on conventional, featurebased methods trained from small-scale datasets @xcite . the release of larger datasets, such as snli @xcite , multinli @xcite , made neural network methods feasible. in the field of cross-modal entailment, the snli-ve @xcite dataset casts the source image as the visual premise, and asks the inference model judge whether the specified textual hypothesis entails or contradicts with the visual premise. the proposed pmr can also be viewed as a cross-modal entailment task, however difference from snli-ve, both premise and hypothetical actions are textual, and the corresponding source images can be viewed as the supplementary evidence while judging the entailment labels. besides, the existing textual and visual-related nli tasks are formulated in a three-way classification, while the proposed pmr is model as a 4-choice classification task. ## conclusion we propose a premise-based cross-modal reasoning (pmr) task, in which the textual premise serves as the presumptions to the source images. with selected images and human-curated premise templates, the collected dataset is formulated as a 4choice classification task, in which we ask human annotators to write one true hypothetical action and three distracting actions. the pmr task gains insights from both the visual commonsense reasoning and the natural language inference tasks. we hope the proposed dataset and task can pave the way for human-like cognitive reasoning from both visual and textual clues. substituting the objects mentioned in candidate actions with objects in o i . secondly, we calculated the relevance scores for p i with each remapped action by a bert-base model, which was trained with randomly sampled premise action pairs. then, we picked the top3 actions with the highest confidence as negatives. this method is simple but effective and efficient compared to the original adversarial matching. due to the remapping strategy, the objects in actions are going to be randomly substituted, which may cause the contradiction with the premise and thus avoid them being false negatives. to figure out the effectiveness of our methods, we performed a human test on the 50 items from the adversarial samples generated by it, and the result showed that the human got an accuracy of 0.86 which demonstrated that our methods can generate high-quality negatives. what's more, overemphasizing the difference with the true positive tends to lead the distractors to be more distinguishable and lower the difficulty of datasets, which was testified by the performance of vl-bert with an accuracy of 1.0 on the test set of adversarial samples generated in the same way of vcr. therefore, we maintain that the remapping strategy is enough to balance the difficulty with the correctness of adversarial samples.