A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues (2023)

Abstract: Conditional inference on joint textual and visual clues is a multi-modal reasoning task that textual clues provide prior permutation or external knowledge, which are complementary with visual content and pivotal to deducing the correct option. Previous methods utilizing pretrained vision-language models (VLMs) have achieved impressive performances, yet they show a lack of multimodal context reasoning capability, especially for text-modal information. To address this issue, we propose a Multi-modal Context Reasoning approach, named ModCR. Compared to VLMs performing reasoning via cross modal semantic alignment, it regards the given textual abstract semantic and objective image information as the pre-context information and embeds them into the language model to perform context reasoning. Different from recent vision-aided language models used in natural language processing, ModCR incorporates the multi-view semantic alignment information between language and vision by introducing the learnable alignment prefix between image and text in the pretrained language model. This makes the language model well-suitable for such multi-modal reasoning scenario on joint textual and visual clues. We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved performance (exact gain by 4.8% on PMR test set) compared to previous strong baselines.

## introduction cross modal reasoning is a hot research topic both in natural language processing and computer vision communities. most cross modal reasoning tasks, such as visual question answering @xcite @xcite figure 1 : a case from the pmr @xcite data set, where the correct option is answer b. the bluecolor words represent the pivotal textual clue to infer the correctness of answers a and b. @xcite , visual dialog @xcite , visual entailment, @xcite and visual commonsense reasoning @xcite @xcite , concentrate on the visual reasoning scenario that relies primarily on image information. the given text (or question) is highly attached to the image and lacks prior permutation, e.g., the common question "why is person 4 pointing to person 1" shown in vcr @xcite data set. for another practical cross modal reasoning scenario @xcite , the textual modality often provides prior permutation or complementary information with the source image, such as the commonsense knowledge, and the personalities, feelings, or relationships of persons, as the premise shown in figure 1 . in this paper, we focus on such conditional inference on joint textual and visual clues, where the specific task form is to select the correct option from the candidate set according to the given textual premise and image. previous methods @xcite @xcite @xcite usually input the concatenated sequence of textual premise, image, and candidate answer into powerful pretrained vision-language models (vlms) and employ a task-specific classi-fier to infer the result with attention to the joint representation obtained from vlms. although these methods work well for reasoning based mainly on visual clues, they suffer from one major shortcoming: the reasoning process does not fully utilize the abstract semantic information of given premise text to perform in-context reasoning. as the case shown in figure 1 , pretrained vlms know "person [1] sits on the couch, not the bed" from the image, yet struggle to effectively infer that the person will "have a rest on the couch" according to "feels very tired" presented in the premise. it may be attributed to that pretrained vlms mostly map different modalities into a unified space @xcite and perform cross modal semantic alignment and fusion. they neglect the in-context learning based on the given multi-modal semantics of language and vision during pertaining, like next sentence prediction. fortunately, pretrained language models (plms) such as bert @xcite , roberta @xcite , bart @xcite , and gpt3 @xcite , are powerfully capable of in-context learning and have achieved successful performance on natural language inference and open-ended text generation. plms can infer the next-step intent according to the given abstract text information compared to pretrained vlms. hence, we propose a simple and effective multi-modal in-context reasoning approach named modcr for this multi-modal reasoning task, taking advantages of vlms and plms. specifically, modcr employs a pretrained visual encoder equipped with a vision mapping network to obtain the image representation and convert it into the learnable visual prefix. the visual prefix and textual premise are regarded as two types of pre-context. they will be fed to the in-context reasoner, i.e., language model, to infer the correctness of answer. considering the semantic gap between visual prefix and text in the language model, we first utilize a multi-grained vision-language semantic alignmenter to gain the multi-view alignment representation between image and text. afterwards, we devise an alignment mapping network to capture the pivotal alignment information and convert it into the learnable cross-modal alignment prefix. finally, we fed the two prefixes, premise, and answer into the language model to perform cross modal reasoning in the instruction template-based slot-filling method. in this way, modcr bridges the semantic gap between visual content and text in the language model through introducing the crossmodal alignment prefix. it makes use of the abstract semantic of premise and objective image information via the self-attention mechanism in plms. to verify the effectiveness of modcr, we conduct extensive experiments on two cross modal reasoning data sets: pmr @xcite and vcr @xcite . the experimental results show that the proposed method significantly outperforms previous strong baselines. the ablation and case studies indicate that modcr is capable of in-context reasoning based on multi-modal information. our contributions can be summarised as follows: vision-aided language models. images can provide explicit and diverse visual information to improve the imaginative representation of language. recent works show that vision-aided language models have achieved promising performance on natural language understanding @xcite and open-ended text generation tasks @xcite such as text completion @xcite , story generation @xcite , and concept-to-text @xcite . some works @xcite proposed to retrieve images corresponding to texts from the image corpus and use visual knowledge to improve the performance on the downstream tasks. recently, some researchers @xcite @xcite proposed to utilize the powerful text-to-image technical to obtain the imagination representation of language and infuse them into the language model via the prefix-tuning @xcite way. in this paper, we also compared the visual prefix-based prompt learning methods @xcite @xcite , which has been verified to improve the performance of pretrained language models. ## conclusion and future work in this paper, we propose a multi-modal context reasoning approach named modcr for the scenario of conditional inference on joint visual and textual clues. it regards the given image and text as the two types of pre-context states and infuses them into the language model via the instruction learning method to perform such multi-modal reasoning. the experimental results on two data sets show the effectiveness of modcr. for the future, we will explore two research directions: 1) how to improve the context learning capability of pretrained vlms. 2) exploring the conditional inference on complex visual and textual clues, where it contains multiple clues lying in more modalities.