A Neural Divide-and-Conquer Reasoning Framework for Image Retrieval from Linguistically Complex Text (2023)

Abstract: Pretrained Vision-Language Models (VLMs) have achieved remarkable performance in image retrieval from text. However, their performance drops drastically when confronted with linguistically complex texts that they struggle to comprehend. Inspired by the Divide-and-Conquer algorithm and dual-process theory, in this paper, we regard linguistically complex texts as compound proposition texts composed of multiple simple proposition sentences and propose an end-to-end Neural Divide-and-Conquer Reasoning framework, dubbed NDCR. It contains three main components: 1) Divide: a proposition generator divides the compound proposition text into simple proposition sentences and produces their corresponding representations, 2) Conquer: a pretrained VLMs-based visual-linguistic interactor achieves the interaction between decomposed proposition sentences and images, 3) Combine: a neural-symbolic reasoner combines the above reasoning states to obtain the final solution via a neural logic reasoning approach. According to the dual-process theory, the visual-linguistic interactor and neural-symbolic reasoner could be regarded as analogical reasoning System 1 and logical reasoning System 2. We conduct extensive experiments on a challenging image retrieval from contextual descriptions data set. Experimental results and analyses indicate NDCR significantly improves performance in the complex image-text reasoning problem.

## introduction image-text retrieval tasks have made remarkable progress owing to pretrained vision-language models (vlms) such as lxmert @xcite , uniter @xcite , os-car @xcite , vil-bert @xcite , clip @xcite , figure 1 : an example from the imagecode @xcite data set, where the description is linguistically complex and images are minimally contrastive. the target image is in red and others are incorrect frames. the bottom part depicts the conventional method and the neural divide-and-conquer reasoning framework. and many others. these vlms are usually trained on the large-scale short text-image corpus by crossmodal semantic alignment methods. they are capable of essential perceptual computing capability and excel at retrieving images from sentences with few objects and simple linguistic, e.g., "there is a duck swimming in the pond". however, when pretrained vlms meet the case of retrieving the accurate image from similar candidates based on a linguistically complex text, as the example shown in figure 1 , previous works @xcite @xcite show that they struggle to understand the elaborate description and perform complex cross-modal reasoning. according to the dual-process theory for human thinking @xcite @xcite , human brains contain two thinking systems: system 1 performs analogical reasoning well, which is fast yet unconscious; system 2 is capable of abstract logical reasoning, which is slow yet conscious and well-suitable for complex reasoning problems. the theory could also hold for the image-text retrieval tasks, and the widely adopted models (e.g., vlms) focus on analogical reasoning as system 1 based on the analysis of deep learning networks @xcite @xcite . for the linguistically complex description that contains multiple conditions, they have inferior performance, and we need to introduce logical reasoning system 2 more to cover and logically incorporate the scattered information in the description based on system 1. inspired by the above investigations and classical @xcite algorithm, we design an end-to-end neural divide-and-conquer reasoning framework named ndcr. as shown in figure 1 , our key idea is to regard the complex description as compound proposition text and solve the challenging retrieval problem in three steps: divide, conquer, and combine. specifically, divide: ndcr first utilizes a proposition generator to divide the complex compound text and produce the global representation of simple proposition sentences with visually printing them. conquer: we devise a visuallinguistic interactor to achieve the interaction between decomposed proposition sentences and images, which resembles system 1. it uses the transformer @xcite -based contextual interactor to achieve the inter-learning of different proposition-image pairs. considering the incorrectness or information loss of simple proposition representation, we also present a modifier to incorporate the context reasoning information to improve their cross-modal reasoning states. combine: we design a learnable neural-symbolic reasoner to integrate reasoning information of simple propositions logically. it first employs a negation executor to obtain a simple proposition sentence's negational reasoning hidden state and corresponding confidence score. then, we use the global reasoning information of compound proposition text as the query signal to perform the conjunction operation across simple propositions and their negational information. finally, as shown in figure 1 , we also combine the inferred results of the neural-symbolic reasoner (resembles system 2) and visual-linguistic interactor (resembles system 1) to obtain the final solution. in this way, the whole framework integrate the capabilities of systems 1 and 2 to obtain better performance. we conduct extensive experiments on a largescale image retrieval from contextual descriptions data set, imagecode @xcite . the experimental results indicate that ndcr achieves the state-of-the-art performance and the ablation and case studies verify the effectiveness of different modules. our contributions are as follows: ## conclusion in this paper, inspired by the divide-and-conquer algorithm and dual-process theory, we introduced an end-to-end neural divide-and-conquer reasoning framework named ndcr to handle the challenging case of image retrievals from linguistically complex text. ndcr contains a proposition generator to divide the compound proposition text into multiple simple proposition sentences, then uses a visuallinguistic interactor to achieve the interaction of simple propositions and images. to improve the logical reasoning capability, we devise a neuralsymbolic reasoner to gain the logical inferring result based on the output of the visual-linguistic interactor. this way, ndcr performs the lowlevel analogically perceptual computing in system 1 (visual-linguistic interactor) and high-level logical reasoning in system 2 (neural-symbolic reasoner). finally, we combine the output result in systems 1 and 2 to obtain the final solution.