Image Manipulation via Multi-Hop Instructions - A New Dataset and Weakly-Supervised Neuro-Symbolic Approach (2023)

Abstract: We are interested in image manipulation via natural language text – a task that is useful for multiple AI applications but requires complex reasoning over multi-modal spaces. We extend recently proposed Neuro Symbolic Concept Learning (NSCL), which has been quite effective for the task of Visual Question Answering (VQA), for the task of image manipulation. Our system referred to as NeuroSIM can perform complex multi-hop reasoning over multi-object scenes and only requires weak supervision in the form of annotated data for VQA. NeuroSIM parses an instruction into a symbolic program, based on a Domain Specific Language (DSL) comprising of object attributes and manipulation operations, that guides its execution. We create a new dataset for the task, and extensive experiments demonstrate that NeuroSIM is highly competitive with or beats SOTA baselines that make use of supervised data for manipulation.

## introduction the last decade has seen significant growth in the application of neural models to a variety of tasks including those in computer vision @xcite , nlp @xcite , robotics and speech @xcite . it has been observed that these models often lack interpretability @xcite , and may not always be well suited to handle complex reasoning tasks @xcite . on the other hand, classical ai systems can seamlessly perform complex reasoning in an interpretable manner due to their symbolic representation @xcite . but these models often lack in their ability to handle low-level representations and be robust to noise. neuro-symbolic models @xcite @xcite overcome these limitations by combining the power of (purely) neural with (purely) symbolic representations. studies @xcite @xcite have shown that neuro-symbolic models have several desirable properties such as modularity, interpretability, and improved generalizability. our aim in this work is to build neuro-symbolic models for the task of weakly supervised manipulation of images comprising multiple objects, via complex multi-hop natural language instructions. specifically, we are interested in weak supervision that only uses the data annotated for vqa tasks, avoiding the high cost of getting supervised annotations in the form of target manipulated images. our key intuition here is that this task can be solved simply by querying the manipulated representation without ever explicitly looking at the target image. the prior work includes weakly supervised approaches @xcite ) that require textual descriptions of images during training and are limited to very simple scenes (or instructions). (see section 2 for a survey). our solution builds on neuro-symbolic concept learner (nscl) proposed by @xcite ) for solving vqa. we extend this work to incorporate the notion of manipulation operations such as change, add, and remove objects in a given image. as one of our main contributions, we design novel neural modules and a training strategy that just uses vqa annotations as weakly supervised data for the task of image manipulation. the neural modules are trained with the help of novel loss functions that measure the faithfulness of the manipulated scene and object representations by accessing a separate set of query networks, interchangeably referred to as quantization networks, trained just using vqa data. the manipulation takes place through interpretable programs created using primitive neural and symbolic operations from a domain specific language (dsl). separately, a network is trained to render the image from a scene graph representation using a combination of l 1 and adversarial losses as done by @xcite . the entire pipeline is trained without any intermediate supervision. we refer to our system as neuro-symbolic image manipulator (neurosim). figure 1 shows an example of i/o pair for our approach. contributions of our work are as follows: ## related work in terms of task complexity, the closest to us are approaches such as tim-gan @xcite , geneva @xcite , which build an encoder-decoder architecture and work with a latent representation of the image as well as the manipulation instruction. they require a large number of manipulated images as explicit annotations for training. in terms of technique, the closest to our work are neuro-symbolic approaches for vqa such as nsvqa @xcite , nscl @xcite , neural module networks @xcite and its extensions @xcite . clearly, while the modeling approach is similar and consists of constructing latent programs, the desired tasks are different in the two cases. our work extends the nscl approach for the task of automated image manipulation. @xcite , @xcite deal with editing global features, such as brightness, contrast, etc., instead of object-level manipulations like in our case. recent models such as instruct-pix2pix @xcite , dall-e @xcite and imagen @xcite ) on text-to-image generation using diffusion models are capable of editing images but require captions for input images; preliminary studies @xcite highlight their shortcomings in compositional reasoning and handling relations. ## conclusion we present a neuro-symbolic, interpretable approach neurosim to solve image manipulation task using weak supervision in the form of vqa annotations. our approach can handle multi-object scenes with complex instructions requiring multihop reasoning, and solve the task without any output image supervision. we also curate a dataset of image manipulation and demonstrate the potential of our approach compared to supervised baselines. future work includes understanding the nature of errors made by neurosim, having a human in the loop to provide feedback to the system for correction, and experimenting with real image datasets.