SSCR : Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning (2020)

Abstract: Iterative Language-Based Image Editing (ILBIE) tasks follow iterative instructions to edit images step by step. Data scarcity is a significant issue for ILBIE as it is challenging to collect large-scale examples of images before and after instruction-based changes. Yet, humans still accomplish these editing tasks even when presented with an unfamiliar image-instruction pair. Such ability results from counterfactual thinking, the ability to think about possible alternatives to events that have happened already. In this paper, we introduce a Self-Supervised Counterfactual Reasoning (SSCR) framework that incorporates counterfactual thinking to overcome data scarcity. SSCR allows the model to consider out-of-distribution instructions paired with previous images. With the help of cross-task consistency (CTC), we train these counterfactual instructions in a self-supervised scenario. Extensive results show that SSCR improves the correctness of ILBIE in terms of both object identity and position, establishing a new state of the art (SOTA) on two IBLIE datasets (i-CLEVR and CoDraw). Even with only 50% of the training data, SSCR achieves a comparable result to using complete data.

## introduction digital design tools like illustrator or photoshop are widely used nowadays. though having considerable user demand, they require prior knowledge and multiple steps to use successfully. these applications would significantly improve the accessibility if they can automatically perform corresponding editing actions based on the language instructions given by users for each step. iterative language-based image editing (ilbie) task follows iterative instructions to edit images step by step, as illustrated in fig. 1 . to accomplish ilbie, models are required not only to modify images but also to understand the visual differences figure 1 : an example of the iterative language-based image editing (ilbie) task. during each turn, the model edits the image from the previous turn based on the current instruction. eventually, a desired image is accomplished after iterative editing. note that the generation is at the pixel level. a gan-based @xcite model, geneva @xcite , is proposed to perform ilbie, where a conditional generator serves as the image editor, and a discriminator provides the training loss by discriminating a resulting image. though it yields promising results, geneva neglects the data scarcity issue of ilbie. as a binary classifier, the discriminator easily suffers from data scarcity given the shortage of ground-truth instruction-and-resulting-image pairs and thus limits the model's generalization ability to new instructions. despite lacking prior experiences with images or instructions, humans can still accomplish editing under unfamiliar image-instruction pairs. for ex-ample, for a given instruction "add a purple cube in front of the blue cylinder", humans can think about the resulting image if the instruction changed to "adding a blue square" or "on the right of". this process is known as counterfactual thinking @xcite , which allows humans to operate in datascarce scenarios by considering alternative instructions from the seen examples. in this paper, we introduce self-supervised counterfactual reasoning (sscr) that incorporates counterfactual thinking to deal with the data scarcity issue. sscr allows the model to think about expected, resulting images under unseen instructions. since there are no ground-truth resulting images, we propose cross-task consistency (ctc), which adopts an iterative explainer to reconstruct the instruction of each step. with ctc, we can supply detailed token-level training loss (e.g., wrong objects or wrong positions), which is better than only using the discriminator, and consider these counterfactual instructions in a self-supervised scenario. the experimental results on i-clevr @xcite and codraw @xcite show that our sscr can improve the correctness of the ilbie task in both aspects of object identity and position. in summary, our contributions are three-fold: ## related work text-to-image (t2i) generates an image that matches the given instruction. t2i is challenging yet important that has a vast potential in practical applications like art generation or automatic design @xcite @xcite . with the success of a generative adversarial network @xcite on the image generation task, several works @xcite @xcite introduce different gan-based models to synthesize an image from a text description. unlike t2i, we focus on image editing, where a model needs to understand the visual differences between two images rather than generating an image from scratch. language-based image editing (lbie) tasks a model to edit an image based on the guided text description. pixeltone @xcite and image spirit @xcite are both rulebased, which accept only pre-defined instructions and semantic labels that can significantly decrease the practicality of lbie. some studies @xcite adopt the conditional gan model to attend on the instruction and perform lbie as image colorization. however, image colorization is not truly an editing task since it only supports fixed object templates, and the objects and the scene of the image remain the same after editing. in contrast, the editing processes of photoshop or illustrator are not accomplished in a single pass. geneva @xcite proposes an iterative gan-based generator to accomplish iterative language-based image editing (ilbie) but neglects the data scarcity issue. counterfactual thinking (roese, 1997) is a concept that refers to the human propensity to consider possible alternatives to events that have happened already. people can consider different outcomes from a wide range of conditions and engage in causal reasoning by asking questions like "what if ...?" or "if i had only...." previous works @xcite have shown how counterfactual fairness improves the robustness of the model and makes it more explainable. furthermore, counterfactual thinking has also been applied to augment training targets @xcite . in this paper, we incorporate counterfactual thinking into that ilbie task that considers counterfactual instructions to deal with the data scarcity issue and improve the generalizability. ## experiments codraw is a more difficult art-like dataset of children playing in a park. there are 58 objects and children with different poses. we use the same split as in codraw where 7,988, 1,002, and 1,002 are for training, validation, and testing. evaluation metrics standard metrics like inception score @xcite or frechet inception distance (fid) @xcite cannot detect whether the editing is correct based on the instruction @xcite . following geneva @xcite , we adopt f1 and relsim to evaluate the editing result. the f1 score is based on a pre-trained object detector @xcite (˜99% accuracy on both i-clevr and codraw), which detects the objects in the predicted images that meet the groundtruth resulting images. to evaluate not only object type but also object position, we build the scene graph according to the object detector. the edges are given by the left-right and front-back relations between the vertices (objects). then, relsim determines how many of the ground-truth relations are in the predicted images: baseline we use the sota model geneva 1 as our baseline: it shares the same model architecture 1 we reproduce the result for geneva by their official github repo ( https://github.com/maluuba/geneva ). we apply the default hyperparameters as them, and issue #2 can support that the results are comparable. as our iterative editor and is trained with the gan objective but without the cross-task consistency (ctc) and our counterfactual reasoning. ## conclusion and future work we present a self-supervised counterfactual reasoning (sscr) framework that introduces counterfactual thinking to cope with the data scarcity limitation for iterative language-based image editing. sscr allows the model to consider new instruction-image pairs. despite without groundtruth resulting images, we propose cross-task consistency (ctc) to provide a more explicit training signal and train these counterfactual instructions in a self-supervised scenario. experimental results show that our counterfactual framework not only trains the image editor better but also improves the generalizability, even under data scarcity. for the real world, both visual and linguistic will be more complicated. to accomplish real-world image editing, large-scale pre-trained language encoders and image generators should be applied to understand the diverse instructions and model the interaction for editing. from a theoretical perspective, our sscr is a model-agnostic framework that can incorporate with any image generator, for gan or non-gan architecture, to perform real-world image editing. currently, the interactive explainer and counterfactual intervention in sscr both improve the editing quality in the token-level. to make it more suitable for real-world images, semantic-level intervention for the diverse natural instructions can support better counterfactual reasoning. also, a stronger explainer that explains not only token-level error but also global editing operation between two images can provide robust self-supervised loss.