Learning to Follow Object-Centric Image Editing Instructions Faithfully (2023)

Abstract: Natural language instructions are a powerful interface for editing the outputs of text-to-image diffusion models. However, several challenges need to be addressed: 1) underspecification (the need to model the implicit meaning of instructions) 2) grounding (the need to localize where the edit has to be performed), 3) faithfulness (the need to preserve the elements of the image not affected by the edit instruction). Current approaches focusing on image editing with natural language instructions rely on automatically generated paired data, which, as shown in our investigation, is noisy and sometimes nonsensical, exacerbating the above issues. Building on recent advances in segmentation, Chain-of-Thought prompting, and visual question answering, we significantly improve the quality of the paired data. In addition, we enhance the supervision signal by highlighting parts of the image that need to be changed by the instruction. The model fine-tuned on the improved data is capable of performing fine-grained object-centric edits better than state-of-the-art baselines, mitigating the problems outlined above, as shown by automatic and human evaluations. Moreover, our model is capable of generalizing to domains unseen during training, such as visual metaphors.

## introduction frameworks for large-scale text-conditional image synthesis, which rely on diffusion processes @xcite @xcite have shown impressive generative capabilities and practical uses. notably, image editing guided by text has garnered considerable attention due to its ease of use and seemingly high-quality results @xcite @xcite . these advances are now utilized in leading industry tools such as adobe photoshop foot_0 , bridging the gap between technology and content creators. while natural language instructions act as a powerful interface for editing images, following them remains a challenging task for several reasons. first, these instructions are often underspecified, requiring models to uncover their implicit meaning. for example, in figure 1 for the input image (a), the user provides a prompt add a lighthouse. the model needs to understand how a lighthouse looks, that only one lighthouse needs to be added and that it needs to be placed on land and not in the water. second, models must be able to localize where the "background" is in the image so that the lighthouse can be added appropriately (grounding). finally, models must follow instructions faithfully, i.e. preserve the elements of the image not affected by the edit instruction (e.g., both houses in figure 1 (a) ). one of the key challenges for making progress on the task of image editing via natural language instructions is the lack of high-quality annotated or naturally occurring data. recently, @xcite proposed a way to handle this by automatically creating a paired dataset utilizing large language models and text-to-image models. they further train a conditional diffusion model on this paired data of synthetic examples and show that their model is capable of editing images conditioned on natural language instructions at run-time. however, closer inspection reveals that the data is noisy and sometimes nonsensical. as can be seen in figure 1 (b) the model adds three lighthouses instead of one, likely due to the underspecification of the instruction and the lack of grounding. furthermore, it compromises faithfulness by removing both houses from the input image. to tackle these challenges we first create a highquality corpus starting from the parallel data released by @xcite . for example, given the caption of the input image in figure 1 , ocean cottage by todd baxter, and an edit instruction add a lighthouse, we use recent advances in chainof-thought prompting @xcite to identify whether the transformation can be performed in the context of the input image and what entity should be transformed. the entity and the input image are fed to an object detection model ground-ingdino @xcite . as can be seen in figure 2 (b), dino draws a bounding box to denote "background". however, grounding in the form of bounding boxes cannot entirely disentangle the entity of interest. thus, we use recent advances in image segmentation @xcite to identify the exact segment containing the entity that needs to be masked (see figure 2 (c )) and and use stable diffusion+inpainting @xcite with the masked image (c) to obtain the edited output (figure 2(d) ). to further account for faithfulness in our paired training data we leverage techniques from vqa @xcite to ensure that the edited im-ages remain faithful to the entities in the original image. for this, we generate relevant questions with 'yes/no' answers regarding the unmodified elements in the image such as "does this image contain a cottage" or questions regarding objects in the instruction "is there a lighthouse in the image" using the vicuna-13b model @xcite . given an image-question pair, we then use the blip-2 vqa-model @xcite to collect responses and re-rank generated images from stable diffusion+inpainting by faithfulness scores in terms of correct answers and select the best one for higher quality. we then fine-tune the model on this newly created parallel data. in addition, both during finetuning and inference, we enhance the supervision signal by denoising parts of the image that need to be changed by the instruction as shown in figure 5 . to summarize, our contributions are: ## fine-tuning with additional supervision our training set curation pipeline led to a set of 52,208 high-quality instruction-images pairs. we split this data in 80:10:10 for the training, validation, and test sets, respectively. we follow the same protocol as instructpix2pix training and use their codebase. for an image x, the diffusion process adds noise to the encoded latent producing a noisy latent @xmath0. we learn a network θ that predicts the noise added to the noisy latent z t given image conditioning c i and text instruction conditioning c t . we minimize the latent diffusion objective and initialize the weights of our model with the instructpix2pix checkpoint. to support image conditioning, we add input channels to the first convolutional layer, concatenating z t and e(c i ). all available weights of the diffusion model are initialized from the pre-trained checkpoints, and weights that operate on the newly added input channels are initialized to zero. we reuse the same text conditioning mechanism that was originally intended for captions to instead take as input the text edit instruction c t . we experiment with two different training strategies with additional supervision on top of the vanilla instructpix2pix. ## models below we describe how we use some of the stateof-the-art baselines as well as ablations of our best model to generate outputs based on the test set instruction-image pairs. ## evaluation automatic evaluation using tifa-score we use a recent metric proposed by @xcite , tifa (text-to-image faithfulness evaluation with question-answering), 4 for evaluating the faithfulness of the generated images to their text inputs. tifa evaluates a generated image using a two-stage pipeline: first, generates a list of question-answer pairs that cover various aspects of the contextual information provided in the given caption, and then uses sota vqa models such as blip-2 to answer these questions and match the correct answers for faithfulness. this framework is proposed for benchmarking diffusion models. the framework generates 7-10 question-answer pairs per image caption (modified, which covers the instructional edit aspects too). we average the score for individual images over all these question pairs. the final score is reported as the average of the tifa score over all the images in the test set. ## conclusion we address challenges in natural language-based image editing tasks and provide a novel approach to enhance the quality of the training data. we improve the supervision signal and tackle the issues of underspecification, grounding, and faithfulness by leveraging advancements in segmentation, chainof-thought prompting, and vqa. our models finetuned on the improved dataset with enhanced supervision signal outperform the existing baselines on object-centric image editing both in terms of automatic and human evaluation. moreover, our models showcase the capability to edit faithfully on out-of-domain datasets. overall our findings highlight the significance of high-quality annotation and grounded supervision signals for precise and faithful image editing.