Cost-Effective Language Driven Image Editing with LX - DRIM (2022)

Abstract: Cross-modal language and image processing is envisaged as a way to improve language understanding by resorting to visual grounding, but only recently, with the emergence of neural architectures specifically tailored to cope with both modalities, has it attracted increased attention and obtained promising results. In this paper we address a cross-modal task of language-driven image design, in particular the task of altering a given image on the basis of language instructions. We also avoid the need for a specifically tailored architecture and resort instead to a general purpose model in the Transformer family. Experiments with the resulting tool, LX-DRIM, show very encouraging results, confirming the viability of the approach for language-driven image design while keeping it affordable in terms of compute and data.

## introduction the fields of image and language processing have mostly progressed independently of one other, each focusing on its own modality. recently, though, there have been promising prospects for advancement in cross-modal processing. a major motivation for this has been the realization that the so-called grounding is necessary for progress in language understanding @xcite , and a major enabling factor has been the emergence of underlying technology that can be successfully applied to both modalities and their cross-modal processing @xcite @xcite . in the image to language direction, there has been considerable progress in the task of image captioning, that is of generating a language description for an input image @xcite @xcite , and the subsidiary task of image retrieval from a language description @xcite @xcite ; while in the language to image direction promising results have been obtained on the task of image generation from an input language description @xcite . conditional generative models based on the transformer architecture @xcite became one of the mainstream approaches for virtually any language processing task @xcite @xcite due to their ability to cope with the intrinsically compositional nature of language and the meaning conveyed by contextualized expressions. recently, these models have also shown promise for image processing tasks, namely in image generation @xcite , showcasing their capacity to handle multi-modal input, and how general purpose the transformer architecture can be, coping also with data rooted in signals that are not linguistic in nature. the dall-e model @xcite delivered promising results in such a task, by receiving a description in the form of a snippet of text (e.g. "a green clock in the form of an hexagon") and creating an image that humans recognize as one that could correspond to that input description. and its extension dall-e 2 @xcite undertakes also a more restricted task, where a specified subarea of the image is to be completed on the basis of the language description. these models achieve these results by leveraging massive quantities of data and compute that are hardly accessible to most research groups and organizations. adopting a distinct line of inquiry, in the present paper we aim at addressing a challenge of language driven image design, consisting of editing an image on the basis of language instructions to do so. here the output image is conditioned not only on a text snippet but also on an input image, such that that image is appropriately altered taking into account the language input. for example, given an image of a piece of furniture, the model is asked to change its color. and then possibly its height, shape, viewing perspective, or the direction of the light. this process should allow one to iteratively and interactively modify the design of some object without any specific image manipulation software, and with no knowledge of how to work with it. this workflow can be exploited in a wide range of innovative applications, such as supporting a shopping assistant that progressively matches images altered by language instructions against current stock and suggests increasingly suitable products, among others examples. also concerned with addressing the issue of resource cost, in this paper we present exploratory research results on affordable language driven image design (ldid). the major contributions and findings of this study are: (i) a suitably instantiated gpt-2 @xcite is an effective option to perform ldid; @xcite in what concerns the task of conditional image generation, our approach offers a more streamlined setup than the one adopted in dall-e; (iii) as a by-product of its ability for ldid, our model may usefully support the subsidiary task of image retrieval; and (iv) extending this set up with a pre-trained language model may improve the performance in some ldid tasks. this study resulted on the creation of a tool, lx-drim, for editing an image on the basis of language instructions. the remainder of this document is structured as follows: section 2 describes the neural model used in this study; section 3 explains the experiments performed and introduces the data sets used; section 4 presents the results obtained; section 5 proceeds with error analysis; section 6 discusses related work; and section 7 closes the paper with concluding remarks. ## model in looking for affordable ldid, we resorted to a gpt-2 small model @xcite , namely its current implementation from the transformers package of huggingface, foot_0 including their english pre-trained gpt-2 as well. 2 gpt-2 has been successfully applied to virtually all language processing tasks. given it was conceived for text, some adaptation is required in order for it to handle images. interestingly, changes to the model architecture can be dispensed with, and the required adaptations can be restricted solely to the way the input data is pre-processed. the minimal twist is to pass the images through a vector-quantized variation auto encoder (vq-vae) that is both capable of describing an image with tokens according to an internal vocabulary of images and of constructing an image from those tokens @xcite . similarly to variational autoencoders, the main goal of vq-vaes is the encoding of an image into a vector, or group of vectors, that can then be decoded as closely as possible into the same original image. however, while in standard variational autoencoders, the latent space is continuous and is sampled from a gaussian distribution, vq-vaes operate on a discrete latent space by maintaining a codebook. this codebook can then be used as vocabulary for text conditioned image generation. therefore, by passing an image through a vq-vae, one gets a sequence of tokens that represents the image. this sequence can be fed to a gpt-2 model like it is done with the sequence of tokens for language, given that the image tokens also have their own embedding in the embedding layer. in this work we use the vq-vae from @xcite foot_2 , with a "vocabulary" for images of size 1024, which is added to the gpt-2 embedding map, and by means of which every image is represented. with this extension to images in place, one can now proceed to train gpt-2 as it is done when it is applied solely to text, whereby given an input token it learns to predict the next one. as training parameters for the gpt-2, we use a batch size of 6 with gradient accumulation of 16, meaning that at each step our model backpropagates with 96 training instances. we evaluate on the development set every 250 steps, and stop training when the development set loss does not decrease from its lowest point after 5 evaluations. after the training of the gpt-2 model, we optionally rank its outputs using clip 4 over the various images from the same input. after using two separate encoders, for image and for text, clip maps their encoding vectors into a common embedding so that a caption and its respective image end up with the same representation @xcite . clip can thus support the ranking of images generated from a caption given the encoded image that is closer (in vector space) to the encoded caption is the one more closely described by the caption. ## experiments with this model in place, the following experiments were undertaken: 5 (i) a warm up experience, aimed at assessing the capability of the model for conditional image generation (cig)-generating an image from a text snippet describing it; (ii) the central experiment of interest here, aimed at assessing how well the model is able to perform conditional image alteration (cia)-generating an image both from another image and from a text snippet describing how the later should be altered; and, in addition, (iii) a comparison between the model and a variant obtained by extending it with a language pre-training phase. ## results the evaluation of a generative task (e.g. summarization, etc.), where typically there can be more than one output that is acceptable as correct, tends to be a problematic endeavour. while one could try to compare to a gold standard in order to perform an automatic evaluation, small differences (of equally acceptable outputs) to the gold example inevitably makes most such metrics, like accuracy, etc., useless, leaving only some kind of distance metric to be resorted to. in contrast to text processing, this problem tends to be further aggravated for images, as metrics that are used to evaluate textual generative tasks, like bleu @xcite or meteor (banerjee and lavie, 2005), work by being able to refer to some parts that are well defined substructures in an expression (e.g. words), but for images there are no clear substructures that can be resorted to, and in most cases these distance metrics work only at the pixel level. ## error analysis to help in error analysis, difficult cases are exemplified in figure 1 . the two leftmost shoes are, respectively, the target image and the (cig) generated image with the description "dark red pumps". both shoes are quite similar in terms of shape, but their color is different. this is a good illustration that color saturation and lightness are subjective and hard to transmit via text. in the target image (1st column), the desired dark red is almost black, and the image generated (cig) from "dark red pumps" (2nd column) is lighter. interestingly, even the tentative correction (cia) of this image with the instruction "are a darker red" still does not produce an image (3rd column) that is not as dark as in the first column. though image retrieval is not a central task of interest in this paper, it is worth noting that this may be even more serious for image retrieval as slight changes in saturation and lightness can make the system choose a different image: when trying to retrieve an image from the database, using the generated image (2nd column), the image that is retrieved is the one at the fourth column. further difficult examples, generated by the cia model, are shown in figure 3 . one problem illustrated there concerns image clarity. even though some images (see 1st column) are correct, they have some fuzzy details. this is likely due to the reduced volume of the training data set. however, as already mentioned, in order to have images with higher resolution given a data set of this size, one would have to sacrifice image relevance and precision. another problem arises when the target image is very different from the source image (see 2nd column). in such cases, the model is basically asked to create a quite different object, for which the small size of the data set provided limited evidence. additional problems occur when the images to be generated are too similar to the source image (see 3rd column), or the generated images are too similar to each other (see 3rd and 4th images in the 1st column). while not necessarily a problem for the overall quality of the output, the first kind of cases becomes an issue for evaluation, as generated images may be more similar to the source image than to the target one. as for the second kind of cases, when the generated images are similar to one another, it may become a problem if object design is the intended use for the tool, and not just image alteration. to address these issues, further techniques to enhance image diversity should be explored in future work, so that the model can suggest a more varied set of images to the user. ## related work a promising application of deep learning to image generation was presented in @xcite , with a generative adversarial network (gan), a forerunner of a research line continued in @xcite , @xcite , @xcite , a.o. a two part network containing a generator and a discriminator was proposed: the generator tries to create fake yet as realist as possible images, while the discriminator tries to distinguish the fake images produced by the generator from real ones. despite this early success being attributed also to the use of convolution neural networks (cnn) @xcite , the concept of gan can be used with other deep learning approaches. such is the case of the more recent work in @xcite , where two transformer models @xcite are used as a discriminator and a generator respectively. with no convolution at its core, they achieve competitive scores when compared to their cnn counterparts. transformers gained their notoriety with their success in languages processing tasks of all kinds, and recently they have been applied to other data modalities. relevant models that use transformers for image generation from captions are dall-e @xcite , and nuwa @xcite . the major difference between them is that nuwa also uses video while dall-e works only with pictures, and that nuwa uses a different type of attention mechanism, 3d nearby attention. the approach proposed in (galatolo et al., 2021) also achieves promising results in image generation with a pre-trained transformer clip @xcite , only by training a genetic algorithm. more recently dall-e 2 @xcite ) improves upon its predecessor by incorporating the clip model for image and caption representation, and through the use of a diffusion model for image generation @xcite . the architecture adopted in our model is similar to the backbone architecture on which the implementation of dall-e is based. our model is different from dall-e, however, in not having any specific optimization performed on the base transformer, like it was done to set up dall-e, and in being of a more reduced size (124m vs. 12b parameters). our system also differs in that it is geared for a task other than the conditional image generation one, of dall-e, namely the task of conditional image alteration. it happens also that it was trained in a much smaller amount of data (10750 vs. 250 million examples). also, related to our research topic, @xcite tackles the same task, though by means of a generator/discriminator architecture, with data that while similar to ours is not the same. to the best of our knowledge, that dataset is not publicly available, so no comparison was possible. @xcite also work with language guided im-age edition, with different datasets that do not tackle the problem of object shape manipulation. work on image editing without language guidance can be found in the work of @xcite , on different datasets. the research presented here appears as a more streamlined approach for the tasks involved in language driven image design since most of the work is performed with a common decoder-only architecture, in the form of a gpt-2 small model. this is a generalist architecture that can be adapted for other tasks, as it was the case here with the cig task, or any other task that can be represented by a sequence (text, audio, image, etc.). ## conclusion the present study explored conditional generative models for language driven image design, by means of an affordable gpt-2 instantiation with only 124m parameters. the central task of interest here was conditional image alteration, consisting of generating a new image given a source image and a textual instruction for its alteration, on which the proposed lx-drim application showed a performance rated at 2.37 (in 1-5) by manual evaluators. resorting to the same data set, the task of conditional image generation, consisting of generating an image given a textual description, was also experimented with. very encouraging results were also obtained, specially taking into account that the data set used here was several orders of magnitude smaller than the one that has been used in the literature for this task. in addition, we found also that as by-product of its cross-modal processing ability, our model may usefully support the subsidiary task of image retrieval through the use of its generated images. empirical experimentation obtained very encouraging results and demonstrated that the proposed approach can support an effective solution to language driven image design and represents a promising research path whose potential is worth being further exploited. the present study focuses on changing a single object in the image, rather than multiple objects in a scene. future work the task of scene manipulation @xcite should be investigated by exploiting the approach developed here with single object manipulation.