TAME - RD : Text Assisted Replication of Image Multi-Adjustments for Reverse Designing (2024)

Abstract: Given a source and its edited version performed based on human instructions in natural language, how do we extract the underlying edit operations, to automatically replicate similar edits on other images? This is the problem of reverse designing, and we present TAME-RD, a model to solve this problem. TAME-RD automatically learns from the complex interplay of image editing operations and the natural language instructions to learn fully specified edit operations. It predicts both the underlying image edit operations as discrete categories and their corresponding parameter values in the continuous space.We accomplish this by mapping together the contextual information from the natural language text and the structural differences between the corresponding source and edited images using the concept of pre-post effect. We demonstrate the efficiency of our network through quantitative evaluations on multiple datasets. We observe improvements of 6-10% on various accuracy metrics and 1.01X-4X on the RMSE score and the concordance correlation coefficient for the corresponding parameter values on the benchmark GIER dataset. We also introduce I-MAD, a new two-part dataset: I-MAD-Dense, a collection of approximately 100K source and edited images, together with automatically generated text instructions and annotated edit operations, and I-MAD-Pro, consisting of about 1.6K source and edited images, together with text instructions and annotated edit operations provided by professional editors. On our dataset, we observe absolute improvements of 1-10% on the accuracy metrics and 1.14X–5X on the RMSE score.

## introduction the prevalence of digital media in education @xcite , healthcare @xcite , business and entertainment @xcite comes with the need for large-scale image * work partly done as an intern at adobe inc. however, the realm of reverse designing remains relatively unexplored compared to forward designing. complex many-to-many relationships in mappings from source to edited images contribute to this gap, making it challenging to establish optimization criteria solely based on factors like the total number of edit operations or specific parameter value ranges. additionally, the lack of standardization in forward design practices among image editors further complicates this challenge. reverse designing holds significant potential for generating comprehensive editing histories, facilitating the replication and reuse of large volumes of image assets. the availability of editing history proves advantageous across various applications including image editing revision control @xcite , automatic tutorial generation @xcite , editing visualization @xcite , quality control @xcite , and forensic analysis @xcite . however, relying only on the pairs of source and edited images for spin the image. convert the image to achromatic tones. apply a filter that accentuates the natural beauty and depth of the image. transform the image to face in the opposite direction. rotate grayscale hdr source target text flip increase the light levels of the image without losing details in the shadows. give the image a more vibrant and lifelike appearance. bright hdr flip the image across the vertical axis.. adjust the strength of the coloration in the image. enhance the image's sharpness. flip saturate sharp source target text give the image a serene and tranquil feel. enhance the image with a greater range of tonality. winter hdr add a warm, earthy tone to the image. enhance the image's detail. create an image that captures a greater range of brightness and colors. add a bright, cheerful tone to the image. sepia sharp hdr summer alter the chromatic intensity of the image. adjust the ratio of dark to light areas in the image. apply a filter that accentuates the natural beauty and depth of the image. saturate contrast hdr source target text source target text bright hdr rotate hdr flip sepia sharp hdr summer hdr sharp flip saturate saturate contrast hdr winter grayscale spin the image. convert the image to achromatic tones. apply a filter that accentuates the natural beauty and depth of the image. transform the image to face in the opposite direction. rotate grayscale hdr get text flip increase the light levels of the image without losing details in the shadows. give the image a more vibrant and lifelike appearance. bright hdr flip the image across the vertical axis.. adjust the strength of the coloration in the image. enhance the image's sharpness. flip saturate sharp source target text give the image a serene and tranquil feel. enhance the image with a greater range of tonality. winter hdr add a warm, earthy tone to the image. enhance the image's detail. create an image that captures a greater range of brightness and colors. add a bright, cheerful tone to the image. sepia sharp hdr summer alter the chromatic intensity of the image. adjust the ratio of dark to light areas in the image. apply a filter that accentuates the natural beauty and depth of the image. saturate contrast hdr spin the image. convert the image to achromatic tones. apply a filter that accentuates the natural beauty and depth of the image. transform the image to face in the opposite direction. rotate grayscale hdr source target text flip increase the light levels of the image without losing details in the shadows. give the image a more vibrant and lifelike appearance. bright hdr flip the image across the vertical axis.. adjust the strength of the coloration in the image. enhance the image's sharpness. flip saturate sharp source target text give the image a serene and tranquil feel. enhance the image with a greater range of tonality. winter hdr add a warm, earthy tone to the image. enhance the image's detail. create an image that captures a greater range of brightness and colors. add a bright, cheerful tone to the image. sepia sharp hdr summer alter the chromatic intensity of the image. adjust the ratio of dark to light areas in the image. apply a filter that accentuates the natural beauty and depth of the image. saturate contrast hdr reverse designing may be insufficient. this limitation arises from the need to interpret the creative decisions and goals of the editor in the reverse design process. to address this, we propose a more promising approach that involves incorporating an additional modality to provide contextual information. we integrate the editors' intent expressed through natural language text. this might elucidate the forward design mechanism they employed or even discern specific reasons or details associated with the editing process that transformed the source image into the edited image. documenting machine-level micro-instructions that can be directly fed to image editing softwares is not scalable, particularly when dealing with a large number of images. consequently, any textual information available for reverse designing purposes will primarily be high-level, vague, and in natural language. our multimodal approach seamlessly integrates image pairs with text-based contextual cues, enabling a more nuanced understanding of the editing process that led to obtaining the edited image from the source image. the inclusion of a text component proves invaluable in complementing visual information, particularly in instances of noise or insufficient clarity. this holistic strategy endeavors to address the gap in reverse design research by prioritizing the interplay between visual and language-based sources of information. ## related work we provide a brief overview of recent works on understanding the relationship between two images, and datasets containing comparisons between two images. forward design. current image editing approaches, including those using language-based instructions @xcite @xcite , excel at forward design by automatically applying specific image edits. however, they lack the key capability for reverse design, which is extracting edit operations given source and edited images. style transfer @xcite and imageto-image translation @xcite methods directly apply changes from one image onto another. however, they do not provide control over the multi-adjustments that underlie the entire transformation as required for reverse design. ## datasets we provide details on the datasets we experiment with: the benchmark gier dataset @xcite and our proposed dataset i-mad. ## experiment results and analyses we discuss the hyperparameter details and metrics in sec. 5.1, baselines and quantitative evaluations in sec. 5.2, and ablation experiments in sec. 5.3. ## conclusion we have presented tame-rd, a method to automatically detect fully-specified edit operations given pairs of source and edited images, and optionally, contextual information in the form of natural language text. our fully-specified edit operations consist of categorical edits paired with corresponding parameter values. we have also proposed a corresponding dataset i-mad, consisting of 100k tuples (source image, edited image, text) annotated with fully-specified edit operations between the image pairs, to enable further research on this task as well as downstream image editing tasks.