C hat E dit: Towards Multi-turn Interactive Facial Image Editing via Dialogue (2023)

Abstract: This paper explores interactive facial image editing through dialogue and presents the ChatEdit benchmark dataset for evaluating image editing and conversation abilities in this context. ChatEdit is constructed from the CelebA-HQ dataset, incorporating annotated multi-turn dialogues corresponding to user editing requests on the images. The dataset is challenging, as it requires the system to dynamically track and edit images based on user requests, while generating appropriate natural language responses. To address these challenges, we propose a framework comprising a dialogue module for tracking user requests as well as generating responses, and an image editing module for editing images accordingly. Unlike previous approaches, our framework directly tracks the user request of the current turn from the entire dialogue history and edits the initial image instead of manipulating the output from the previous turn, mitigating error accumulation and attribute forgetting issues. Extensive experiments on the ChatEdit dataset demonstrate the superiority of our framework over previous methods and also improvement rooms, encouraging future research. We will release the code and data publicly to facilitate advancements in complex interactive facial image editing.

## introduction with the rise of deep generative models such as gans @xcite @xcite and ddpms @xcite @xcite , significant progress has been achieved in instructionbased facial image editing @xcite @xcite . however, an emerging scenario is multi-turn interactive editing, allowing users to iteratively refine their editing instructions through system interaction @xcite @xcite . existing approaches @xcite @xcite typically treat multi-turn editing as a sequence of successive single-turn edits, leading to issues such as attribute forgetting and error accumulation, as depicted in fig. 2 (first line). moreover, these tech- @xcite facial image ✗ ✗ celeba-hq ✗ 4 hairclip @xcite facial image ✗ ✗ celeba-hq ✗ 2 † styleclip @xcite facial image ✗ ✗ celeba-hq ✗ 8 chatpainter @xcite realistic image ✗ ✗ ms coco dialogue -tigan @xcite facial image ✓ ✗ celeba-hq ✗ 8 talk-to-edit @xcite facial image ✓ ✓ celeba-dialog user utterance 5 chatedit @xcite facial image ✓ ✓ chatedit dialogue 21 †: hairclip only considers hairstyle and hair color. 44 text descriptions are collected for hairstyle and 12 text descriptions for hair color. niques don't fully harness interactivity and user experience. for example, @xcite @xcite solely process user inputs without offering natural language feedback, while others @xcite rely on rigid, hand-crafted response templates, limiting flexibility and naturalness. to facilitate the research on multi-turn interactive facial image editing, we introduce a novel benchmark dataset named chatedit. sourced from the celeba-hq dataset @xcite , chatedit enhances a selected set of 12k images with annotated multi-turn dialogues that align with user edit requests for facial images. the annotations include user utterances, system responses, and the user's "belief state", which represents the user edit requests from the beginning of the dialogue to the current turn. fig. 1 illustrates that success on chatedit necessitates the system to track the user edit requests, edit images based on tracked requests, and provide natural language responses to engage with users. to evaluate the performance of multi-turn interactive editing, we define three tasks: (i) user edit request tracking, (ii) image editing, and (iii) response generation. correspondingly, we introduce a comprehensive set of metrics that evaluate both response and editing quality. based on our benchmark dataset, we propose a baseline framework for multi-turn interactive image editing. our framework seamlessly integrates a language dialogue module and a visual image editing module. specifically, we employ an end-to-end task-oriented dialogue (tod) model to extract the user's image edit requests and generate appropriate responses based on the current conversation context. these tracked user requests are transformed into a text prompt to guide the text-based image editing model, styleclip @xcite , in performing the desired manipulations on the input raw image. as illustrated in fig. 2 's second line, our benchmark dataset and the proposed framework enable direct editing on the input raw image instead of cascaded modifications to images from previous turns, thereby reducing error accumulation and attribute forgetting issues. we perform extensive experiments with diverse settings to investigate the effectiveness of our proposed framework on the chatedit dataset. the results suggest that our proposed framework is superior to the previously prevalent cascaded single-turn editing methods regarding both image editing quality and response diversity. to sum up, our contributions are three-fold: (1) we introduce the chatedit benchmark dataset, which could serve as a valuable resource for advancing research in multi-turn interactive facial image editing. (2) we propose a novel framework that seamlessly combines a task-oriented dialogue module and an image editing module. this framework effectively tracks user requests, performs image editing, and generates system responses. importantly, it addresses the issues of attribute forgetting and error accumulation prevalent in previous methods. (3) through qualitative and quantitative evaluations on the chatedit dataset, we demonstrate the superiority of our proposed multi-turn editing framework over previous cascaded single-turn editing methods. we believe these results not only highlight the strengths of our proposed approach but also spur further exploration in this field. ## related work facial image editing traditional image editing techniques have focused on modifying specific given attributes such as age @xcite , hairstyle @xcite , or other predefined attributes @xcite @xcite . in recent years, with the development of pre-trained vision-language models @xcite @xcite , there has been a growing interest in humancomputer interaction scenarios @xcite . this leads researchers to explore interactive image editing, where users can dynamically adjust their editing requests through interaction with the system @xcite @xcite @xcite . for instance, tigan @xcite generates images iteratively based on successive editing steps during a conversation. however, these methods merely "listen" to user requests without generating system responses, constraining their interaction capability. a recent work called talk-to-edit @xcite attempts to address this by introducing a rule-based method to generate system response. however, this approach lacks flexibility and struggles with unforeseen scenarios that are not predefined. furthermore, these methods treat multi-turn editing as a sequence of individual single-turn edits, resulting in issues such as error accumulation and attribute forgetting as the number of interactions increases. ## chatedit benchmark dataset chatedit is designed to simulate a scenario where a user interacts with a system to manipulate a facial image. each sample in the chatedit dataset comprises a facial image and a dialogue between the user and the system regarding the editing of the image. notably, each turn of the dialogue is annotated with the user belief state, representing the user requests that guide both response generation and image editing. in this section, we will first introduce how to construct the dataset and its statistics. then, we introduce benchmark tasks of chatedit and corresponding evaluation metrics. ## method in this section, we introduce our proposed framework for chatedit. as illustrated in fig. 5 , the system consists of a task-oriented dialogue module and an image editing module. we adopt a pretrained language model-based tod model as the unified dialogue module for both user request tracking and response generation. specifically, it takes the dialogue context prepended with different prompts as input and outputs the tracked user requests and corresponding system response. as for the image editing module, a text-based image editing model styleclip @xcite is utilized, which receives the user requests tracked by the dialogue model as input and edits the input image accordingly. user requests hairstyle: bangs expression: no smiling system @xmath0 ## conclusion this paper introduces the chatedit benchmark dataset, which we believe could facilitate the research on multi-turn interactive facial image editing. the dataset poses significant challenges as it requires systems to accurately track user requests from dialogues, perform image editing based on these requests, and generate appropriate responses. we propose a baseline framework that seamlessly combines a task-oriented dialogue module and an image editing module. the introduction of the taskoriented dialogue not only enables interaction with users but also extracts concise user requests from the dialogue context to direct the image editing, avoiding the attribute forgetting and error accumulation issues in previous single-turn methods. the empirical results highlight the efficiency of our approach and the potential for further advancements in this exciting research area.