Improving Cross-modal Alignment for Text-Guided Image Inpainting (2023)

Abstract: Text-guided image inpainting (TGII) aims to restore missing regions based on a given text in a damaged image. Existing methods are based on a strong vision encoder and a cross-modal fusion model to integrate cross-modal features. However, these methods allocate most of the computation to visual encoding, while light computation on modeling modality interactions. Moreover, they take cross-modal fusion for depth features, which ignores a fine-grained alignment between text and image. Recently, vision-language pre-trained models (VLPM), encapsulating rich cross-modal alignment knowledge, have advanced in most multimodal tasks. In this work, we propose a novel model for TGII by improving cross-modal alignment (CMA). CMA model consists of a VLPM as a vision-language encoder, an image generator and global-local discriminators. To explore cross-modal alignment knowledge for image restoration, we introduce cross-modal alignment distillation and in-sample distribution distillation. In addition, we employ adversarial training to enhance the model to fill the missing region in complicated structures effectively. Experiments are conducted on two popular vision-language datasets. Results show that our model achieves state-of-the-art performance compared with other strong competitors.

## introduction text-guided image inpainting (tgii), involving computer vision (cv) and natural language processing (nlp), aims to restore visual content for a missing area in a damaged image based on a given text @xcite . with the development of cv and nlp, it plays an essential role in many real-world applications, such as image editing @xcite , damaged image restoration @xcite , and image rendering @xcite . therefore, it has become one of the most crucial areas in cv and nlp. ## method in this section, we will introduce our cma model as shown in figure 2 . cma model consists of a vision-and-language encoder, an image generator and global-local discriminators. in addition, details about training and inference are elaborated. ## conclusion in this work, we explore a novel cma model for text-guided image inpainting. in the cma model, we integrate a vision encoder and a text encoder into a vision-and-language encoder, which is different from previous works. the vision-and-language encoder allocates more computation on modeling modality interactions instead of visual encoding. in addition, we introduce two objectives to improve cross-modal alignment, dubbed cross-modal alignment and in-sample distillation. the cross-modal alignment objective guides the model to fuse crossmodal features. experimental results demonstrate that the proposed model delivers new state-of-theart performance, followed by further analyses to provide comprehensive insights.