DU - VLG : Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training (2022)

Abstract: Due to the limitations of the model structure and pre-training objectives, existing vision-and-language generation models cannot utilize pair-wise images and text through bi-directional generation. In this paper, we propose DU-VLG, a framework which unifies vision-and-language generation as sequence generation problems. DU-VLG is trained with novel dual pre-training tasks: multi-modal denoising autoencoder tasks and modality translation tasks. To bridge the gap between image understanding and generation, we further design a novel commitment loss. We compare pre-training objectives on image captioning and text-to-image generation datasets. Results show that DU-VLG yields better performance than variants trained with uni-directional generation objectives or the variant without the commitment loss. We also obtain higher scores compared to previous state-of-the-art systems on three vision-and-language generation tasks. In addition, human judges further confirm that our model generates real and relevant images as well as faithful and informative captions.

## introduction pre-trained models for vision-and-language tasks have made remarkable progress recently @xcite @xcite . existing pre-trained models either focus on text-to-image synthesis or image-to-text generation @xcite . these models are often pre-trained with image-text pairs which are aligned in semantics. however, due to the limitations of model structure, existing models cannot be adapted to each other. in addition, pre-training objectives are designed either for text generation conditioned on the image or image generation conditioned on the text, limiting the model to learn better semantic alignment from bi-directional generation @xcite . we argue that image-to-text and text-to-image generation appear as dual tasks, which both require strong visual and textual representations aligned in the same semantic space. images and text descriptions are of different information quantity and density. the images often contain more information, but are with heavy redundancy, while text descriptions are semantically condensed, but may neglect details. uni-directional generation paradigm may induce the model to amplify this property. take fig. 1 as an example, the unidirectional model may fail in capturing details. inspired by this observation, we propose to utilize bi-directional generation objectives to learn better generalization of image and text representations. to this end, we present du-vlg, a framework with dual sequence-to-sequence pre-training for vision-and-language generation. under the encoder-decoder transformer framework, our model takes text and raw images as inputs and gen-erate text and images autoregressively. concretely, images are represented as continuous patch features in the encoder and discrete visual tokens in the decoder. with the hybrid image embedding schema, du-vlg is able to unify vision-and-language generation in a single model. in order to utilize dualities of image-text pairs, we further propose two pairs of dual pre-training tasks: multi-modal denoising autoencoder task and modality translation task. for the multi-modal denoising autoencoder task, our model takes imagetext pairs with some image patches or words randomly masked as inputs and learns image-text alignment through reconstruction of the corrupted modality. for modality translation tasks, we form image captioning and text-to-image generation as dual pre-training tasks, which further enhance model ability of semantic alignment. different from existing multi-modal pre-trained models, our model learns image-text alignment through bidirectional generation objectives. moreover, we propose a novel commitment loss to drive the model to acquire better image representation. concretely, the commitment loss is designed to connect visual embeddings in the decoder to patch-based features in the encoder. in tandem with our model design, the commitment loss aims to unify image understanding and generation in a single model, which allows for better utilization of bi-directional generation objectives. we conduct experiments on various vision-andlanguage generation tasks. we first study effects of dual pre-training tasks and the commitment loss. on both image captioning and text-to-image generation tasks, du-vlg outperforms its variant without commitment loss or the variants that only learns uni-directional generation objectives. for image captioning, we achieve better bleu-4 and cider than existing pre-trained models on coco dataset @xcite . for text-to-image generation, our model achieves better results than both transformer-based and gan-based methods on both coco and cub dataset @xcite . human judges confirm that our model generates captions and images with high-quality. importantly, we test our model on a challenging vision-and-language generation task: visual commonsense reasoning @xcite . results demonstrate that our model is able to handle challenging multi-modal generation tasks effectively. the main contributions of du-vlg are as fol-lows: the rest of the paper is organized as follows. we describe our model in § 2 and introduce our proposed pre-training task and commitment loss in § 3. training details are presented in § 4. in § 5, we discuss experimental results. related work is listed in § 6 and we finally draw our conclusion in § 7. ## model in this section, we describe our proposed model. overall, our model design is mainly inspired by two observations: (1) sharing parameters that play the same role boosts model performance @xcite and (2) image understanding and generation require representing image features in different granularity @xcite . hence, we use a standard transformer with the encoder-decoder structure @xcite , as illustrated in fig. 2 . our model takes images and text as inputs and treats image and text generation as sequence generation problems. importantly, we propose to use a hybrid image embedding schema in the encoder and the decoder. ## dual pre-training tasks and pre-training objectives next, we introduce our pre-training method. pretraining corpus consists of millions of aligned image-text pairs. in order to effectively learn vision-and-language understanding and generation, we propose dual pre-training tasks. dual pretraining tasks drive the model to learn from reconstruction of the image or text description based on given context. we propose two pairs of pre-training tasks: (1) multi-modal denoising autoencoder task ( § 3.1) and ( 2 ) modality translation task ( § 3.2), as shown in fig. 3 . in § 3.3, we formulate a commitment loss to connect image understanding and generation. ## results in this section, we start with comparing our proposed pre-training objectives in § 5.1. we then conduct automatic evaluation on three vision-andlanguage generation tasks ( § 5.2) and further report human evaluation on both caption and synthesized ## conclusion we presented a novel framework, du-vlg, which unifies vision-and-language generation tasks with an encoder-decoder transformer. we propose to use a hybrid image embedding schema in the encoder and decoder. in addition, we pre-train the model with novel dual pre-training tasks, along with a new commitment loss, to guide better image and text understanding and generation. experiments show that our proposed dual pre-training objectives significantly improve performance on three vision-and-language generation tasks. human evaluation further confirms that our model with dual pre-training tasks improves generation quality on image captioning and text-to-image generation.