Style-Aware Contrastive Learning for Multi-Style Image Captioning (2023)

Abstract: Existing multi-style image captioning methods show promising results in generating a caption with accurate visual content and desired linguistic style. However, existing methods overlook the relationship between linguistic style and visual content. To overcome this drawback, we propose style-aware contrastive learning for multi-style image captioning. First, we present a style-aware visual encoder with contrastive learning to mine potential visual content relevant to style. Moreover, we propose a style-aware triplet contrast objective to distinguish whether the image, style and caption matched. To provide positive and negative samples for contrastive learning, we present three retrieval schemes: object-based retrieval, RoI-based retrieval and triplet-based retrieval, and design a dynamic trade-off function to calculate retrieval scores. Experimental results demonstrate that our approach achieves state-of-the-art performance. In addition, we conduct an extensive analysis to verify the effectiveness of our method.

## introduction stylized image captioning aims to generate a natural language description with stylized elements for a given image @xcite . with the advance of deep learning in humancomputer interaction equipment, it has been integrated into many real-world applications like education robots @xcite , visual dialog @xcite , and vision-language navigation @xcite . therefore, it has attracted more attention from academia and industry and has become one of the essential areas in the natural language processing (nlp) and computer vision (cv) community. however, many methods propose translating images into captions of a single caption style @xcite . these methods need to train multiple models to handle multiple styles, which is very inefficient. therefore, a rising demand for stylized image captioning is to learn an efficient model to handle multiple styles simultaneously. recently, many efforts have been made in multi-style image captioning @xcite @xcite . despite their success, existing methods suffer from a drawback: they focus on accurate visual content and desired linguistic style but overlook the relation between linguistic style and visual content. as shown in figure 1 (a), the generated caption contains accurate visual content and the desired style, but with misinformation that is easily detectable by humans, i.e., model generates "beautiful" in the caption to cater to the style of "happy". indeed, linguistic style @xcite reflects personality, emotion and sentiment. in human behavior, these factors significantly influence the course of cognitive behavior @xcite . when people with different emotions see the same image are likely to describe different contents because they pay attention to different aspects. for example, people who are happy with animals may describe an image of a dog as a pretty dog with sparkling eyes and supple hair. however, one afraid of dogs may describe it as a scary dog with fierce teeth and sharp claws. the former focus on the dog's eyes and hair, while the latter on the dog's teeth and paws. therefore, people with different emotions focus on different potential visual content. to generate human-like stylized captions, a stylized captioning model should learn to mine potential visual content relevant to different linguistic styles. due to the success of contrastive learning @xcite , some works (e.g., uniter @xcite , vilt @xcite ) employ contrastive learning objectives to encourage cross-modality alignment. in this work, we propose a style-aware visual encoder with contrastive learning to mine potential visual content relevant to styles. specifically, style-aware visual encoder first mines potential visual features based on given image and style pairs. then, we use a contrastive loss to pull potential visual features of the anchor and positive pair together while pushing those of anchor and negative pairs apart. in addition, apart from requiring generated captions with accurate visual content and desired linguistic style, multi-style image captioning also needs to ensure that potential visual content and style are relevant. moreover, since multi-style image captioning contains more fine-grained styles than single-stylized image captioning, it is difficult to directly distinguish the style by the caption. as shown in figure 1 (b) and (c), captions may be the same for two different styles. therefore, multi-style image captioning is required to consider if matching among image, style and caption. different from previous works @xcite that optimize generated captions only based on style, we propose a style-aware triplet contrast loss, which can learn the triplet matching among image, style, and caption by contrasting it with the positive triplet against negative ones. moreover, motivated by hard negatives sampling in retriever training @xcite , we present three retrieval schemes to mine positive and negative examples for contrastive learning: object-based retrieval, roi-based retrieval and triplet-based retrieval. these three schemes calculate scores according to object overlap rate, potential visual feature similarity, and triple feature similarity, respectively. meanwhile, we design a dynamic trade-off function to calculate retrieval scores and analyze the impact of different retrieval schemes. we conduct an extensive evaluation on three datasets, i.e., personality-captions @xcite , @xcite and flickrstyle10k @xcite . our method significantly outperforms other strong competitors and achieves state-of-the-art performance. ## method this section will elaborate on our style-aware contrastive learning (saco) method for multistyle image captioning, followed by our proposed novel retrieval schemes. the details of our approach are shown in figure 2 . lastly, details about training and fine-tuning are elaborated. ## conclusion in this work, we dive into the relationship between linguistic style and visual content. the first is potential visual content has varied for different styles. we propose a style-aware visual encoder that learns to capture the representation of potential visual content. second, since the model is required to distinguish whether the image, style and caption are matched, we present a style-aware triplet contrast objective to improve the model's capability to discriminate triplet matching. in addition, we propose three novel retrieval schemes to sample positive and negative examples for contrastive learning. results show that our method delivers new state-of-the-art performance.