F a D - VLP : Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning (2022)

Abstract: Multimodal tasks in the fashion domain have significant potential for e-commerce, but involve challenging vision-and-language learning problems—e.g., retrieving a fashion item given a reference image plus text feedback from a user. Prior works on multimodal fashion tasks have either been limited by the data in individual benchmarks, or have leveraged generic vision-and-language pre-training but have not taken advantage of the characteristics of fashion data. Additionally, these works have mainly been restricted to multimodal understanding tasks. To address these gaps, we make two key contributions. First, we propose a novel fashion-specific pre-training framework based on weakly-supervised triplets constructed from fashion image-text pairs. We show the triplet-based tasks are an effective addition to standard multimodal pre-training tasks. Second, we propose a flexible decoder-based model architecture capable of both fashion retrieval and captioning tasks. Together, our model design and pre-training approach are competitive on a diverse set of fashion tasks, including cross-modal retrieval, image retrieval with text feedback, image captioning, relative image captioning, and multimodal categorization.

## introduction artificial intelligence has taken the fashion industry by storm in recent years. significant advances have been made in tasks like recommendation @xcite and virtual try-on @xcite . in addition to these primarily visual tasks, multimodal tasks are of particular interest in fashion for e-commerce applications: for example, text-to-image retrieval enables a shopper to identify a desired clothing item via a language query @xcite . a key opportunity to enhance customers' shopping experiences is in the development of interactive multimodal shopping assistants, whereby a user could converse with a system to identify a desired product @xcite . as in figure 1 , a smart assistant is desired to perform multiple diverse tasks, e.g., cross-modal retrieval, image retrieval with text feedback, multimodal categorization, image captioning, and relative image captioning. among them, perhaps the most notable task in fashion is image retrieval with text feedback, where the goal is to retrieve a target image given a reference image coupled with a user's language feedback (e.g., "show me a similar shirt in light blue with no print") @xcite @xcite . in addition to retrieval-based tasks, a central capability of conversational shopping assistants is in captioning-based tasks: describing items in detail @xcite or the differences among them. however, existing works on image retrieval with text feedback have almost exclusively studied that task in isolation, focusing on specialized architectures and fusion methods, with data limited by particular benchmarks @xcite . to train a model that can perform well on several fashion-specific multimodal use cases, we observe an opportunity in the vast availability of multimodal fashion data on e-commerce platforms. while vision-language pre-trained (vlp) models have been highly successful for the general domain @xcite @xcite , prior work has suggested that general vlp models are helpful but suboptimal for the fashion domain @xcite @xcite . fashion images represent a domain shift from the pre-training data @xcite , and fashion tasks often require fine-grained representations rather than coarse representations from general vlp models @xcite . to this end, we propose a domain-specific fashion pre-training procedure that takes advantage of fashion image-text data from multiple fashion catalogues. our approach is inspired by the way that users might shop, via comparisons: a user may first identify a product, express a desired change in language, and then look for a new product that better matches their preferences. given that data in this triplet form-reference product, modification, target product-is not nearly as common as the paired image-text data, we propose a lightweight method for constructing weakly-supervised pseudo-triplet data from image-text pairs. additionally, we propose a unified, decoder-based model architecture for both retrieval-based and captioning-based fashion tasks. together, we refer to our architecture and pre-training approach as fad-vlp: fashion decoder with vision-and-language pre-training. to summarize, we make the following contributions. we propose a unified architecture for retrieval-based and captioning-based fashion tasks (section 3.1) and a fashion pre-training framework, including 2 novel pre-training tasks based on weakly-supervised pseudo-triplets (section 3.2). our approach achieves competitive performance on 7 downstream fashion tasks: image-to-text retrieval, text-to-image retrieval, image retrieval with text feedback, category recognition, subcategory recognition, image captioning, and relative image captioning (sections 4 and 5.1). finally, we conduct a thorough ablation study to analyze the effects of our pre-training procedure (section 5.2). ## related work a substantial body of work has focused on using the transformer architecture @xcite in the context of vision-and-language pre-training (vlp) @xcite @xcite @xcite . recent works have begun to focus on the fashion domain @xcite @xcite @xcite @xcite . vlp works generally differ in their choice of model architecture and pre-training objectives. model architecture. most existing vlp models, especially in the fashion domain, use encoderstyle modules for both image and text, focusing on multimodal understanding tasks (which do not involve generation-e.g., image-text retrieval, multimodal classification). there are two main classes of these models: (i) single-stream early fusion @xcite @xcite , and (ii) two-stream late fusion @xcite @xcite . the nature of the downstream tasks often influences the choice of number of streams; e.g., image-text retrieval is most practical with late fusion architectures which can have faster inference. in this work, we propose a flexible decoder-based model architecture, which embraces the advantage of both early and late fusion mechanisms, and is capable of not only multimodal understanding tasks, but also captioning tasks (e.g., image captioning and relative image captioning). pre-training objectives. several pre-training tasks have been effectively used for vlp. some of the most popular include masked modeling or matching objectives for the different modalities @xcite @xcite ; others include cross-modal contrastive learning @xcite @xcite , caption generation @xcite , and object tagging @xcite . fashion data has some unique properties that could be leveraged at pre-training, partly to mitigate the domain shift which makes generic vlp less effective for fashion @xcite . for example, there are more structured attributes in fashion captions, which entitles people to naturally do comparisons when choosing their desired shopping items. inspired by this, we propose that weak triplet-based comparison is used as the basis for additional pre-training tasks. text decoder vision encoder multimodal decoder contrastive caption image caption text decoder vision encoder multimodal decoder relative caption reference image relative caption vision encoder target image text decoder vision encoder multimodal decoder relative caption vision encoder (a) aligner / captioner (b) relative captioner (c) fuser contrastive reference image target image figure 2: our proposed fad-vlp architecture consists of an image encoder, a text decoder, and a multimodal decoder, with three configurations that conform to various retrieval and captioning tasks. shared colors indicate shared parameters, curved arrows represent cross attention, and tokens with a bold border denote pooled representations. ## method we introduce fad-vlp, our architecture and pretraining method for fashion tasks. we first detail our architecture design (figure 2 ), which unifies several retrieval and captioning settings. we then describe our pre-training approach. ## experiments in this section, we provide the details of our pretraining dataset and downstream fashion tasks. implementation details are given in appendix a. ## results & discussion this section describes our results on our downstream tasks and an ablation study. ## conclusion our work introduces a novel fashion-specific pretraining framework based on weakly-supervised triplets, constructed from paired fashion image-text data, and flexible decoder-based model architecture capable of both retrieval and captioning tasks in fashion. our approach outperforms baselines on a diverse set of fashion tasks and highlights the value of fashion-specific pre-training, as well as the promise of triplet-based pre-training.