KG - FLIP : Knowledge-guided Fashion-domain Language-Image Pre-training for E -commerce (2023)

Abstract: Various Vision-Language Pre-training (VLP) models (e.g., CLIP, BLIP) have sprung up and dramatically advanced the benchmarks for public general-domain datasets (e.g., COCO, Flickr30k). Such models usually learn the cross-modal alignment from large-scale well-aligned image-text datasets without leveraging external knowledge. Adapting these models to downstream applications in specific domains like fashion requires fine-grained in-domain image-text corpus, which are usually less semantically aligned and in small scale that requires efficient pre-training strategies. In this paper, we propose a knowledge-guided fashion-domain language-image pre-training (FLIP) framework that focuses on learning fine-grained representations in e-commerce domain and utilizes external knowledge (i.e., product attribute schema), to improve the pre-training efficiency. Experiments demonstrate that FLIP outperforms previous state-of-the-art VLP models on Amazon data and on the Fashion-Gen dataset by large margins. FLIP has been successfully deployed in the Amazon catalog system to backfill missing attributes and improve the customer shopping experience.

## introduction modern e-commerce websites exhibit products with multi-modal information (e.g., product images, product titles, and product bullet points) to inform customers' purchase decisions. the effective exploitation of such multi-modal product information is crucial for product understanding and downstream vision-language (vl) applications, such as product categorization, search, and recommendation. meanwhile, recent large-scale visionlanguage pre-training (vlp) models have led to impressive performance improvements on many general-domain vl tasks @xcite . as a result, there has been a surge of * work done during internship at amazon. interest in adapting such vlp models to facilitate various applications in e-commerce scenarios. unlike the well-aligned coarse-grained languageimage datasets in the general domain, the paired data in the e-commerce domain have two characteristics. first, both of the product titles/descriptions and the images contain richly detailed (i.e., finegrained) product information compared to datasets in the general domain. second, the product textual information and images usually share only partial information while containing complementary information (i.e., not well-aligned). thus, an effective pre-training method needs to align the common portion and fuse the distinct facts from each modality in a fine-grained manner. rather than aligning the entire image and text pair at a global level using contrastive loss as clip @xcite does, we designed our pre-training tasks to focus on a finer level of text tokens and image patches. in addition, previous vlp methods relied solely on the inductive bias of the model to align crossmodality representations through vast amounts of paired data. such an approach is data-hungry, inefficient, and disregards the availability of structured product knowledge. thus, we propose to leverage existing knowledge in the e-commerce catalog to facilitate such alignment. specifically, for each type of product (e.g., dress), the catalog stores its applicable attributes (e.g., neckline style) and enumerated attribute values (e.g., v-neck, crewneck). such attribute knowledge can serve as anchor points to help vlp models efficiently acquire salient semantic relations between modalities. to address the above challenges, we propose kg-flip: a knowledge-guided fashion-domain language-image pre-training to improve the vlp models for e-commerce data. the design of kg-flip is inspired by the state-of-the-art generalpurpose vlp model blip @xcite . we adapt its design for our use case by 1) replacing the widelyused image-text contrastive (itc) objective with the masked language-image modeling (mlim) pretraining objective, to facilitate multi-model fusion at the token level instead of cross-model alignment; 2) leveraging the structured knowledge of product attribute schema information to guide the pretraining process, and facilitate the vlp model to learn more fine-grained product representations. these enhancements can be generalized to other real-world applications, where image-text pairs are not well-aligned in semantics and external knowledge can be leveraged to guide the pre-training. ## method this section delineates kg-flip. section 3.1 introduces the architecture of kg-flip. then, section 3.2 presents pre-training objectives of the model. finally, section 3.3 explains how we inject attribute knowledge into kg-flip. ## conclusion in this paper, we introduced a knowledge-guided fashion-domain language-image pre-training framework for e-commerece, dubbed kg-flip. by utilizing the product attribute knowledge to guide mlim and itm pre-training objectives, our kg-flip model facilitates the vision-language pre-training and enhances the product representation learning for e-commerce data that are partially aligned while also containing complementary information. the evaluation results have demonstrated its prominent performance against other state-of-the-art benchmarks on both amazon and fashion-gen datasets. the kg-flip model has been deployed in a real-world application and improved the customer shopping experience.