F ashion KLIP : Enhancing E -Commerce Image-Text Retrieval with Fashion Multi-Modal Conceptual Knowledge Graph (2023)

Abstract: Image-text retrieval is a core task in the multi-modal domain, which arises a lot of attention from both research and industry communities. Recently, the booming of visual-language pre-trained (VLP) models has greatly enhanced the performance of cross-modal retrieval. However, the fine-grained interactions between objects from different modalities are far from well-established. This issue becomes more severe in the e-commerce domain, which lacks sufficient training data and fine-grained cross-modal knowledge. To alleviate the problem, this paper proposes a novel e-commerce knowledge-enhanced VLP model FashionKLIP. We first automatically establish a multi-modal conceptual knowledge graph from large-scale e-commerce image-text data, and then inject the prior knowledge into the VLP model to align across modalities at the conceptual level. The experiments conducted on a public benchmark dataset demonstrate that FashionKLIP effectively enhances the performance of e-commerce image-text retrieval upon state-of-the-art VLP models by a large margin. The application of the method in real industrial scenarios also proves the feasibility and efficiency of FashionKLIP.

## introduction the explosive growth of multi-modal content on the web has promoted the research of various crossmodal tasks. image-text retrieval, which finds correlated texts (or images) for a given image (or text) @xcite , is a popular cross-modal task with strong practical values in a wide range of industrial applications. recently, the booming of vision-language 1 all the codes and model checkpoints have been released to public in the easynlp framework @xcite . url: https://github.com/alibaba/easynlp . however, in the field of e-commerce, the imagetext retrieval task has its own challenges. here, we suggest that image-text pairs of products have unique characteristics that are different from the general domain (such as ms-coco @xcite , flickr30k @xcite and conceptual captions @xcite ), with examples shown in figure 1 . 1) while most texts in the general domain contain descriptions with complete sentence structures, descriptions or queries in e-commerce are usually composed of multiple phrases, describing product details such as materials or styles. 2) images in the general domain usually have rich backgrounds; in contrast, a product image mainly consists of a large commodity figure in the center without a lot of background objects. these unique domain characteristics make generaldomain models difficult to be directly adopted to the image-text retrieval tasks in e-commerce. recently, several domain-specific vlp models including fashionbert @xcite , kalei-dobert @xcite , commercemm @xcite , ei-clip @xcite and fashion-vil @xcite are proposed based on e-commence image-text pairs, which greatly improve the performance of e-commerce imagetext retrieval. despite the success, the fine-grained cross-modal alignment issue remains unsolved, which may result in the inaccurate matching of details between images and texts. although some e-commerce vlp models use fine-grained information from either image perspectives @xcite or patch-based image classification @xcite , they are short of semantic-level alignments across modalities. some other work @xcite focuses on entities in text modalities, but rarely considers cross-modal interactions. in the general domain, fine-grained interactions could be achieved with object detection @xcite , scene graph parsing @xcite , or semantic analysis @xcite . unfortunately, these tools lose their effectiveness in the e-commerce domain. to improve the fine-grained alignment between images and texts in e-commerce, this paper proposes an e-commerce knowledge-enhanced vlp model -fashionklip. particularly, we first propose a data-driven strategy to construct a multimodal conceptual knowledge graph in e-commerce (called fashionmmkg) from a large-scale ecommerce image-text corpus, where the fashion concepts are automatically extracted and organized in the form of a semantic hierarchy, each associated with its representative images. the fashion-mmkg is later incorporated as the prior crossmodal fashion knowledge in training a clip-style model to support e-commerce image-text retrieval. for model training, we learn the representation alignment of image-text pairs across the two modalities by contrastive learning, and further optimize the alignment at the conceptual level. the conceptual alignment is further obtained by matching the text representations with the visual prototype representations of the fashion concepts in fashion-mmkg. our contributions can be summarized as follows: vision-language pre-training. vlp models can be categorized into single-stream models @xcite @xcite , which first concatenate multi-modal inputs for interactions, and dual-stream models @xcite @xcite , which obtain the representations of the image and text respectively and learn the alignment afterwards. although single-stream models may lead to high retrieval accuracy due to the early fusion of images and texts, the inference efficiency is sacrificed to a certain extend. recently, to focus more on fine-grained semantic level interactions of images and texts, some works improve the similarity strategy by calculating between the image patch and the text token @xcite or leverage fine-grained image information through object detectors @xcite @xcite . others introduces structured scene graphs for semantic knowledge @xcite . despite their success in general domain, such methods are hard to be adopted to e-commerce data. fashion-based retrieval. fashionbert @xcite ) first adopts pre-training tasks such as masking strategy to e-commerce images and texts. kaleidobert @xcite extracts a series of multi-grained image patches for augmentation to guide masking strategy for fine-grained matching. commercemm @xcite proposes pre-training tasks to align uni-modal with multi-modal features for more consistent alignment. ei-clip @xcite defines the entityaware retrieval task from the linguistic perspective by introducing a causal model to concatenate different meta-data as e-commerce entities. lately, fashion-vil @xcite designs a flexible architecture for various downstream tasks. however, current methods still suffer from insufficient fine-grained semantic alignment, which may diminish the cross-modal understanding capability of models at semantic level. ## methodology this section introduces how fashionmmkg is constructed and how fashionklip incorporates conceptual-level interactions of cross-modal fashion knowledge from fashionmmkg. ## experiments we conduct comprehensive evaluations on fash-iongen @xcite to show that fashionklip outperforms sota methods. ## industrial application in this section, we verify the effectiveness of fash-ionklip on our alibaba global e-commerce platform. specifically, we apply it to product search with two specific retrieval tasks including image-toproduct (i2p) and text-to-product (t2p) retrieval, as shown in figure 5 . for t2p, we employ a weighted scoring function to compute the similarity score between a query text and a product (with a title and an image) as follows: score @xmath0, where 0 < α < 1, score t2t and score t2i refer to the embedding similarity score between the query text and the product title, together with the query text and the product image. similarly, for i2p, we have score @xmath1. in total, the collected dataset contains 58,463 products (with images and titles) and 3,021 queries. ## conclusion and future work this paper proposes a novel data-driven approach to construct a multi-modal conceptual knowledge graph in e-commerce namely fashionmmkg. an e-commerce knowledge-enhanced vlp model namely fashionklip is then constructed by learning the conceptual-level alignments from the prior knowledge in fashionmmkg. our empirical study shows that fashionklip outperforms state-of-theart vlp models in the e-commerce domain. we conduct experiments under industrial scenarios and verify its practical value in real-world applications and confirm the efficiency of fashionklip. in the future, we will apply the knowledge-enhanced strategy for general large-scale pre-training and bring benefit to more multi-modal tasks.