Utilizing Cross-Modal Contrastive Learning to Improve Item Categorization BERT Model (2022)

Abstract: Item categorization (IC) is a core natural language processing (NLP) task in e-commerce. As a special text classification task, fine-tuning pre-trained models, e.g., BERT, has become a mainstream solution. To improve IC performance further, other product metadata, e.g., product images, have been used. Although multimodal IC (MIC) systems show higher performance, expanding from processing text to more resource-demanding images brings large engineering impacts and hinders the deployment of such dual-input MIC systems. In this paper, we proposed a new way of using product images to improve text-only IC model: leveraging cross-modal signals between products’ titles and associated images to adapt BERT models in a self-supervised learning (SSL) way. Our experiments on the three genres in the public Amazon product dataset show that the proposed method generates improved prediction accuracy and macro-F1 values than simply using the original BERT. Moreover, the proposed method is able to keep using existing text-only IC inference implementation and shows a resource advantage than the deployment of a dual-input MIC system.

## introduction item categorization (ic) is a core natural language processing (nlp) technology in e-commerce. since millions types of products are provided in e-commerce markets, it is important to map these products to their locations in a product category tree efficiently and accurately so that buyers can easily find their interested products. therefore, ic models with a high accuracy are needed for the success of e-commerce business. in spite that ic shares the same setup as a text classification task, it possesses its unique aspects, including (a) handling a large number of prediction labels, (b) a severe long-tailed distribution of labels, and (c) noisy raw * equal contributor inputs due to the fact that these inputs are generally provided by merchants in a heterogeneous way. these unique aspects make ic be a challenging task in practice. fine-tuning pre-trained models, e.g., bert @xcite , has become a main stream approach on building high-performance nlp applications. when using this paradigm to build ic models, is there any way to achieve an even higher performance? this is the first research question we tackled in this paper. one approach to improving ic models that generally use products' text titles alone is utilizing products' images. previously, multimodal ic models using both text and image inputs have been actively investigated and applied in practice. however, such dual-input multimodal ic models bring more burden to the operation. comparing to handling text data, the resource needed for storing/transferring/processing image data is much higher. for industry-scaled ic systems, adding image processing in its inference stage is costly. is there any other way to get benefits by utilizing products' associated images but not paying such a high cost? this is the second research question we focused in this paper. to tackle these two questions, inspired by @xcite , we propose a solution of running a cross-modal contrastive learning, a special self-supervised learning (ssl), between products' images and text titles, to adapt pre-trained models to fit the ic task domain better. then, the adapted pre-trained models will be used to build ic models using the fine-tuning paradigm. moreover, by using the cross-modal ssl training, images can be used to improve text-based pretrained models in the model training stage and a series of costly changes/operations in the inference stage can be avoided. ## related works fine-tuning pre-trained models has been becoming a main stream method for building high quality ic systems. for example, in a recent data challenge for building multimodal ic systems, which was organized in the sigir'20 e-commerce workshop 1 , fine-tuning bert models @xcite has been used by most of the participants @xcite @xcite . in order to improve ic performance, one direction has been exploring utilizing more metadata associated with products. @xcite is a seminal work where multi-label classification using both titles and images was conducted on products listed on the walmart.com website. they used a convolutional neural network (cnn) to extract representations from both titles and images, and designed several policies to fuse the outputs of the two models. this led to improved performance over individual models separately. in the sigir'20 multimodal ic data challenge, different text-image fusion methods have been explored. roughly in order of increasing complexity, the methods included simple decision-level late fusion @xcite , highway network @xcite , and co-attention @xcite . by reviewing the experiment results from several teams, we can observe that (a) dual-input ic models show higher performance than any uni-modal ic model, (b) performance gains brought by using images are limited but not neglectable. when considering ic performance's profound impacts on e-commerce business values, including products' visual information is necessary. contrastive learning (cl) has been found to be an effective self-supervised learning (ssl) approach for training high-quality representations. for example, in computer vision, simclr @xcite uses the consistence between an anchor image and its transformed version and the in-consistence between the anchor and other instances in a batch (negative instances) to guide learning visual representations. without using labels, it achieves visual representations with a quality on par or even higher than the ones trained based on traditional supervised learning. inspired by the success of simclr in computer vision, cl-based text representation learning has been a hot research 1 https://sigir-ecom.github.io/ ecom2020/data-task.html topic in nlp. both simcse @xcite and @xcite used dropout operations existing in transformer architecture @xcite to be an effective text augmentation way and obtained effective text representations. the ssl idea has been tried in the nlp domain for further improving bert models. for example, @xcite proposed using ssl to improve the pre-trained bert model prior to running down-stream nlp tasks, e.g., various tasks in the glue benchmark test. when generating augmented instances for providing positive pairs, a back-translation method is used. the cl setup was based on moco architecture by using a momentum mechanism @xcite . @xcite also used the ssl to improve pre-trained bert model for more accurate relation extraction (re) task. when doing text augmentation, it considered the re task's unique property and proposed a taskspecific augmentation method. in these two works, ssl training was found to be useful for improving follow-up fine-tuning tasks' performance. regarding the ssl methods being used, an interesting trend is considering cross-modal signals. @xcite proposed contrastive visual representation learning from text (convirt) model to use text descriptions associated with medical images to help training more accurate medical image representations. note that in the medical image domain, the annotated image data size is much limited, in a contrast, medical text notes are more adequate. @xcite is a seminal work from openai. by using a massive set of image-text pairs, about 350 million, clip used a simple cross-modal contrastive learning to pretrain a quite powerful vision-language joint model. the trained model shows many impressive applications, like superior performance on many zero-shot image classification tasks. note that we only show contrastive pairs, i.e., both positive and negative, to the top product "red bull energy drink". for a text title x t , we obtain a text representation t with a bert encoder f t (, d) where d is a dropout mask, and a projection function g t , which uses a simple multiple layer perception (mlp) structure. to obtain a positive pair, simcse just runs the same text title throughout the transformer encoder pipeline with a different dropout mask d + . for ith text title, the training objective of sim-cse is like: for a mini-batch of n text titles, where sim() represents a similarity computation and τ is a temperature parameter. the total loss computed by simcse is an average among all text titles as shown in figure 2 , the top title serves as an anchor and is sent to a bert model twice to obtain two similar but varying text representations because of using varying dropout masks. the augmented version serves as a positive pair to the anchor while other two titles in the batch serve as negative pairs. through using an infonce loss @xcite , bert encoder is changed to pull positive pairs closer while pushing negative pairs away. clearly, without any supervision, bert encoder can be further adapted to provide a representation better fitting to the ssl data set. ## discussion with the success of pre-trained models like bert @xcite on many nlp tasks, fine-tuning bert models has become an leading approach. to further improve fine-tuned ic model performance, inspired by related works @xcite , self-supervised learning (ssl) is used to adapt original bert models to better match with the ic task. regarding the ssl method, we compared two approaches. the first approach is unsupervised simcse @xcite that only uses text titles. the simcse provided a simple but effective way to generate semantically similar pairs (positive) by feeding the same product title into a bert model with varying dropout masks. using other titles in the mini-batch to be negative pairs, the bert model can be adjusted by minimizing the infonce loss @xcite . to improve ssl further, we proposed using the cross-modal contrastive learning to utilize product images to bring additional modeling power to improve our text representations. following @xcite , for a product title, we use its associated product image to be a positive pair while other products' images to be negative pairs. the same cross-modal computation was also applied on a product image to provide contrastive signal from the other direction. our experiments on the three genres in the amazon review dataset show that both ssl enriched bert models have higher fine-tuning performance. between the two ssl methods, the ssl using con-virt method is more effective. moreover, this new way of utilizing images means that we only need process images during the model training stage and can only deploy text-only bert (enriched by con-virt ssl training) in our inference stage. this will dramatically reduce engineering and computation costs compared to the method of deploying a dual-input mic systems. one limitation of our research is that we only explored the simcse method when using unlabeled text data. it is possible that other semisupervised learning methods like uda @xcite may help on improving the final fine-tuning performance. we will leave the exploration on more semi-supervised learning methods in future. in addition, there are several directions to extend the current work in the future, including (1) improving contrastive learning based ssl, for example using nearest neighbor to get better positive pairs similar to @xcite , and (2) improving our algorithms to better address training set bias such as high label noise and very imbalanced data set in real ic data.