Multimodal Item Categorization Fully Based on Transformer (2021)

Abstract: The Transformer has proven to be a powerful feature extraction method and has gained widespread adoption in natural language processing (NLP). In this paper we propose a multimodal item categorization (MIC) system solely based on the Transformer for both text and image processing. On a multimodal product data set collected from a Japanese e-commerce giant, we tested a new image classification model based on the Transformer and investigated different ways of fusing bi-modal information. Our experimental results on real industry data showed that the Transformer-based image classifier has performance on par with ResNet-based classifiers and is four times faster to train. Furthermore, a cross-modal attention layer was found to be critical for the MIC system to achieve performance gains over text-only and image-only models.

## introduction item categorization (ic) is a core technology in modern e-commerce. since there can be millions of products and hundreds of labels in e-commerce markets, it is important to be able to map these products to their locations in a product category taxonomy tree efficiently and accurately so that buyers can easily find the products they need. therefore, ic technology with high accuracy is needed to cope with this demanding task. products can contain text (such as titles) and images. although most ic research has focused on using text-based cues, images of products also contain useful information. for example, in some sub-areas like fashion, the information conveyed through images is richer and more accurate than through the text channel. in this paper, we propose an mic model entirely based on the transformer architecture @xcite for achieving * equal contributor a simplification of the model and faster training speed. we conducted experiments on real product data collected from an e-commerce giant in japan to (a) test the performance of the transformerbased product image classification, and (b) systematically compare several bi-modal fusion methods to jointly use both text and image cues. recently, a mic data challenge was organized in the sigir'20 e-commerce workshop foot_0 . rakuten france provided a dataset containing about 99k products where each product contained a title, an optional detailed description, and a product image. the mic task was to predict 27 category labels from four major genres: books, children, household, and entertainment. several teams submitted their mic systems @xcite @xcite . a common solution was to fine-tune pre-trained text and image encoders to serve as feature extractors, then use a bi-modal fusion mechanism to combine predictions. most teams used the transformer-based bert model @xcite for text feature extraction and resnet @xcite for image feature extraction, including the standard resnet-152 and the recently released big transfer (bit) model @xcite . for bi-modal fusion, the methods used were more diverse. roughly in order of increasing complexity, the methods included simple decision-level late fusion @xcite , highway network @xcite , and co-attention @xcite . it is interesting to note that the winning team used the simplest decision-level late fusion method. in other recent work, a cross-modal attention layer which used representations from different modalities to be the key and query vectors to compute attention weights was studied. in @xcite , product descriptions and images were jointly used to predict product attributes, e.g., color and size, and their values in an end-to-end fashion. in addition, based on the fact that product images can contain information not clearly aligned with or even contradicting the information conveyed in the text, a special gate was used to control the contribution of the image channel. a similar idea was used in @xcite on multimodal named entity recognition research on twitter data. although the field has converged on using transformer-based models for processing text in recent years, resnet-based image processing is still the dominant approach in mic research. one immediate difficulty in combining the two types of models is the big gap between the training speeds. owing to the superior parallel running capability enabled by self-attention in the transformer architecture, text encoder training is much faster than the image encoder, and the training bottleneck of the mic system becomes solely the image encoder. in addition, using two different deep learning architectures simultaneously makes building and maintaining mic systems more complex. one solution is to use transformers as the encoder of choice for both modalities. furthermore, a detailed comparison of different fusion methods on large-scale multimodal industry product data is still missing. our work addresses these two directions of research. ## model our mic model is depicted in figure 1 foot_1 . it consists of feature extraction components using a transformer on uni-modal channels (i.e., text titles and images), a fusion part to obtain multimodal representations, and a multi-layer perceptron (mlp) word_1, ... word_n 1 2 4 5 3 6 7 8 9 bert vit v_0 v_1 h_2 ... v_9 h_0 h_1 h_2 ... h_n cls p_1 p_2 ... p_9 cls w_1 w_2 ... w_n mlp mlp late fusion (decision) early fusion (shallow) mlp cross-modal attention mlp early fusion (cross-attention) (a) (b) (c) alpha 1-alpha category category category head to make final predictions. ## discussion although various approaches have been explored in mic research, we found that a mic system built entirely out of the transformer architecture was missing. combining the well-established bert text model and the newly released vit image model, we proposed an all-transformer mic system on japanese e-commerce products. from experiments on real industry product data from an e-commerce giant in japan, we found that the vit model can be fine-tuned four times faster than bit and can have improved performance. furthermore, fusing both text and image inputs in an mic setup using the cross-modal attention fusion method led to model performance better than each model separately, and we found that this fusion method worked better than late fusion and the early (shallow) fusion of simply concatenating representations from the two modalities. there are several directions to extend the current work in the future, including (1) considering jointly modeling texts and images in one transformer model like fashionbert @xcite , and (2) using self-training to go beyond the limit caused by the size of labeled image data for the image model.