SMARTAVE : Structured Multimodal Transformer for Product Attribute Value Extraction (2022)

Abstract: Automatic product attribute value extraction refers to the task of identifying values of an attribute from the product information. Product attributes are essential in improving online shopping experience for customers. Most existing methods focus on extracting attribute values from product title and description.However, in many real-world applications, a product is usually represented by multiple modalities beyond title and description, such as product specifications, text and visual information from the product image, etc. In this paper, we propose SMARTAVE, a Structure Mltimodal trAnsformeR for producT Attribute Value Extraction, which jointly encodes the structured product information from multiple modalities. Specifically, in SMARTAVE encoder, we introduce hyper-tokens to represent the modality-level information, and local-tokens to represent the original text and visual inputs. Structured attention patterns are designed among the hyper-tokens and local-tokens for learning effective product representation. The attribute values are then extracted based on the learned embeddings. We conduct extensive experiments on two multimodal product datasets. Experimental results demonstrate the superior performance of the proposed approach over several state-of-the-art methods. Ablation studies validate the effectiveness of the structured attentions in modeling the multimodal product information.

## introduction product attributes are important features that carry useful information about the product. they form an essential component of e-commerce platforms, which provide guidance for customers to compare products and make purchasing decisions. product attributes also facilitate retailers on various applications, including product search @xcite , product recommendations @xcite , and question answering systems @xcite @xcite . however, as shown in previous studies @xcite , product attributes are often noisy and incomplete with a lot of missing values for most retailers. therefore, it is an important research problem to extract the product attributes with missing values. attribute value extraction has attracted a lot of attention from both academia and industry in recent years, with a plethora of research @xcite @xcite being proposed to tackle this problem. most existing works @xcite @xcite rely solely on the product title and description from the product profiles, which is often insufficient to obtain values for all attributes. in real-world applications, products are usually associated with rich information from other modalities, such as product specification and image. this additional information can improve the attribute value extraction in two main aspects. first, attribute values are sometimes absent in the product title and description. for example, as illustrated in figure 1 , the value '17.9 fl oz' corresponding to the attribute 'size' is only mentioned in the ocr text from the product image. the value 'liquid' of the attribute 'item form' is not mentioned in any text source of the product, but can be inferred from the product visual information. in these cases, the product image is able to recover the missing values of the attributes. second, product title and description might contain multiple candidate values for an attribute, while other modalities could consolidate the correct value. for example, while both 'hair food' and 'natural' from the product title are reasonable values for attribute 'brand', the ocr text in the image clearly suggests that 'hair food' is the correct answer. there are a few recent works @xcite that incorporate the product image for attribute value extraction, which achieve promising results. however, these techniques suffer from two major limitations. first, each modality of the product is encoded independently with an individual encoder, followed by a light fusion/crossmodality layer on the top. in this way, the correlations among different modalities are not fully captured, leading to less effective embeddings. second, the texts from individual modalities are simply concatenated and fed into a single encoder, making these methods inefficient to handle long input. to address these challenges, in this paper, we propose a novel structured multimodal transformer for product attribute value extraction (smartave), which jointly encodes the structured product information from multiple modalities in a unified transformer model. we first introduce a set of hyper-tokens to represent all modalities of the product, and use local-tokens to indicate the original text and visual inputs within each modality. structured attention patterns are designed among the hyper-tokens and local-tokens for modeling the correlation between modalities and learning effective product representation. intuitively, the hyper-tokens serve as 'hubs' routing local-tokens from different modalities to interact with each other. the attribute values are then extracted based on the learned embeddings from the structured encoder. evaluations on two multimodal product datasets show the superior performance of our model over several state-of-the-art methods. the experimental results also demonstrate the effectiveness of the structured attention mechanism in modeling multimodal product data with large input sequences. we summarize the main contributions as follows: transformer that extracts product attribute values from product title, description, specification, visual information, and texts in the prod-uct image. title description attribute spec1 specm ocrn image brand hair food ... argan oil feed your ... you don't need. recipes, not ... mineral oil free to feed frizzy ... shampoo ocr1 hair food 17.9 fl oz (530 ml) hyper tokens local tokens stacked layers hair food ... argan oil feed your ... you don't need. recipes, not ... mineral oil free to feed frizzy ... shampoo hair food 17.9 fl oz (530 ml) b e smartave encoder extraction layer title description attribute spec1 specm ocrn image brand hair food ... argan oil feed your ... you don't need. recipes, not ... mineral oil free to feed frizzy ... shampoo ocr1 hair food 17.9 fl oz (530 ml) hyper tokens local tokens hyper-to-hyper attention hyper-to-local attention local-to-hyper attention local-to-local attention attribute crf decoder b e value figure 2: overview of smartave model architecture. encoder: our structured encoder jointly encodes the product with multimodal information via structured attentions, and learns effective representation for all the modalities. extraction layer: the attribute value is decoded from the learned embedding. efficient transformers our work is also related to those efficient transformer methods @xcite ) that deal with large and structured input. transformer xl @xcite designs a mechanism that is able to encode long text sequences beyond a fixed size. longformer @xcite and etc @xcite propose to extend the cls token to multiple global tokens to deal with large text input. hibert @xcite introduces a hierarchical attention pattern by dividing the input into several blocks with equal sizes. random sparse attention mechanism is proposed in @xcite which reduces the quadratic attention computations to linear time. these methods achieve promising results on dealing with large and structured text sequences. however, they are not directly applicable to structured data with multiple modalities. a comprehensive study of efficient transformers can be found in @xcite .