Multimodal Joint Attribute Prediction and Value Extraction for E -commerce Product (2020)

Abstract: Product attribute values are essential in many e-commerce scenarios, such as customer service robots, product recommendations, and product retrieval. While in the real world, the attribute values of a product are usually incomplete and vary over time, which greatly hinders the practical applications. In this paper, we propose a multimodal method to jointly predict product attributes and extract values from textual product descriptions with the help of the product images. We argue that product attributes and values are highly correlated, e.g., it will be easier to extract the values on condition that the product attributes are given. Thus, we jointly model the attribute prediction and value extraction tasks from multiple aspects towards the interactions between attributes and values. Moreover, product images have distinct effects on our tasks for different product attributes and values. Thus, we selectively draw useful visual information from product images to enhance our model. We annotate a multimodal product attribute value dataset that contains 87,194 instances, and the experimental results on this dataset demonstrate that explicitly modeling the relationship between attributes and values facilitates our method to establish the correspondence between them, and selectively utilizing visual product information is necessary for the task. Our code and dataset are available at https://github.com/jd-aig/JAVE .

## introduction product attribute values that provide details of the product are crucial parts of e-commerce, which help customers to make purchasing decisions and facilitate retailers on many applications, such as question answering system @xcite , product recommendations (gong, figure 1 : an example of predicting attributes and extracting values from the textual product description with the aid of the visual product information. though plenty of systems have been proposed to supplement product attribute values @xcite @xcite @xcite , the relationship between product attributes and values are not sufficiently explored, and most of these approaches primarily focus on the text information. attributes and values are, however, known to strongly depend on each other, and vision can play a particularly essential role for this task. intuitively, product attributes and values are mutually indicative. given a textual product description, we can extract attribute values more accurately with a known product attribute. we model the relationship between product attributes and values from the following three aspects. first, we apply a multitask learning @xcite ) method to predict the product attributes and the values jointly. second, we extract values with the guidance of the predicted product attributes. third, we adopt a kullback-leibler (kl) @xcite measurement to penalize the inconsistency between the distribution of the product attribute prediction and that of the value extraction. furthermore, beyond the textual product descriptions, product images can provide additional clues for the attribute prediction and value extraction tasks. figure 1 illustrates this phenomenon. given a description "this golden band collar shirt can be paired with black shoes", the term "golden" can be ambiguous for predicting the product attributes. while by viewing the product image, we can easily recognize the attribute corresponding to "golden" is "color" instead of "material". moreover, the product image can indicate that the term "black" is not an attribute value of the current product; thus, it should not be extracted. this may be tricky for the model based on purely textual descriptions, but leveraging the visual information can make it easier. in addition, multimodal information shows promis-ing efficiency on many tasks @xcite @xcite @xcite @xcite @xcite . therefore, we propose to incorporate visual information into our task. first, we selectively enhance the semantic representation of the textual product descriptions with a global-gated cross-modality attention module that is anticipated to benefit attribute prediction task with visually grounded semantics. moreover, for different values, our model selectively utilizes visual information with a regional-gated cross-modality attention module to improve the accuracy of values extraction. our main contributions are threefold: ## experiment we compare our proposed methods with the following baselines: wsm is the method that uses attribute values in the training set to retrieve the attribute values in the testing set by word matching. sep-bert is the pretrained bert model with feed-forward layers to perform these two subtasks separately. rnn-lstm @xcite , attn-birnn @xcite , slot-gated @xcite , and joint-bert @xcite are the models to address intent classification and slot filling tasks, which are similar to the attribute prediction and value extraction, and @xcite 86.10 83.28 slot-gated @xcite 86.70 83.35 joint-bert @xcite 86.93 83.73 scalingup ( xu et al., 2019 we adopt these models to our task. rnn-lstm and attn-birnn use a bidirectional lstm and an attention-based model for joint learning, respectively. slot-gated introduces a gate-based mechanism to learn the relationship between these two tasks. joint-bert finetunes the bert model with joint learning. scalingup @xcite adopts bilstm, crf, and attention mechanism for introducing hidden semantic interaction between attribute and text. we report the results of our text-only and multimodal models, i.e., jave and m-jave. in addition, to eliminate the influences of different text encoders, we also conduct experiments with bilstm as the text encoder. details about hyper-parameters are shown in ## related work recent approaches related to the attribute value pair completion task can be classified as the following two categories. ## conclusion we jointly tackle the tasks of e-commerce product attribute prediction and value extraction from multiple aspects towards the relationship between product attributes and values, and we prove that the models can benefit a lot from visual product information by the aid of global and regional visual gates. we collect a multimodal e-commerce product attribute value extraction (mepave) dataset foot_1 . the experimental results on this dataset show that the correlations between product attributes and values are valuable for this task, and visual information should be selectively used.