Large Scale Generative Multimodal Attribute Extraction for E -commerce Attributes (2023)

Abstract: E-commerce websites (e.g. Amazon, Alibaba) have a plethora of structured and unstructured information (text and images) present on the product pages. Sellers often don’t label or mislabel values of the attributes (e.g. color, size etc.) for their products. Automatically identifying these attribute values from an eCommerce product page that contains both text and images is a challenging task, especially when the attribute value is not explicitly mentioned in the catalog. In this paper, we present a scalable solution for this problem where we pose attribute extraction problem as a question-answering task, which we solve using MXT, that consists of three key components: (i) MAG (Multimodal Adaptation Gate), (ii) Xception network, and (iii) T5 encoder-decoder. Our system consists of a generative model that generates attribute-values for a given product by using both textual and visual characteristics (e.g. images) of the product. We show that our system is capable of handling zero-shot attribute prediction (when attribute value is not seen in training data) and value-absent prediction (when attribute value is not mentioned in the text) which are missing in traditional classification-based and NER-based models respectively. We have trained our models using distant supervision, removing dependency on human labeling, thus making them practical for real-world applications. With this framework, we are able to train a single model for 1000s of (product-type, attribute) pairs, thus reducing the overhead of training and maintaining separate models. Extensive experiments on two real world datasets (total 57 attributes) show that our framework improves the absolute recall@90P by 10.16% and 6.9 from the existing state of the art models. In a popular e-commerce store, we have productionized our models that cater to 12K (product-type, attribute) pairs, and have extracted 150MM attribute values.

## mxt framework given a set of product-types (pts) @xmath0, p 2 , . . . , p m } and attribute-names @xmath1, a 2 , . . . , a n }, we define mxt p,a as a multi-pt, multi-attribute, and multi-modal generative model that is trained on pt-attributes from (p, a), and can be used to generate attribute value for any product in the trained pt-attribute set. the overall architecture of our model is described in figure 2. ## experimental setup & results we evaluated mxt against two state of the art methods on attribute extraction: (1) cma-clip: a multi-task classifier that uses clip @xcite for learning multi-modal embeddings of products followed by using two types of crossmodality attentions: (a) sequence-wise attention to capture relation between individual text tokens and image features, and (b) modality-wise attention to capture weightage of text and image features relative to each downstream task, (2) ner-mqmrc: this framework @xcite poses named entity recognition (ner) problem as multi question machine reading comprehension (mqmrc) task. this is the state of the art model for the text-based attribute extraction task. in this model, given the text description of a product (context), they give attribute names as multiple questions to their bert based mrc architecture, which finds span of each attribute value answer from the context. left table in the figure 1 compares the recall@90p% of the three models. we show the performance on top-5, top-10 and top-15 attributes (by number of products in which they are present. we can see that mxt outperforms mqmrc and cma-clip on both product types. e-commerce5pt: this is a benchmark dataset from ner-mqmrc paper @xcite . we take a subset of this dataset (removing numerical attributes) consisting of 22 product-attributes across 5 product types. this is a benchmark dataset for ner based models since all attribute values are present in the text in this dataset. the dataset has 273,345 and 4,259 products in train and test data respectively. we compare average f1 scores (averaged across attributes for each product type) of mxt model with ner-mqmrc on this dataset where our model outperforms ner-mqmrc on 16/22 attributes. right table in the figure 1 shows the average f1-scores (across attributes in each product type) of mxt and ner-mqmrc models. ## deployment in a popular e-commerce store, we have deployed mxt for 6 english speaking markets @xmath2. ## conclusion & future work in this paper, we presented mxt, a large scale multi-modal product attribute generation system to extract product attributes from the products listed in ecommerce stores. our model infers the attribute values using both textual and visual information present on the product pages. we introduced a novel architecture comprising a t5 based encoder and decoder along with two fusion layers to fuse text and image embeddings. we showed our model can beat the existing state of the art extractive as well as predictive models on the benchmark datasets. our model is scalable to multiple product types and countries by just specifying them in the input text prompt. we further showed that our model is able to perform zero-shot inference, as well as it can generate attribute values not present in the text. there are several future directions to ex-plore which can further improve the performance of our model. first, we would like to create an ensemble of ner-based and generative models so that we can leverage the power of extraction based models which work very well for numerical attributes (e.g. size, length etc.). second, our current approach does not use relational information among the products. since similar products can have common attribute values, we can use graph based approaches to capture that relational information. specifically, we can approach the attribute extraction problem through either link prediction or node classification. in the former method, we aim to predict missing links between products and their attributes. alternatively, the latter approach involves using similarity between product features, including text, images, and co-viewing information, to determine graph edges for classification of product nodes.