Learning Cross-Task Attribute - Attribute Similarity for Multi-task Attribute-Value Extraction (2021)

Abstract: Automatic extraction of product attribute-value pairs from unstructured text like product descriptions is an important problem for e-commerce companies. The attribute schema typically varies from one category of products (which will be referred as vertical) to another. This leads to extreme annotation efforts for training of supervised deep sequence labeling models such as LSTM-CRF, and consequently not enough labeled data for some vertical-attribute pairs. In this work, we propose a technique for alleviating this problem by using annotated data from related verticals in a multi-task learning framework. Our approach relies on availability of similar attributes (labels) in another related vertical. Our model jointly learns the similarity between attributes of the two verticals along with the model parameters for the sequence tagging model. The main advantage of our approach is that it does not need any prior annotation of attribute similarity. Our system has been tested with datasets of size more than 10000 from a large e-commerce company in India. We perform detailed experiments to show that our method indeed increases the macro-F1 scores for attribute value extraction in general, and for labels with low training data in particular. We also report top labels from other verticals that contribute towards learning of particular labels.

## introduction online e-commerce marketplaces (e.g., flipkart) operate by efficiently matching customer queries and browsing habits to appropriate seller inventory. inventory is stored in a catalog which consists of images, structured attributes (keyvalue pairs) and unstructured textual description as shown in figure 1 . products of same kind (e.g., digital camera) are thus described using a unique set of attributes (e.g., zoom, resolution) -helping faceted navigation, merchandizing, search ranking and comparative summary. onboarding products in a catalog requires populating the structured as well as unstructured parts. the time a seller has to spend on a product addition request is proportional to the quantum of information that he/she has to provide. on the other hand, correctness and completeness of catalog results in better product discovery, leading to a trade-off with its onboarding time. a good amount of attributes information is present in product description as well. this motivates us to extract the information from unstructured text instead of explicitly asking sellers for attributes. additional information in description (e.g., precise features, relation between products) as shown in figure 1 helps to enrich the catalog as well. the extracted attributes can be used to check consistency between unstructured and structured data provided by seller and thus quality control of addition request. we design supervised deep learning techniques for the problem of attribute value extraction. figure 2 , shows a typical input sentence and the corresponding b, i, o tags. the task of our model is to predict the tags given an input sentence. this problem is related supervised sequence labelling problem @xcite . however, this technique needs a lot of training data points (sentence -label pairs) to perform effectively, which in turn requires massive annotation efforts on the part of e-commerce companiesreduction of which is an ongoing challenge; open-tag @xcite uses active learning to annotate only most informative examples. e-commerce companies however have the products categorized as different verticals, e.g. dress, jeans, etc. each of these verticals have a different set of attributes, and hence needs to annotated using different models. a lot of the attributes among these verticals are common, or related though. hence, it should be possible to borrow information from annotations given in different verticals, to improve the prediction performance of a given vertical. the only challenge is that correspondences between similar labels of different verticals is not available readily. our main contribution here is thus to develop a multi-task learning (mtl) model @xcite which can simultaneously learn attribute extraction and attribute-attribute similarity for multiple verticals (here we report with only two verticals at a time). we do so by using a soft coupling loss function across pairs of similar (context,label) combinations between the two tasks, where similarity is learned using attention mechanism. the naive version of such an objective will be prohibitively large to optimize. we propose to use a cosine similarity based shortlist, which makes the solution feasible. we validate our method using a large corpus (more than 10000 product descriptions, across 6 verticals) collected from the e-commerce company -flipkart. extensive experimentation shows that our method improves performance of prediction on almost all the verticals, and especially shows upto 50% improvement for many labels which have low number of training examples. this is especially interesting since we find that number of instances with an attribute is highly skewed across the attributes. detailed analysis also confirms that the attention mechanism indeed discovers similar attributes from other verticals to borrrow information from. multi-task learning (mtl): significant theoretical interest exists in mtl since it offers excel-lent generalization performance in domains where training data is scarce @xcite . in nlp, @xcite proposed a unified deep learning architecture for many common tasks e.g. pos tagging, chunking, etc. @xcite presented a new representation mtl framework that learns cross-task sharing structure at every layer in a deep network. (rijula kar, 2018) proposed a task-sensitive representation learning framework that learns mentiondependent representations for ned, and violates norm to share parameters in the final layer. @xcite treats each attribute as a question finding the best answer span corresponding to its value in product context -modelled by a bert encoder shared across all attributes for scalability. a distilled masked language model improving generalizability is then integrated with the encoder into a unified mtl framework. @xcite applies to thousands of product categories organized in a hierarchical taxonomy. however, existing methods do not automatically discover attribute-attribute similarity from data, without taking attribute hierarchy as input. ## methods in this section, we describe a novel multi-task approach to improving the accuracy of a supervised attribute value extraction system. we start with the attribute-value extraction system, based on deep bidirectional lstm model, described in opentag @xcite . our main idea here is to leverage the information contained in instances of related tasks, e.g. in our case related domains / verticals of products. the key challenge in our case is that the set of labels across verticals need not be same, or even aligned. for example, the label processor type is a valid label for laptop vertical but does not make sense for dress vertical. on the other hand, the set of values for the common label brand will be very different for the vertical dress compared to the vertical laptop. hence, our core challenge here is to determine the similarities between labels automatically in the context of each vertical in order to leverage the information from a related vertical. the proposed architecture is described in figure 3 . ## experimental results in this section, we report results from our proposed method for multi-task attribute extraction, against single task attribute extraction. we implemented our model using tensorflow on a 8-core centos machine. we used 300 dimensional pre-trained glove vectors. we have also experimented with other customized word embeddings e.g. fasttext, but did not achieve significantly better results. for this work, we use single layer bilstm as the embedding layer. the hidden layer size for bilstm layer was set to 700. we have experimented with other embedding layer architectures, e.g. hidden layer sizes ranging from 300 to 900, and also two layer bilstms with hidden layer sizes (500,700). however, the performance of single layer lstm with hidden layer size 700 was found to be similar or better than others. for training, the batch size was chosen to be 30 for both the crf loss batches and for coupling loss batches sampled from the shortlist l. adam was used as optimizer and we trained for maximum of 30 epochs. we trained the model for 30 epochs. evaluation metric as reported below, the datasets for this problem show extreme skew in terms of occurrence of labels. hence, we use the standard metrics of macro precision, macro recall, and macro f1 score. we also report the microaccuracy. while computing the macro-metrics (precision, recall and f1), we ignore the 'o' label. it is clear that macro-f1 score without the 'o' label, is the most representative metric here, from an application point of view. ## conclusion in this paper, we study attribute-value extraction from production description in the e-commerce domain. many of the attributes occur in very few descriptions. hence the amount of supervised training data available for these attributes is very low, which leads to low prediction performance we thus propose a novel multi-task learning based algorithm which borrows information from related domains (i.e., category/vertical) in order to improve prediction performance of infrequently occurring attributes. we validate the proposed method with extensive experimental evaluation on a large dataset of six verticals from a prominent, e-commerce company. the proposed technique not only achieves higher accuracy on verticals with similar labels, but also can be used for discovering attribute similarities across verticals.