Knowledge-Selective Pretraining for Attribute Value Extraction (2023)

Abstract: Attribute Value Extraction (AVE) aims to retrieve the values of attributes from the product profiles. The state-of-the-art methods tackle the AVE task through a question-answering (QA) paradigm, where the value is predicted from the context (i.e. product profile) given a query (i.e. attributes). Despite of the substantial advancements that have been made, the performance of existing methods on rare attributes is still far from satisfaction, and they cannot be easily extended to unseen attributes due to the poor generalization ability. In this work, we propose to leverage pretraining and transfer learning to address the aforementioned weaknesses. We first collect the product information from various E-commerce stores and retrieve a large number of (profile, attribute, value) triples, which will be used as the pretraining corpus. To more effectively utilize the retrieved corpus, we further design a Knowledge-Selective Framework (KSelF) based on query expansion that can be closely combined with the pretraining corpus to boost the performance. Meanwhile, considering the public AE-pub dataset contains considerable noise, we construct and contribute a larger benchmark EC-AVE collected from E-commerce websites. We conduct evaluation on both of these datasets. The experimental results demonstrate that our proposed KSelF achieves new state-of-the-art performance without pretraining. When incorporated with the pretraining corpus, the performance of KSelF can be further improved, particularly on the attributes with limited training resources.

## introduction with the fast expansion of e-commerce systems, attribute value extraction (ave), which aims to retrieve the values of attributes from the product profiles, has attracted significantly more research attention since it can be used to supplement product information and hence serve better product search @xcite , recommendations @xcite , and other ecommerce tasks. as shown in figure 1 , the goal of ave is to extract values of different attributes, e.g., "zebco" from the profile for the attribute brand. automating this process can considerably enrich the product information with fewer human efforts, which has become a hot research topic for many e-commerce stores @xcite . existing research on ave mainly falls into two paradigms: (1) sequence-tagging, and (2) question answering (qa). the sequence-tagging paradigm formulates ave as a named entity recognition problem, where the model is built to identify the attribute types of the tokens in product profiles @xcite @xcite @xcite @xcite . however, the sequence-tagging methods suffer from the limitations that they do not scale to cope with a large number of attributes and cannot be extended to unseen attributes @xcite . hence, recent works have proposed to formulate ave as a qa task, where the product profile and attribute are taken as the context and query, respectively @xcite @xcite , and the model is trained to extract the attribute value span from the context. the qa paradigm has shown to be more flexible and scalable than sequence-tagging methods, achieving state-of-the-art performance on the ave task. while the qa paradigm has yielded improved results, they still face significant limitations that need to be addressed. previous research demonstrates that the qa-based methods do not perform well on the attributes with limited training resources @xcite , even though @xcite propose to build better representations by expanding the query with values of the same attribute from the training set and accordingly achieves promising overall performance. the sub-optimal performance on attributes with limited training resources hinders the practical application of qa-based ave methods to e-commerce, because new and unseen attributes continue to arise with the introduction of new products. to overcome the aforementioned problems, we develop a task-specific pretraining method for the qa-based ave, which provides the model strong generalization capabilities, allowing the model to quickly adapt to rare and unseen attributes. inspired by the recent popularity of pretrained language models @xcite @xcite that leverage largescale unlabeled corpora from the web to perform pretraining, we collect product profiles as well as the corresponding product attributes from various public e-commerce websites. to automatically and efficiently construct data for pretraining, we conduct string match in the collected product information, and obtain the large-scale corpus consisting of (profile, attribute, value) triples. using these data, we further pretrain a public language model for an ave task employing the qa paradigm. to further improve the performance, we adopt the query expansion technique. given a product and an attribute to be filled with a value, @xcite employs values of the same attributes retrieved from training corpus as knowledge to expand the query, improving upon previous methods that only regard the attributes as queries. the assumption behind their method is that such expanded knowledge provides informative hints on helping identify the correct value span of the target attribute. however, these retrieved values constitute a vast set which may include those values that are not closely related to the current product, thus providing little hints and even introducing unexpected noise. for example, the value of attribute "material" for a "shirt" has nothing to do with that for a "desk". this is particularly severe in the pretraining scenario because the pretraining data is typically in large-scale and there will be a huge number of helpless values from a variety of products for the same attribute. ideally, a desirable solution should be able to effectively identify the usefulness of knowledge and employ informative knowledge for query expansion. therefore, on the basis of query expansion-based ave, we propose a knowledge-selective framework (kself). when performing query expansion, the proposed method first categorizes the retrieved values into the more-related and less-related, based on their similarity to the current product. then we construct the expanded queries by properly exploiting the two types of attributes during training and inference. kself can be closely combined with the constructed pretraining corpus, allowing for more efficient utilization of training data, which further improves the generalization ability of the model. to benchmark the kself, we conduct experiments on the ae-pub dataset @xcite . however, we notice that ae-pub @xcite ) contains significant noise. to encourage future research on this task, we further propose a new benchmark ec-ave of high quality collected from a well-structured e-commerce website. the results demonstrate that our proposed kself achieves new state-of-the-art performance on both datasets without pretraining. when pretrained on the largescale corpus and then finetuned on the downstream datasets, kself can further attain better performance, particularly on the attributes with limited training resources. we summarize our contributions as follows: ## conclusions our study aims to improve the performance of attribute value extraction for rare attributes, which is of significant importance for real-life applications. we propose a pretraining method to improve the generalization ability of the qa-based ave model. in addition, we propose a knowledgeselective query expansion framework that can effectively exploit the training data.