C o M ave: Contrastive Pre-training with Multi-scale Masking for Attribute Value Extraction (2023)

Abstract: Attribute Value Extraction (AVE) aims to automatically obtain attribute value pairs from product descriptions to aid e-commerce. Despite the progressive performance of existing approaches in e-commerce platforms, they still suffer from two challenges: 1) difficulty in identifying values at different scales simultaneously; 2) easy confusion by some highly similar fine-grained attributes. This paper proposes a pre-training technique for AVE to address these issues. In particular, we first improve the conventional token-level masking strategy, guiding the language model to understand multi-scale values by recovering spans at the phrase and sentence level. Second, we apply clustering to build a challenging negative set for each example and design a pre-training objective based on contrastive learning to force the model to discriminate similar attributes. Comprehensive experiments show that our solution provides a significant improvement over traditional pre-trained models in the AVE task, and achieves state-of-the-art on four benchmarks.

## introduction product features are crucial components of ecommerce platforms and are widely used in applications such as product recommendation @xcite , product retrieval @xcite , and product question answering @xcite . each product feature typically consists of an attribute and one or more values, providing detailed product descriptions to help customers make purchasing decisions. in recent years, attribute value extraction (ave) @xcite @xcite methods have received increasing attention because they can automatically extract product features from a massive amount of unstructured product text, with impressive results in e-commerce platforms, such as amazon, aliexpress, and jd. however, as e-commerce grows, some emerging domains, such as finance, insurance, and healthcare, bring two new challenges: a) multi-scale values. unlike normal products (e.g., clothing) with only short values (e.g., color: red), insurance products can have a value of a longer phrase or even multiple sentences. for example, the value of attribute renewal rule in figure 1 contains more than 25 words (in green), rendering it impractical to retrieve them using related techniques such as name entity recognition (ner) @xcite . b) fine-grained divisions of attributes. compared with the coarse division of attributes in traditional e-commerce (e.g., color, size, and material), the division in insurance products is more refined, resulting in different attributes often having similar types. for instance, in the insurance clauses in figure 1 , maximum insurance age and maximum renewal age are both ages, and grace period and hesitation period are both periods. this fine-grained division makes the distinction be-tween the different attributes subtle, thus increasing the difficulty to distinguish between them. although recent pre-trained language models (plms) such as bert @xcite and roberta @xcite achieve tremendous success on a spectrum of nlp tasks, including ave, we argue that they are not sufficient for the challenges above. first, the conventional masking language model (mlm) focuses on token-level recovery and does not consider multi-scale values. second, there is still a gap between the unsupervised general objectives and the downstream ave in terms of task form, such that the model cannot benefit from pre-training when retrieving attributes, let alone distinguishing between fine-grained similar attributes.. in this paper, we propose comave, a novel plm for ave tasks. relying on the large-scale corpus of triples ⟨text, attribute, value⟩ collected by distant supervision, we propose three pre-training objectives to address the challenges: a) multi-scale masked language model (msmlm). we extend token-level recovery to the phrase as well as the sentence level, using different masking mechanisms to force the model to perceive spans of various lengths, thus providing a basis for identifying values at different scales. b) contrastive attribute retrieval (car). to adapt the model to the fine-grained division of attributes, we require it to retrieve the correct attributes from a challenging candidate set of semantically similar attributes. the candidates are mainly collected by clustering and a contrastive loss is designed to help the model perceive the subtle differences between them. c) value detection (vd). to close the gap between pre-training and downstream ave and further enhance the model's perception of values extraction, we let the model recognize all values without considering the corresponding attribute. to fully evaluate our pre-trained comave, we construct a new challenging benchmark ins. it consists of financial and medical texts from real scenarios and the corresponding manual annotations and is full of the two challenges we mentioned. comprehensive experiments on four ave datasets including ins demonstrate that, equipped with only a simple fine-tuning output layer, our comave not only achieves state-of-the-art results on the hardest ins but also outperforms all the compared methods on existing benchmarks. our contributions are summarized as follows: ## preliminaries given a natural language text t and a set of candidate attributes set @xmath0, a 2 , ..., a |a| }, where a i is an attribute, the goal of ave is to extract a set note that although formally ave is similar to ner, the two still have significant differences, as we mentioned in section 1. first, the division of attributes is more fine-grained than the division of entity types (e.g., location and person). second, the scale of entities is generally shorter, while that of values varies from token level to sentence level. therefore, conventional ner methods are difficult to directly port to ave tasks. ## experiments datasets to comprehensively evaluate our method, we used the following four datasets covering both english and chinese: 1) ins is a chinese ave dataset which is collected from the real product data of alipay foot_1 platform. it contains various types of large-scale insurance products from real scenarios, including wealth insurance, health insurance, travel insurance, life insurance, etc. from each product document, the attributes and values are manually annotated. there are 29 global attributes and the samples are divided into 9112/1138/1138 for train/val/test, respectively. note that the results are the amounts of the values over 2400 attributes obtained from aliexpress foot_3 . in order to make a fair comparison with previous models that could not handle a large number of attributes, we selected 4 frequent attributes (i.e. brandname, material, color, category) and divided the relevant instances randomly by 7:1:2, referring to the dataset publisher. 4) mae @xcite is an english multi-modal ave dataset that contains 200 million samples and 2000 attributes. following @xcite , we built an maetext dataset to focus on the textual modal. same as ae-pub, we also selected the 20 most frequent attributes from train/val/test sets. ## related work with the development of e-commerce, attribute value extraction which aims to retrieve the attributes and extract the values from the target data resource in order to obtain the structured information of the products recently attracts lots of at-tention. several previous methods @xcite employ traditional sequence tagging models. furthermore, aveqa @xcite first tries to use mrc based method to handle the task, but it can not be applied when each attribute has several different values. jave @xcite ) designs a multi-task model which divides the task into two sub-task: attributes prediction and value extraction. adatag @xcite uses a hyper-network to train experts' parameters for each attribute in order to build an adaptive decoder. queaco @xcite adopts a teacher-student network to leverage weakly labeled behavior data to improve performance. maveqa @xcite mixes multisource information with a novel global and local attention mechanism. however, none of the existing methods pay attention to the two challenges mentioned in section 1. language model pre-training @xcite and task-specific fine-tuning achieve significant improvement on many nlp tasks. recently, some work @xcite @xcite further modified the mlm to achieve better results. in information extraction tasks, uie @xcite is proposed as a universal pre-training model for several extraction tasks by generation, it is generic but lacks further fitting for different extraction tasks. currently, there is no task-specific pre-training model for attribute value extraction. ## conclusion in this paper, we presented a new pre-training model for attribute value extraction, called co-mave which is pre-trained by three novel objectives with a large-scale corpus. multi-scale masked language model is designed to force the model to understand multi-scale values by recovering masked spans at both the phrase and sentence levels. contrastive attribute retrieval improves the discrimination of fine-grained attributes based on contrastive learning. meanwhile, value detection is adopted to reinforce the value extraction and further benefit downstream ave tasks. extensive experiments indicate that comave achieves stateof-the-art results on four benchmarks compared with the existing baselines and plms. in future work, we will expand our work on more scenarios and industries, and also explore the optimization of the downstream fine-tune model.