M ix PAVE : Mix-Prompt Tuning for Few-shot Product Attribute Value Extraction (2023)

Abstract: The task of product attribute value extraction is to identify values of an attribute from product information. Product attributes are important features, which help improve online shopping experience of customers, such as product search, recommendation and comparison. Most existing works only focus on extracting values for a set of known attributes with sufficient training data. However, with the emerging nature of e-commerce, new products with their unique set of new attributes are constantly generated from different retailers and merchants. Collecting a large number of annotations for every new attribute is costly and time consuming. Therefore, it is an important research problem for product attribute value extraction with limited data. In this work, we propose a novel prompt tuning approach with Mix ed P rompts for few-shot A ttribute V alue E xtraction, namely MixPAVE. Specifically, MixPAVE introduces only a small amount (< 1%) of trainable parameters, i.e., a mixture of two learnable prompts, while keeping the existing extraction model frozen. In this way, MixPAVE not only benefits from parameter-efficient training, but also avoids model overfitting on limited training examples. Experimental results on two product benchmarks demonstrate the superior performance of the proposed approach over several state-of-the-art baselines. A comprehensive set of ablation studies validate the effectiveness of the prompt design, as well as the efficiency of our approach.

## introduction product attributes are important features associated with products, which form an essential component of e-commerce platforms @xcite . these attributes contain detailed information of the products, and provide figure 1 : an example of a "desktop" product from mave dataset. there are multiple attributes with their corresponding values in the product during training. we want the model to adapt quickly to new attributes. useful guidance for customers to compare products and make purchasing decisions. they also facilitate merchants on various applications, including product recommendations @xcite , product search @xcite , and product question answering @xcite . therefore, product attribute value extraction has recently attracted a lot of interest from both academia and industry, with a plethora of research @xcite @xcite @xcite being developed to solve this problem. existing works @xcite @xcite mostly learn extraction models to extract values for a predefined set of attributes, assuming all attributes are covered in the training examples. however, in real-world scenarios, new products are emerging everyday with their unique set of new attributes. in fact, even within the existing products, there are new attributes being generated by the merchants. for example, figure 1 shows a "desktop" product, where the values corresponding to the attributes of "brand", "ram", "operating system" and "cpu" are annotated in the product title and description, but the "graphic card" attribute is not included in the training attributes. in this case, a desired model should quickly adapt to new attributes by providing a few examples. one straightforward solution is to merge the new attributes with existing ones, and retrain the extraction model. the main drawback of this approach is that the full model needs to be retrained on all training data every time when there are new attributes, making it computational expensive and thus impractical to use. an alternative approach is to conduct full fine-tuning of the existing model on the small data associated with the new attributes. this approach is more data-efficient compared to the retraining with all data. however, the fine-tuned model is likely to overfit to the new attributes (due to very few training examples) especially for largescale transformer models @xcite . another popular approach is to finetune only a subset of the parameters, such as the extraction head @xcite or the bias terms @xcite . prior research has also attempted at adding extra blocks or adapters @xcite to the existing model. however, in general these strategies under-perform the full fine-tuned model. inspired by the recent advances on prompt tuning @xcite @xcite , in this paper, we propose a novel prompt tuning approach with mixed prompts for few-shot attribute value extraction, namely mixpave. in contrast to previous finetuning or partial tuning approaches, we introduce two sets of learnable prompts, textual prompts and key-value prompts, and augment the transformerbased extraction model with the mixture of these two prompts. specifically, textual prompts are prepended to the input sequence of each transformer layer while key-value prompts are inserted to the key and value matrices in the self-attention block. then mixpave is then learned by only finetuning the prompts (less than 1% of all parameters) on the new attributes, while keeping the other parameters in the transformer frozen. therefore, our approach not only benefits from parameter-efficient training, but also avoids model overfitting with limited training examples. evaluations on two product datasets show the superior performance of our model over several state-of-the-art methods. the experimental results also demonstrate the effectiveness and efficiency of the proposed prompts in the few-shot scenarios. we summarize the main contributions as follows: ## analysis and discussion to better understand the effectiveness of mixpave, we further conduct a series of ablation studies. ## conclusions product attribute value extraction on new attributes is an important problem in many real-world applications. in this work, we propose a novel prompt tuning approach with mixed prompts for few-shot attribute value extraction (mixpave). in particular, our model introduces only a small amount of trainable parameters, consisting of two sets of learnable prompts, while keeping the backbone extraction model frozen. our mixpave not only benefits from parameter-efficient training, but also avoids model overfitting on limited training examples. experimental results on ae-110k and mave demonstrate the effectiveness and efficiency of the proposed approach.