Stronger, Lighter, Better: Towards Life-Long Attribute Value Extraction for E -Commerce Products (2024)

Abstract: Attribute value extraction involves identifying the value spans of predetermined attributes in product texts. This area of research has traditionally operated under a closed-world assumption, focusing on products from a static set of categories and their associated attributes. However, products in e-commerce stores are ever-increasing and evolving, calling for life-long learning. If continuously trained on the fast-increasing products and attributes, most existing solutions not only struggle for parameter efficiency but also endure foreseeable defects due to data contamination, catastrophic forgetting, etc. As a remedy, we propose and study a new task, which aims to effectively maintain a strong single model for many domains in a life-long learning fashion, without jeopardizing the model performance and parameter efficiency. We introduce factorization into the model and make it domain-aware by decoupling the modeling of product type and attribute, as a way to promote de-contamination and parameter efficiency while scaling up. Tuning the model with distillation prevents forgetting historical knowledge and enables continuous learning from emerging domains. Experiments on hundreds of domains showed that our model attains the near state-of-the-art performance with affordable parameter size, the least historical knowledge forgetting, and the greatest robustness against noises, whilst adding only a few parameters per domain when compared with competitive baselines.

## introduction attribute value extraction (ave) @xcite , similar to named entities recognition (ner) @xcite , aims to extract surface-form names (e.g. red/vanilla/plastic) of pre-defined product attributes (e.g. color/scent/material) from product texts (e.g. product title/description). more specifically, we define each domain as the combination of one product category/type (pt, e.g. laundry_detergent) and one attribute (attr, e.g. scent) -extracting scent values from laundry detergent products is an example of a domain. training one model per domain (one-for-each) means we need to maintain tens of thousands of models, which is not scalable and does not fully leverage the correlation between multiple domains jointly. therefore, training a single model for all domains (all-in-one) attracts intensive studies @xcite @xcite @xcite . these works aim to train a scalable all-in-one model that absorbs stacked data from all domains in a closed-world fashion. they achieve decent performance while assuming a fixed list of pre-defined domains is available. however, products in the real world keep evolving in an openworld fashion. new attributes, new values, and new product types necessitate effective knowledge transfer and life-long adaptation, without the need to retrain the entire model from scratch. effective parameter sharing and parameter efficiency serve critical roles in such knowledge transfer and scalable adaptation across many domains. intuitively, similar pts can share the same attributes and values. for example, laun-dry_detergent and hand_sanitizer may both have cedarwood as the scent value. the approach of fully shared parameters does not always result in the desired knowledge transfer and can sometimes negatively impact performance, especially when dealing with complex and evolving data. for example, different pts possess drastically different values on the same attribute: size for shoes can be 7/7.5/8 and size for shirt takes values such as xs/s/m/l/xl. also, the same value can be adopted by different domains: orange can be a typical flavor value for drink products, while the same value is usually mentioned in umbrella products as a color. although existing works @xcite @xcite started to adopt various techniques to add additional inputs to make the model better aware of the domain it is handling and therefore alleviate domain contamination and facilitate knowledge transfer, we argue that because model parameters are essentially accessible to all domains, such fine-grained, domain-specific knowledge may find it hard to be stored organically in the form of loosely organized model parameters, exactly according to how the knowledge distributes and evolves. for the first time, we study the open-world lifelong ave task that requires model capability on both parameter efficiency for scale-up extraction and life-long extraction of evolving new data. we developed a unified solution, dalla -domain-aware life-long adapter. to alleviate the domain contamination without sacrificing parameter efficiency, we propose a factorized adapter to decouple the product type and the attribute by assigning them with smaller sub-matrices after performing low-rank decomposition on model weight matrices. factorization not only facilitates parameter savings but, more importantly, decouples the pt/attr aspects, allowing them to operate independently while maintaining the model's awareness of the entire domain. weight reconstruction only utilizes specific product type and attribute sub-matrices. this enables parameter sharing on a fine-grained (pt/attr) level instead of shared across all domains, and strengthens domain-relevant knowledge transfers. furthermore, we introduce a life-long distillation scheme based on the factorized adapter, to effectively boost underperforming or totally new domains while avoiding catastrophic forgetting of existing well-performing domains. our main contributions are summarized as: ## domain-aware life-long adapter problem formulation: we address the attribute value extraction (ave) problem by regarding it as a sequential tagging problem, as a popular choice. based on the bio schema , ave assigns a sequence of labels @xmath0, ..., y n token ] to a given product profile sentence @xmath1, ..., x n token ] with n token tokens, given a product type (pt) and an attribute (attr). we take the combination of a product type and an attribute as a domain (p t, attr). given the domain, an ave model should mark out all the entities in the product profile as attribute values. an entity is a span of tokens @xmath2, ...x j ](1 ≤ @xmath3. specifically, the first token of an entity mentioned in the sentence is labeled as b, the other tokens inside that entity mention are labeled as i, and the non-entity tokens are labeled as o. open-world life-long ave task: previous works @xcite @xcite show remarkable performances on closed-world data but fail to address the concern of life-long learning capabilities on the open-world data, which keeps evolving on either attribute value or domain set. similar to most previous works, directly retraining with open-world data leads to catastrophic forgetting. unlike closed-world settings, for open-world life-long ave task, we are inspired by @xcite and seek to incrementally learn a model f with t sequential domains each domain is regarded as a unit dataset absorbed by f for training and testing. an ideal solution for the open-world, life-long ave task requires: (1) scalability: the model can provide cost-effective training on large-sized and number-increasing domains. we seek parameter efficiency by a domain-aware factorized adapter, depicted in section 2.1. (2) robustness: the model can overcome catastrophic forgetting and domain contamination during evolving training. we decouple sub-matrices for pts and attr for conditional parameter sharing and allow each pt/attr sub-matrix to update independently to avoid contamination and propagation, detailed in section 2.1. furthermore, we distill knowledge through the factorized sub-matrices for knowledge forgetting prevention and similar pts/attrs association, presented in section 2.2. (3) generalizability: the model is applicable to unseen data. we will use domain-aware sub-matrices to enable unseen domain extraction in section 2.2. ## evaluation of life-long adaptation alleviating catastrophic forgetting we first study dalla's capability of overcoming catastrophic forgetting and domain contamination. to evaluate catastrophic forgetting alleviation, we split the domains into seen and unseen domains. initially, we train the model on the seen domains during the closed-world training stage. subsequently, we proceed with open-world training by conducting @xmath4. we then record the performance drops on seen domains after each set of open-world training sessions. there are two settings in the open-world training, one is when no additional seen domain data is available (n @xmath5, no additional seen samples; unseen data only). we aim to use this setting to test the direct knowledge transfer capability to unseen domains. the other setting also includes new values from seen domains (n @xmath6, 10% additional seen samples). we aim to use this setting to test if the model can be assertive in learning new knowledge while not forgetting the knowledge on the existing seen domains. from the left part of in figure 3 , we present two groups of the results: direct transfer to the unseen (n @xmath9. for the first group results, dalla shows the second-best performance among other baselines, because maveqa, originated from where the dataset comes, achieves nearly 99% on all attributes as reported in the paper @xcite . we discussed more details about maveqa in the appendix c. the second group exhibits not only the macro-f1 scores but also the performance changes considering the effect of adding in a small portion of seen domain examples. dalla improves the performance while the other baselines suffer from decreased performance in the case of (p t old , attr old ). this suggests that the other baselines suffer from contamination from the seen domain samples, attempting to maintain performance, let alone strengthen desired knowledge transfer to enhance the performance of the domains that consist of existing pts and attrs. we see bert and maveqa loss score in the case of (p t new , attr new ). the others use their proposed domain-aware component to build specific knowledge for the totally new domains, while they directly concatenate pt and attr with the model input is insufficient to capture the current domain well. compared with bert f t and suopentag, dalla still shows the superiority of domain-aware ability due to independent updates on pt/attr sub-matrices. in the remaining two cases, (p t new , attr old ), and (p t old , attr new ), we find learning on new pts with existing attrs is harder than new attrs with existing pts according to smaller gains and performance drop shown in the case of (p t new , attr old ). in fact, the value form for the same attr among different pts can be dramatically different. e.g. size for shoes should be numerical like 7/7.5/8, while size for shirt takes values in characters such as xs/s/m/l/xl. moreover, we observe from the data that many attributes can be shared by tens to hundreds of product types but much sparser for product types to attributes, which implies learning well on attr sub-matrices is harder than that on pt sub-matrices. if the attr sub-matrix is underfitting, directly adopting it as initialization still requires huge efforts in model training instead of providing a beneficial warm-up. ## analysis ## conclusion in this paper, we study the open-world life-long attribute value extraction task and devise a factorized life-long adapter module to enable model scalability and robustness. factorized adapter decouples product type and attribute, allowing domain interaction on either pt-level or attr-level. therefore, the extraction model is domain-aware but updated independently for de-contamination. moreover, we endow the model with life-long adaptation ability by distilling the decoupled information to avoid catastrophic forgetting. dalla can achieve decent performance on the benchmark with the ability to control forgetting and error propagation and only consume affordable parameters.