EIVEN : Efficient Implicit Attribute Value Extraction using Multimodal LLM (2024)

Abstract: In e-commerce, accurately extracting product attribute values from multimodal data is crucial for improving user experience and operational efficiency of retailers. However, previous approaches to multimodal attribute value extraction often struggle with implicit attribute values embedded in images or text, rely heavily on extensive labeled data, and can easily confuse similar attribute values. To address these issues, we introduce EIVEN, a data- and parameter-efficient generative framework that pioneers the use of multimodal LLM for implicit attribute value extraction. EIVEN leverages the rich inherent knowledge of a pre-trained LLM and vision encoder to reduce reliance on labeled data. We also introduce a novel Learning-by-Comparison technique to reduce model confusion by enforcing attribute value comparison and difference identification. Additionally, we construct initial open-source datasets for multimodal implicit attribute value extraction. Our extensive experiments reveal that EIVEN significantly outperforms existing methods in extracting implicit attribute values while requiring less labeled data.

## introduction product attributes are crucial in e-commerce, aiding retailers in product representation, recommendation, and categorization, and assisting customers in product searching, comparison, and making informed purchasing decisions @xcite @xcite . despite their importance, the accurate listing of these attributes remains a challenge. sellers often fail to specify all relevant attribute values or list them incorrectly, leading to inefficiencies and potential customer dissatisfaction @xcite . to address these issues, the task of attribute value extraction (ave) has † work done as an intern at amazon. existing approaches for multimodal attribute value extraction can be broadly categorized into three categories: extractive, discriminative, and generative (more detailed discussion is provided in appendix a). most extractive studies focus on extracting attribute values that are explicitly stated in product text data @xcite @xcite . however, in real-world scenarios, an attribute value that needs to be obtained may not appear as a subsequence of the product text, but can be inferred from the product image, implied text context or prior knowledge about this product type @xcite @xcite . take products in figure 1 for example. the value "round neck" of the "neckline" attribute does not appear in product textual information, but can be easily identified from its product image. similarly, the value "rain boot" corresponding to the attribute "boot style" in the second product is not explicitly stated but is implicitly embedded in its textual context "transparent waterproof" and visual information. in addition, previous discriminative and generative approaches for multimodal ave are highly datahungry, requiring large amounts of labeled data for training but still perform poorly in extracting implicit attribute values @xcite . furthermore, similar implicit attribute values are easily confused by the recent generative ave model @xcite . to tackle these challenges, we introduce eiven, a data and parameter-efficient multimodal generative framework for multimodal implicit attribute value extraction. eiven utilizes the rich inherent knowledge of a pre-trained llm and vision encoder to lessen reliance on extensive attributespecific data. additionally, to address the issue of model confusion caused by similar attribute values, we introduce a novel technique termed "learningby-comparison". this approach feeds the model with pairs of instances that share the same attribute but potentially have different attribute values, forcing the model to compare and distinguish them. our contributions are summarized as follows: ## eiven framework problem formulation. given a product's image and text context and a specified attribute, our goal is to extract the value for the corresponding attribute. specifically, in our task of extracting implicit attribute values, the ground truth attribute value does not appear as a subsequence of the text context, but can be inferred from the product image, text context, or prior knowledge. in this work, we formulate the task of extracting implicit attribute values as the problem of generating answers given a question and product information. for example, the question could be "what is the sleeve style of this product?" and the generated answer could be "short sleeve" by inferring from the product's image and text context. figure 2 presents an overview of our efficient multimodal llm, and figure 3 illustrates our learning-by-comparison strategies. next, we explain our key components in detail. ## open-source multimodal implicit ave dataset multimodal implicit ave is an emerging problem, and there is currently a lack of truly open-sourced datasets for multimodal implicit ave. 2 existing ave datasets either do not contain product images or lack implicit attribute values. thus, in this section, we introduce and make available several datasets to facilitate further research in this area. specifically, we present three multimodal implicit ave datasets: clothing, footwear, and general. the statistics of these datasets are summarized in ## conclusion in this paper, we propose eiven, an efficient generative framework using multimodal llm for implicit attribute value extraction. eiven leverages the rich internal knowledge of pre-trained llm to reduce reliance on attribute-specific labeled data and adopts lightweight adapters for parameterefficient fine-tuning of llm. besides, to enhance the visual understanding ability of our model, we feed multi-granularity visual features into llm and propose learning-by-comparison strategies to alleviate model confusion among attribute values. we also release the first open-source dataset. through extensive experiments on three multimodal implicit attribute value extraction datasets, we found that eiven can significantly outperform previous works using fewer labels, making it an efficient solution for implicit attribute value extraction.