Visual Zero-Shot E -Commerce Product Attribute Value Extraction (2025)

Abstract: Existing zero-shot product attribute value (aspect) extraction approaches in e-Commerce industry rely on uni-modal or multi-modal models, where the sellers are asked to provide detailed textual inputs (product descriptions) for the products. However, manually providing (typing) the product descriptions is time-consuming and frustrating for the sellers. Thus, we propose a cross-modal zero-shot attribute value generation framework (ViOC-AG) based on CLIP, which only requires product images as the inputs. ViOC-AG follows a text-only training process, where a task-customized text decoder is trained with the frozen CLIP text encoder to alleviate the modality gap and task disconnection. During the zero-shot inference, product aspects are generated by the frozen CLIP image encoder connected with the trained task-customized text decoder. OCR tokens and outputs from a frozen prompt-based LLM correct the decoded outputs for out-of-domain attribute values. Experiments show that ViOC-AG significantly outperforms other fine-tuned vision-language models for zero-shot attribute value extraction.

## introduction product attribute value extraction aims at retrieving the values of attributes from the product's unstructured information (e.g. title, description), to serve better product search and recommendations for buyers. existing uni-modal or multi-modal attribute value extraction models require sellers to manually provide (type) product descriptions, which is time-consuming and frustrating. in addition, these approaches mainly focus on supervised learning, weakly-supervised learning, and few-shot learning to train or fine-tune language models for attribute value prediction @xcite @xcite . these approaches need labeled data for training and can not be extended to unseen attribute values for new products. to extract unseen attribute values, text-mining models @xcite , inductive graph-based models @xcite , and multi-modal large language models @xcite try to generate potential attribute values from both product descriptions and images. however, these approaches suffer from several limitations: (1) it is difficult for classification or graph-based prediction models to scale to a large number of attribute values because the decision boundaries between classes become more complex and harder to learn, and increase the computational complexity. (2) traditional information extraction models or the above multi-modal models need the inputs for product textual descriptions from the sellers (see figure 1 ). it is challenging and timeconsuming for the sellers to manually type and provide the product descriptions because sometimes sellers themselves don't know the correct answers, which may cause ambiguity for attribute values. to address the above limitations, we propose an ocr and product captions enhanced zeroshot cross-modal model (vioc-ag) to generate attribute values, which only need the product images as the inputs. in other words, the seller only needs to take a photo of the product that he wants to sell without manually providing the product textual descriptions, resulting in a better user experience. there are two main challenges for zero-shot cross-modal aspect generation. the first challenge is the modality gap between vision and language caused by cross-modal generation. although there exist many large generative image-to-text transformers (i.e. blip-2 @xcite )), they target at the image captioning or visual question answering tasks. our experiments in sec. 4 show that simply fine-tuning these large vision language models performs poorly on the product attribute value generation task. this is because there is a task disconnection between language modeling (used for image captioning) and aspect generation. thus, we take advantage of the pre-trained clip @xcite ability to align visual and textual representations in a shared embedding space to avoid the modality gap. to alleviate task disconnection, we train a task-customized text decoder with a projection layer, which follows a text-only training process. specifically, we tend to transfer clip textual description embeddings back into textual aspects by learning a task-customized decoder for the frozen clip text encoder using only text. the second challenge is the out-of-domain aspects caused by zero-shot generation. for zeroshot aspects, the model is susceptible to generate aspects that are not actually present in the input image but frequently appear during training (object hallucination). due to the characteristics of the product attribute value generation task, some aspects (i.e. brand, capacity, etc.) are shown directly on the product. thus, we correct the generated outputs from the trained task-customized text decoder with the ocr tokens. for further final aspects correction, we generate potential attribute value answers by designing prompt templates for pre-trained visual question-answering llms. the effectiveness of each module is shown independently in sec. 4.2. extensive experimental results on mave @xcite dataset show that our proposed model vioc-ag significantly outperforms other existing vision language models for zero-shot attribute value generation. vioc-ag also achieves competitive results with generative llms with textual product description inputs, showing the positive potential that users only need to take photos of the selling products for aspect generation. to extract unseen attribute values, text-mining models @xcite extract explicit attribute values directly from text, and zeroshot models @xcite predict new attribute values by inductive link prediction of graphs. however, all these approaches can only extract attribute values from textual inputs. in other words, these models are from a single modality. then, some multi-modal models use both the product image and title with the description as the inputs to learn a better product representation for attribute value extraction @xcite @xcite @xcite . though performance is improved by fusing more semantic information from multiple modalities, more input data is needed during the training stage. to enable image-first interactions from sellers and make it simple for the users, we propose a zero-shot cross-modal model motivated by image captioning @xcite @xcite @xcite for attribute value generation, where only images are used as inputs. ## conclusion in this paper, we formulate the attribute value extraction as a cross-modal generation task, which only requires product images as the inputs. we propose vioc-ag to generate unseen product aspects, which includes a text-only trainable projector and task-customized decoder to alleviate both the modality gap and task disconnection. for zero-shot inference, vioc-ag employs ocr tokens and results from a frozen prompt-based llm to correct the decoded outputs for out-of-domain attribute values. results on mave demonstrate that our proposed model vioc-ag outperforms other stateof-the-art fine-tuned vision-language models and it can achieve competitive results with textual generative llms, showing the bright future directions of cross-modal zero-shot attribute value generation.