MIND : Multimodal Shopping Intention Distillation from Large Vision-language Models for E -commerce Purchase Understanding (2024)

Abstract: Improving user experience and providing personalized search results in E-commerce platforms heavily rely on understanding purchase intention. However, existing methods for acquiring large-scale intentions bank on distilling large language models with human annotation for verification. Such an approach tends to generate product-centric intentions, overlook valuable visual information from product images, and incurs high costs for scalability. To address these issues, we introduce MIND, a multimodal framework that allows Large Vision-Language Models (LVLMs) to infer purchase intentions from multimodal product metadata and prioritize human-centric ones. Using Amazon Review data, we apply MIND and create a multimodal intention knowledge base, which contains 1,264,441 intentions derived from 126,142 co-buy shopping records across 107,215 products. Extensive human evaluations demonstrate the high plausibility and typicality of our obtained intentions and validate the effectiveness of our distillation framework and filtering mechanism. Further experiments reveal the positive downstream benefits that MIND brings to intention comprehension tasks and highlight the importance of multimodal generation and role-aware filtering. Additionally, MIND shows robustness to different prompts and superior generation quality compared to previous methods.

## introduction understanding customers' intentions behind their purchase behaviors remains crucial in e-commerce as it potentially benefits several downstream tasks, such as product recommendation @xcite @xcite and search * equal contribution orbit trackball mouse wireless solar keyboard customer co-buy record llm they are both electronic devices low typicality they are both of white color hallucination intention very useful to computer users functionalities both ergonomic and eco-friendly grounded lvlm intention figure 1 : examples showing the process of distilling purchase intentions from large language models and large vision-language models. without product images, large language models tend to generate intentions with low typicality and hallucinated facts, while leveraging large vision-language models resolve such issue. to combat this, @xcite proposed to distill purchase intentions from large language models, such as opt @xcite , by prompting them with real purchasing records and relevant product metadata. human-in-the-loop annotations are also carried out to verify the plausibility and typicality of the generated intentions and train a discriminator for large-scale critic filtering. @xcite further entangled human annotations with instruction tuning to align the distilled intentions with a human-centric perspective. while these works provide a straightforward approach to intention acquisition, several limitations still persist. first, previous works on e-commerce intention knowledge base construction have solely focused on the text modality, thereby sacrificing significant supervision signals from visual modalities, such as product images. this oversight hinders the model from obtaining a more comprehensive understanding of the product, consequently compromising the quality of the generated intentions, as demonstrated in the left lower part of figure 1 . furthermore, recent work has shown that intentions derived using current distillation methods exhibit bias towards product-centric aspects, excessively emphasizing product properties and metadata @xcite . consequently, interactions between the products and customers, including potential use cases and features of interest to customers, are absent from the derived intentions, despite being fundamental in facilitating customers' shopping experience. finally, human annotations are heavily deployed in current intention collection methods, which serve as a critical step in controlling the quality of the generated results. this poses a challenge towards constructing scalable yet diverse intention knowledge bases with minimum human supervision cost. to address these issues, we propose mind, a multimodal shopping intention distillation framework. mind instructs large vision-language models (lvlms) to generate purchase intentions in a three-step manner, based on real user co-buy records and product metadata. specifically, we select llava @xcite as a representative lvlm and incorporate both visual information from the product images and text information from the product name into the generation process. to better align the generated raw intentions with human preferences and alleviate human annotation costs for further quality control, we propose a human-centric role-aware mechanism. this mechanism first instructs llava to discover similar features between the products and then imitates a customer agent to decide whether the products would be bought together under previously generated intentions. by applying mind to a subset of the amazon review dataset @xcite , we construct a multimodal intention knowledge base. it features 1.26 million of intentions over 126,142 co-buy shopping records across 107,215 products. human evaluations further confirm: (1) the exceptional quality of our generated intentions, which have higher plausibility and typicality than previous generation methods, and (2) the effectiveness of our proposed human-centric role-aware mechanism. furthermore, we apply our generated intentions to two downstream tasks in the intentionqa benchmark @xcite , which evaluates a language model's abilities to discriminate and utilize purchase intentions. extensive experiments show that distilling our generated intentions into large language models' provide substantial benefits on both tasks via fine-tuning. further ablation studies reveal the importance of incorporating visual cues of products in mind and the necessity of integrating our proposed role-aware filter mechanism. moreover, analyses demonstrate the remarkable diversity of mind's intentions and the exceptional robustness of mind when using different prompts. ## intrinsic evaluations by applying mind to products we collected from amazon reviews @xcite , we construct a multimodal intention knowledge base, with statis-tics shown in ## experiments and analyses in this section, we first study the downstream benefits brought by intentions generated by mind. then, we conduct in-depth analyses to demonstrate the advantages of multimodal generation in mind compared to generating only with textual information, the superior capability of the human-centric role-aware filter in comparison to other filtering measures, knowledge diversity in mind generations, and its robustness when generating with different prompts. ## conclusions in this work, we present mind, a multimodal distillation framework for enhancing e-commerce purchase understanding by automating the pipeline of intention generation and quality filtering via multiple-step instructions over lvlms. by applying mind to real-world e-commerce data, we construct the very first multimodal purchase intention knowledge base featuring over 1.2 million intentions. these intentions have been proven to be invaluable in distilling student models that exhibit improved performance in e-commerce intention comprehension and utilization tasks. further analyses reveal the effectiveness of mind by validating the proposed filtering mechanism and highlighting the strengths of mind in comparison to folkscope.