IPL : Leveraging Multimodal Large Language Models for Intelligent Product Listing (2024)

Abstract: Unlike professional Business-to-Consumer (B2C) e-commerce platforms (e.g., Amazon), Consumer-to-Consumer (C2C) platforms (e.g., Facebook marketplace) are mainly targeting individual sellers who usually lack sufficient experience in e-commerce. Individual sellers often struggle to compose proper descriptions for selling products. With the recent advancement of Multimodal Large Language Models (MLLMs), we attempt to integrate such state-of-the-art generative AI technologies into the product listing process. To this end, we develop IPL, an Intelligent Product Listing tool tailored to generate descriptions using various product attributes such as category, brand, color, condition, etc. IPL enables users to compose product descriptions by merely uploading photos of the selling product. More importantly, it can imitate the content style of our C2C platform Xianyu. This is achieved by employing domain-specific instruction tuning on MLLMs, and by adopting the multi-modal Retrieval-Augmented Generation (RAG) process. A comprehensive empirical evaluation demonstrates that the underlying model of IPL significantly outperforms the base model in domain-specific tasks while producing less hallucination. IPL has been successfully deployed in our production system, where 72% of users have their published product listings based on the generated content, and those product listings are shown to have a quality score 5.6% higher than those without AI assistance.

## introduction with the rise of the circular economy, secondhand e-commerce has played a vital role in our daily lives. unlike business-to-consumer (b2c) * these authors contributed equally to this work. figure 1 : intelligent product listing on c2c platforms e-commerce (e.g., amazon, walmart), secondhand e-commerce is often operating in the form of consumer-to-consumer (c2c) transactions. different from professional sellers on b2c platforms, individual sellers in second-hand marketplaces are usually inexperienced. they face unique challenges when listing their products -navigating through the complicated listing procedure, and creating high-quality product descriptions. these issues not only affect the success rate of product listings but also impact the overall quality and discoverability of the listed products. to address the above issues, it is imperative to simplify the listing process for individual users by leveraging automation to generate high-quality product descriptions. a typical product listing process involves users manually filling in basic product attributes, uploading product photos, and composing content descriptions. among these steps, preparing product photos is relatively straightforward. if we can automatically generate product descriptions based on the uploaded photos, it would significantly reduce the listing effort and enhance user experience, as illustrated by figure 1 . fortunately, product photos contain a wealth of information, enabling us to infer basic attribute information such as category, brand, and model from the imagery in most cases. moreover, recent advancements in multimodal large language models (mllms) @xcite have significantly improved both visual understanding and natural language generation capabilities, making it feasible to generate product descriptions based on product photos in an automatic manner. several large e-commerce platforms, including ebay @xcite and amazon @xcite , have begun to explore this direction by introducing product listing assistants. however, these tools are still in their infant stages. they still require substantial user input, and the generated content is commonly in the professional marketing styles which lowers the information authenticity for a c2c platform. in the context of second-hand e-commerce, we encounter more challenges. lack of domain knowledge. to generate highquality product descriptions, models must possess strong capabilities for domain understanding @xcite . c2c ecommerce differs from traditional b2c platforms, its product listings often exhibit more unique and varied characteristics. unlike professional marketing descriptions that emphasize persuasive language, product descriptions in c2c platforms typically exhibit a more colloquial style, focusing on information authenticity. this helps foster trust between buyers and sellers and potentially facilitates transactions. however, existing mllms often fall short in these areas. hallucination problem. ideally, users only need to upload a photo, and the corresponding content description including core product attributes is automatically generated. however, achieving this goal imposes a significant challenge on the current mllms @xcite . in practice, mllms sometimes produce product attributes going beyond the image itself. this is known as the hallucination problem in large language models(llms). as the core part of the product listing experience, we need to find a proper solution. challenges for production deployment. deploying generative llms on production systems, particularly for applications with a large-scale user base, imposes high requirements on system latency, cost consumption @xcite , and content safety @xcite . meeting these demands necessitates a comprehensive system engineering effort. to address the above issues, we develop an intelligent product listing (ipl) system, aiming to improve the efficiency and effectiveness for product listings on our production system. firstly, we present a notable case study of injecting domain knowledge into a mllm through further instruction tuning of an open-source model. our domain-specific model significantly enhances the base model's understanding of domain knowledge and enables it to generate product descriptions in the unique style characteristic of c2c platforms. secondly, we introduce an innovative multimodal retrieval-augmented generation (rag) approach for visual-based content generation, leveraging identical product retrieval, to enhance description quality and mitigate hallucination risks in practical applications. finally, we have successfully deployed the system in an online environment, delivering services to real-world individual users. this system demonstrates high user acceptance and effectively enhances the efficiency and quality of product listings. our extensive empirical studies demonstrate that ipl has the potential to transform the landscape of product listings, offering a robust, scalable solution to challenges faced by individual sellers and platforms alike. ## approach the overall architecture of our intelligent product listing system can be illustrated in figure 2 , which comprises an online multi-modal retrieval-augmented generation (rag) process for identifying similar products, and an offline-trained domainspecific mllm for product description generation. in our product listing system, user-uploaded photos undergo category prediction, retrieval of similar products, and extraction of key attributes (e.g., brand, model, etc.) from the descriptions of these similar products. subsequently, the product photo, category, and extracted attributes are fed into the domain-specific mllm as contextual information to generate the product description. with this automatically generated description, users only need to make minimal adjustments to complete the product listing. ## deployment key considerations for llm deployment included minimizing online latency, ensuring user experience, and addressing safety risks associated with content generation. we deployed the system online, with the llm model hosted on nvidia® tesla® v100 machines. through various acceleration techniques, such as model quantization, vit operation optimization, key-value caching, kernel operation fusion, and parallel computation @xcite @xcite , the overall pipeline's average response time (rt) was reduced from 5 seconds to below 3 seconds. the adoption of streaming output ensured user experience by reducing wait times.we perform preemptive risk assessment on user-uploaded product photos and security checks on generated descriptions to prevent non-compliant content, thereby effectively avoiding public opinion risks. for more detailed error detection and exception handling, please refer to appendix d. ## results in the following subsections, we discuss the five key research questions regarding our domain model and the online rag system: ## conclusion we presented ipl system, a novel framework that generates high-quality, accurate product descriptions based on images, enhancing item listing efficiency in the c2c market. by leveraging mllms trained via domain injection, our model gains deeper domain-specific knowledge and style compared to the original model (qwen-vl). the implementation of online rag, which uses similar product images as reference, reduces hallucination in mllms, resulting in more precise descriptions. the effectiveness of our framework is demonstrated through human evaluations, machine assessments, and online a/b testing.