Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labels (2023)

Abstract: Generating an informative and attractive title for the product is a crucial task for e-commerce. Most existing works follow the standard multimodal natural language generation approaches, e.g., image captioning, and employ the large scale of human-labelled datasets to train desirable models. However, for novel products, especially in a different domain, there are few existing labelled data. In this paper, we propose a prompt-based approach, i.e., the Multimodal Prompt Learning framework, to accurately and efficiently generate titles for novel products with limited labels. We observe that the core challenges of novel product title generation are the understanding of novel product characteristics and the generation of titles in a novel writing style. To this end, we build a set of multimodal prompts from different modalities to preserve the corresponding characteristics and writing styles of novel products. As a result, with extremely limited labels for training, the proposed method can retrieve the multimodal prompts to generate desirable titles for novel products. The experiments and analyses are conducted on five novel product categories under both the in-domain and out-of-domain experimental settings. The results show that, with only 1% of downstream labelled data for training, our proposed approach achieves the best few-shot results and even achieves competitive results with fully-supervised methods trained on 100% of training data; With the full labelled data for training, our method achieves state-of-the-art results.

## introduction product title generation aims to comprehend the content of a given product provided by merchants, which may come in various forms such as an input product image and a set of attributes, and then automatically generate an appealing and informative title. the generated title should contain essential product characteristics, along with the product details, e.g., brand name, category, style, size, material, and colour @xcite @xcite . therefore, a desirable title can highlight the characteristics and advantages of the product, leading to time savings for consumers, enhancing their overall shopping experience, and ultimately increasing product sales. admittedly, in e-commerce, the ability to perform product title generation automatically offers the possibility of relieving merchants from the timeconsuming analysis of complex product details and writing concise and appealing titles; and alerting merchants of important product characteristics and advantages @xcite @xcite . in general, the task of product title generation can be defined as a data-to-text problem. following existing efforts on data-to-text tasks @xcite @xcite , figure @xcite shows the conventional product title generation approach: the encoder-decoder framework. the image encoder and attribute encoder respectively transform the product image and product attributes into visual and attribute representations, which the text decoder subsequently decodes into a product title. such encoder-decoder-based methods have achieved great success in advancing the state-of-the-art of various data-to-text tasks, e.g., image captioning @xcite , multimodal machine translation @xcite , and video captioning @xcite . however, these methods rely on a large volume of annotated data, which is particularly time-consuming to collect. this issue is especially severe in the e-commerce title generation scenario, where products from different categories always contain category-specific attributes. therefore, the product title generation model trained on existing products cannot be directly used on novel products, such as with new categories or new designs. nevertheless, it is difficult to collect and label sufficient training data in a timely manner, which prevents the rapid deployment of such encoder-decoder models online. as shown in figure 1(b) , we propose the multimodal prompt learning (mpl) framework, which deals with the situation where the training data is scarce. in detail, we observe that novel product title involves different domain product characteristics (e.g., category-specific attributes) and different writing styles, directly adopting a model or transferring a model pre-trained on existing available product data to novel product data will significantly degrade the performance, especially when the labelled data (i.e., image-attribute-title pairs) is insufficient in quantity @xcite . to this end, we first construct a set of multimodal prompts from different modalities, i.e., visual prompts, attribute prompts, and language prompts. during training, given the limited data of novel products (i.e., image i -attribute a -title t ), to make full use of it, mpl introduces the unimodal prompt training to enable the different prompts to preserve the corresponding domain characteristics and the writing styles of novel products from different modalities/perspectives. in implementations, (i) we introduce the visual prompts p i to train the model by generating the title t in the @xmath0; in this way, the proposed mpl framework can accurately and efficiently generate novel product titles with limited training data by 1) introducing multimodal prompts to learn domain characteristics and writing styles of novel products; 2) learning to accurately highlight the product characteristics and advantages across multiple modalities. it enables our approach to be rapidly well-adapted to the novel product domain, helping sellers save time in deploying new products, optimizing consumers' consumption experience, and thus boosting sales. the experiments and analyses on a largescale dataset, i.e., amazon product dataset @xcite , across five novel product categories prove the effectiveness of our approach. overall, the contributions are as follows: the related works are discussed from 1) product description and 2) few-shot learning. ## approach in this section, we will introduce the proposed multimodal prompt learning (mpl) method in detail. ## experiments in this section, we first describe a large-scale dataset, the widely-used metrics, and the settings used for evaluation. then, we present the results of in-domain and out-of-domain experiments. ## analysis in this section, we conduct several analyses under the out-of-domain setting to better understand our proposed approach, ## conclusion in this paper, we present the multimodal prompt learning (mpl) framework to accurately and efficiently generate titles of novel products with limited training data. our mpl introduces various prompts across different modalities to sufficiently learn novel domain characteristics and writing styles, which are aligned and exploited to generate desirable novel product titles. the out-ofdomain and in-domain experiments on a large-scale dataset across five novel product categories show that, with only 1% downstream labelled data for training, our approach achieves competitive results with fully-supervised methods. moreover, with the full training data used in previous works, our method significantly sets the state-of-the-art performance, which proves the effectiveness of our approach and shows its potential to deploy novel products online in time to boost product sales.