I nstruct PTS : Instruction-Tuning LLM s for Product Title Summarization (2023)

Abstract: E-commerce product catalogs contain billions of items. Most products have lengthy titles, as sellers pack them with product attributes to improve retrieval, and highlight key product aspects. This results in a gap between such unnatural products titles, and how customers refer to them. It also limits how e-commerce stores can use these seller-provided titles for recommendation, QA, or review summarization. Inspired by recent work on instruction-tuned LLMs, we present InstructPTS, a controllable approach for the task of Product Title Summarization (PTS). Trained using a novel instruction fine-tuning strategy, our approach is able to summarize product titles according to various criteria (e.g. number of words in a summary, inclusion of specific phrases, etc.). Extensive evaluation on a real-world e-commerce catalog shows that compared to simple fine-tuning of LLMs, our proposed approach can generate more accurate product name summaries, with an improvement of over 14 and 8 BLEU and ROUGE points, respectively.

## related work pts falls within the broader domain of text summarization techniques @xcite . both extractive and abstractive summarization approaches have been applied for pts. for example, @xcite propose a multi-task learning framework, where one network summarizes the product name, while another learns to generate search queries. @xcite propose a multi-source pointer network to generate short product names from longer input names and background knowledge. @xcite developed an enhanced feature extraction approach to generate short product names by incorporating external word frequency information and named entities as additional features. an different approach based on generative adversarial networks that encode multi-modality features (such as product images and attribute tags) is presented by @xcite . @xcite adopt bi-lstms to extract key words for product name summaries. subsequently, @xcite tackled the vocabulary mismatch problem by integrating pretrained embeddings with trainable character-level embeddings as inputs to bi-lstms. an adversarial generation model that can generate personalized short names is proposed by @xcite . our approach differs from prior work in two aspects. firstly, previous studies primarily focused on generating a single product name summary, which may not cater to the diverse use cases in ecommerce applications. in contrast, our approach offers the flexibility to generate diverse summary types (e.g. specific number of words, specific summary specificity etc.). secondly, drawing inspiration from the recent success of llms @xcite , we are the first to propose an instruction-based approach for pts. ## instructpts approach we now outline our proposed instructpts approach: we describe the base model, and provide details about the instruction fine-tuning. ## automatic evaluation results text generation performance: a consistent pattern is that as model size increases, so do the bleu and rouge metrics. for instance, flan-t5-xl improves by roughly 5 bleu1 points over flan-t5-base (for all strategies). we note a similar trend for rougel. impact of training strategy: training strategy has a significant impact. for the same model size, instructpts models obtain the best performance, e.g. instructpts with flan-t5-xl obtains an improvement of 13.3 bleu1 points over the sft and cc models. finally, we note a convergence between cc and sft for the flan-t5-xl models, with near identical performance. our results show the advantages of instruction tuning for pts. instruction following: summary length: ## human evaluation study to address the known limitations of automatic summarization evaluation, we perform a human study. we aim to answer the following questions: h1: in a pairwise comparison, which model generates better product name summaries? h2: are the generated summaries valid? h3: what is the preferred summary length by humans for a given product name? ## extrinsic evaluation with retrieval we have shown that instructpts can robustly summarize titles, following instructions for length and phrasal inclusion (cf. §3). to assess the fidelity of the summarized titles, we perform a retrieval-based extrinsic evaluation to determine how well the original products can be retrieved by using the summary titles. we hypothesize that a good summary with retain enough of the unique characteristics of the original product to be able to retrieve it. additionally, this evaluation analyzes the trade-offs between summary length vs. ranking metrics of a target product under consideration. setup: we use a catalog of 5m products as our testbed. the product titles are summarized using instructpts (flan-t5-xl) with different instructions. the summary titles are then used as queries to review the top-k products in the catalog index using the bm25 algorithm. we also use the original title as an upper bound. evaluation: evaluation is performed with standard ir metrics, mean reciprocal rank (mrr) and hit@k. higher values indicate that the summary retains more distinguishing information from the original product title. results: the mrr of 0.398 indicates that, on average, the ground-truth product is ranked in the 2nd and 3rd position. furthermore, the hit@20 score of 0.641 shows that in 64.1% of cases the groundtruth product is featured among the top 20 results. this study shows that our summaries retain key aspects that help identify the product in a set of 5m. it also provides guidance on how much the titles can be compressed. ## online deployment instructpts has been used in a leading global ecommerce service for various downstream shopping tasks. it can be applied for various content generation tasks related to product summarization, comparison, question suggestion, and review summarization. a 4k sample of generated content with embedded product titles from instructpts were evaluated for quality, and 96% were found to meet the validity criteria. ## conclusion we presented instructpts, a new approach for product title summarization, and demonstrated the effectiveness of instruction-tuning for this task. through ift we can train a highly accurate and controllable model for generating various types of summaries. empirical studies using automatic and human evaluation studies showed that the model size has a significant impact in generating reliable and meaningful summaries, and at the same time it ensures the model's ability to follow requirements specified in the instructions. instructpts has been deployed in systems where product titles from a billion-scale catalog are summarized for various downstream applications, such as question answering and summarization. future work will focus on more fine-grained instructions focusing on higher levels of specificity, and support for handling constraints based on brands/sizes/colors.