I ntention QA : A Benchmark for Evaluating Purchase Intention Comprehension Abilities of Language Models in E -commerce (2024)

Abstract: Enhancing Language Models’ (LMs) ability to understand purchase intentions in E-commerce scenarios is crucial for their effective assistance in various downstream tasks. However, previous approaches that distill intentions from LMs often fail to generate meaningful and human-centric intentions applicable in real-world E-commerce contexts. This raises concerns about the true comprehension and utilization of purchase intentions by LMs. In this paper, we present IntentionQA, a double-task multiple-choice question answering benchmark to evaluate LMs’ comprehension of purchase intentions in E-commerce. Specifically, LMs are tasked to infer intentions based on purchased products and utilize them to predict additional purchases. IntentionQA consists of 4,360 carefully curated problems across three difficulty levels, constructed using an automated pipeline to ensure scalability on large E-commerce platforms. Human evaluations demonstrate the high quality and low false-negative rate of our benchmark. Extensive experiments across 19 language models show that they still struggle with certain scenarios, such as understanding products and intentions accurately, jointly reasoning with products and intentions, and more, in which they fall far behind human performances.

## introduction purchase intentions are mental states where agents or humans commit themselves to purchasing the products @xcite . understanding customers' purchase intentions and making reasonable inferences accordingly are crucial for revolutionizing e-commerce services, whose benefits have been demonstrated in myriads of downstream tasks, task 1 requires the language model to determine the customer's intention in purchasing two products, and task 2 involves recommending a product that fulfills the customer's intention and matches their currently purchased product. thus, an important yet under-explored question arises: can lms comprehend the customers' purchase intention and how effective are they in performing such tasks? to dive into this, we first break down the comprehension of intention into two key aspects, as shown in figure 1 . first, we have intention understanding, which evaluates lms' capacity to accurately infer customers' purchase intentions based on the products bought. second, we consider intention utilization, which investigates lms' ability to predict additional purchases based on customer's intentions. together, they make up the entire process of intention comprehension and play a significant role in enhancing e-commerce search services. although lms have been extensively used in intention knowledge distillation, their actual performances in this area have not been adequately benchmarked. this is because current methods that leverage lms have been adopting an open-ended generation fashion, which is difficult to consistently evaluate @xcite . additionally, the extensive and constantly growing number of products on e-commerce stores makes it infeasible and expensive to construct human-curated benchmarks. to address these challenges and benchmark lms on purchase intention comprehension in ecommerce, we introduce intentionqa, a doubletask multiple-choice question answering (mcqa) dataset, featuring intention understanding and intention utilization respectively. intentionqa contains 4,360 problems for two tasks and covers varying difficulty levels, allowing for fine-grained evaluation. the mcqa setting enables using consistent evaluation metrics to assess the lms' intention comprehension abilities. specifically, we design a pipeline that automatically synthesizes qa pairs by transforming humanannotated intentions from folkscope @xcite , each involving a pair of co-buy products and the corresponding intention of purchasing them, into questions by masking out the intention or one of the products. to achieve this, we define contextbased product similarity and intention similarity metrics. they are computed over aser @xcite , a large-scale eventuality knowledge graph, which we leverage as a reference for our automatic distractor sampling strategy. for each question, we include 3 negative distractors alongside the gold answer through a strict similarity filtering process. we then assign difficulty labels to each qa pair based on the product similarity between the co-buy products in the original intention assertion. these steps are done without human supervision, enabling our benchmark construction pipeline to generalize and accommodate larger-scale product databases and practical applications. we further conduct human evaluations to demonstrate the high quality and low false-negative rate of intentionqa, followed by extensive experiments across 19 language models with varying sizes and approaches. results demonstrate that the existing language models still struggle with certain scenarios, such as understanding products and intentions accurately, jointly reasoning with the products and intentions, and more. in the long run, we hope that our benchmark serves as an important cornerstone toward intention-aware e-commerce services that promote integrating intention reasoning abilities into product recommendations. ## intentionqa in this section, we introduce intentionqa, a multiple-choice question answering (mcqa) benchmark consisting of two tasks targeting different aspects of purchase intention comprehension and with progressive difficulties, to evaluate the intention understanding and utilization abilities of lms thoroughly. ## conclusions in conclusion, this paper presents intentionqa, a double-task mcqa dataset designed to assess the intention comprehension capabilities of lms. extensive experiments and analyses demonstrate that lms face significant challenges in certain scenarios, trailing far behind human performance levels, while fine-tuning on external resources brings considerable performance gains. we hope our work sheds light on the limitations of current lms in e-commerce intention understanding tasks and opens up a new paradigm of leveraging lm in ecommerce services.