The Whole is Better than the Sum: Using Aggregated Demonstrations in In-Context Learning for Sequential Recommendation (2024)

Abstract: Large language models (LLMs) have shown excellent performance on various NLP tasks. To use LLMs as strong sequential recommenders, we explore the in-context learning approach to sequential recommendation. We investigate the effects of instruction format, task consistency, demonstration selection, and number of demonstrations. As increasing the number of demonstrations in ICL does not improve accuracy despite using a long prompt, we propose a novel method called LLMSRec-Syn that incorporates multiple demonstration users into one aggregated demonstration. Our experiments on three recommendation datasets show that LLMSRec-Syn outperforms state-of-the-art LLM-based sequential recommendation methods. In some cases, LLMSRec-Syn can perform on par with or even better than supervised learning methods. Our code is publicly available at https://github.com/demoleiwang/LLMSRec_Syn.

## introduction motivation. large language models (llms) are known to perform well as a zero-shot solution for many natural language processing tasks @xcite @xcite . recently, there are some works that focus on using llms to perform recommendation with promising accuracies @xcite @xcite @xcite and to provide explanations @xcite . most of these works developed llm prompts for zeroshot sequential recommendation. to investigate whether llm can serve as a strong zero-shot sequential recommender, @xcite devised a prompt that is filled with historical items in chronological order, candidate items, and instruction to rank the candidate items. @xcite proposed a threestep prompting method, where llms first summarizes the user preference based on the user's past interacted items. it then identifies representative items from the past interacted items that capture the user preference, and finally recommends items among the candidate items which are aligned with the representative items. among the very few oneshot sequential recommendation works, @xcite and @xcite explored in-context learning using the test user's second last item as the ground truth next-item and all earlier interacted items as input to create self-demonstrations. nevertheless, previous experiments have shown that in-context learning (icl) based sequential recommendation methods perform poorly compared with the supervised learning-based methods (e.g., sas-rec) due to the complex recommendation task definition @xcite @xcite . the illustrative comparison of these two methods is shown in figure 1 . to develop an effective in-context learning approach for llms to perform sequential recommendation, we first define the sequential recommendation problem as follows. problem definition. we denote each input user instance u i to be a (x i , c i , y i ) tuple where x i denotes the sequence of past interacted items (excluding y i ) by u i , c i denotes the candidate items to be recommended (|c i | = m ), and y i denotes the ground truth next-item which is also the last item interacted by u i . note that y i appears in c i (y @xmath0. a llm-based sequential recommendation method is required to assign a rank rank(d) ∈ [1, m ] to each item d in c i . our objective is to ensure that the method ranks y i , i.e., rank(y i ), as high as possible for all. the above definition includes c i as input as it is usually infeasible for llms to take all items as input due to limited prompt length. moreover, having c i does not introduce bias in the evaluation. the above definition is also adopted in @xcite . we also assume a dataset of users' interacted item sequences from which we can construct demonstrations for icl, and a llm which is too large for pretraining or finetuning. overview of our study. past works has shown that the effectiveness of icl in adapting llms to new tasks is significantly influenced by instruction wording @xcite , label design @xcite , selection of demonstrations @xcite @xcite , and number of demonstrations @xcite ; @xcite . our study thus begins by systematically investigating how the instruction format, task consistency (between test and demonstration), demonstration selection, and the number of demonstrations affect icl-based sequential recommendation. through our preliminary experiments, we obtain four findings including the one that observes degradation of recommendation accuracy when the number of demonstrations increases. as each demonstration takes up significant length, it is also easy for multiple demonstrations to exceed the prompt limit of llms. moreover, as llms are known to miss out relevant information in a long input prompt @xcite , we thus embark on a follow-up study on designing a more efficient icl scheme based on aggregated demonstration. figure 2 shows a comparison of the frameworks for zero-shot, few-shot, and aggregated one-shot llm-based sequential recommender systems. the key idea in aggregated demonstration is to combine multiple training users into one demonstration. this reduces the repetition of instruction text in the icl prompt. it also seeks to summarize multiple training users relevant to the test instance in a compact manner. we also develop a novel icl method using aggregated demonstration for sequential recommendation known as llmsrec-syn. the length of llmsrec-syn prompt increases only gradually with number of demonstration users, llmsrec-syn can cope with more relevant information from the demonstration users within a concise input context. we finally show llmsrec-syn outperforms other zero-shot and one-shot icl methods in an extensive set of experiments. contribution. our contributions can be summarized as follows: (1) we systematically explore the icl approach to sequential recommendation by empirically investigating the effect of instruction format, task consistency, demonstration selection, and number of demonstrations; @xcite we propose a new in-context learning method for sequential recommendation called llmsrec-syn which leverages on a novel concept of aggregated demonstration; (3) we experiment on three popular recommendation datasets and show that llmsrec-syn outperforms previous llm-based sequential recommendation methods. ## related work in-context learning. several works show that llms can effectively adapt to different nlp and multimodal tasks, including machine trans-lation @xcite , visual question answering @xcite , and foreground segmentation @xcite . this adaptation is achieved by learning from a few task-relevant demonstrations, commonly known as in-context learning (icl) @xcite . despite the above successes, icl's performance is still significantly affected by the wording of instructions @xcite , label design @xcite , demonstration selection @xcite @xcite , and number of demonstrations @xcite ; @xcite . icl is much less studied in llmbased sequential recommendation. as sequential recommendation is distinct from the pretraining tasks of llms and also different from the abovementioned tasks, new designs of demonstration(s) and icl prompt is necessary. llms for sequential recommendation. early sequential recommendation works adopt techniques such as markov chains @xcite and neural networks (e.g., rnn @xcite , cnn @xcite , self-attention @xcite , and @xcite ). to investigate if llms can be used as effective sequential recommenders without training, @xcite formulated sequential recommendation as conditional ranking, employing zero-shot llm methods to reflect user preferences from past interactions and recency. @xcite developed a threestep llm prompting to summarize user preferences, while @xcite and @xcite introduced a one-shot icl method that utilizes the previous item interactions of the target user as a demonstration. to address position bias, @xcite proposed to randomize the candidate item order. in this work, we explore using training data demonstrations, not just user own history, and introduce aggregated demonstration for combining relevant users. ## what makes in-context learning work for sequential recommendation in this section, we conduct a preliminary empirical study to investigate the role of various aspects of demonstrations. these aspects include the wording of prompts, task consistency between demonstrations and test instances, selection of demonstrations, and number of demonstrations. while previous studies have explored the use of llm as sequential recommenders in a zero-shot manner @xcite , this is the first study to comprehensively discuss how in-context learning can improve sequential recommendation. ## in-context learning with aggregated demonstrations finding 4 suggests llms have difficulties coping with multiple demonstrations in sequential recommendation. a similar finding by @xcite also suggests that the current language models often struggle to utilize information in long input contexts. in particular, their performance tends to significantly degrade when the relevant information is located in the middle of long contexts, also known as the "lost in the middle" phenomenon. the in-context learning prompts for sequential recommendation can easily exceed the prompt length limit of llm when more than 4 demonstrations are to be accommodated. such prompts not only suffer from "lost in the middle", but also incur additional costs of calling llm apis. to address the above challenge, we propose aggregated demonstration which combines k (@xmath1. this simple yet effective in-context learning method for sequential recommendation is called llmsrec-syn. as the prompt length of aggregated demonstration only increases marginally when we increase k, llmsrec-syn can accommodate more member demonstration users. based on finding 3, llmsrec-syn begins with selecting k demonstration users that are similar to the test user. we use similarity between the llm embeddings of demonstration and test users. we also follow finding 1 and adopt instruction template (a) for the test user. based on finding 2, we also adopt demonstrate template (t3) for the aggregated demonstration to maintain consistency with the task for test user. next, we construct the aggregated demonstration's historical item-interactions, candidate items, and the desired ranking of the candidate items from its member demonstrations, as shown in figure 10 in the appendix. historical item-interactions. let h denote the historical item-interactions and h is empty initially. we first rank the k selected demonstration users by similarity score. we then add the most recent interacted item from the most similar demonstration to h. we repeat the same step for the remaining demonstrations in their similarity order. when we run out of most recent interacted items from k selected demonstrations, we continue to add the next recent interacted items of these demonstrations to h until the number of historical items reaches m ax h . candidate items. let c denote the candidate items of the aggregated demonstration and c is empty initially. we first gather all the ground truth next items from the k selected demonstrations and add them to c. next, we randomly add other items from the item pool to c so as to meet the required number of candidate items. ranking of candidate items. to rank the candidate items in c, we place the ground truth next item of the most similar demonstration at rank 1, followed by that of next similar demonstration until we run out of the ground truth next items of all k selected demonstrations. next, we assign random ranks to the remaining items in c. once the aggregated demonstration is constructed, it is added to the prompt the same way a training user is added as a demonstration. we add it to the corresponding test user and use them as input for the llm. there are several advantages of the proposed llmsrec-syn: 1) standard demonstration only has one ground truth next item in the ranking list. in contrast, the aggregated demonstration includes more next items at high positions in the ranking list. this approach can avoid sparse signals and provide more guidance to llms for recommending to the test user; 2) llmsrec-syn is less sensitive to the number of demonstrations; 3) cost of llmsrec-syn does not increase much with the number of demonstrations; and 4) llmsrec-syn keeps to the prompt length limit of llms. ## conclusion this paper investigates in-context learning (icl) for llm-based sequential recommendation. our study identifies key factors such as instruction format and demonstration selection that influence icl's effectiveness. we further introduce the llmsrec-syn method which utilizes our proposed aggregated demonstration to efficiently incorporate relevant information from multiple training users. tested on three datasets, llmsrec-syn consistently outperforms existing llm-based sequential recommendation methods. future work includes a detailed analysis of llmsrec-syn's unexpected success compared to some supervised methods and the optimization of aggregated demonstration strategies.