Enhancing High-order Interaction Awareness in LLM -based Recommender Model (2024)

Abstract: Large language models (LLMs) have demonstrated prominent reasoning capabilities in recommendation tasks by transforming them into text-generation tasks. However, existing approaches either disregard or ineffectively model the user-item high-order interactions. To this end, this paper presents an enhanced LLM-based recommender (ELMRec). We enhance whole-word embeddings to substantially enhance LLMs’ interpretation of graph-constructed interactions for recommendations, without requiring graph pre-training. This finding may inspire endeavors to incorporate rich knowledge graphs into LLM-based recommenders via whole-word embedding. We also found that LLMs often recommend items based on users’ earlier interactions rather than recent ones, and present a reranking solution. Our ELMRec outperforms state-of-the-art (SOTA) methods, especially achieving a 124.3% to 293.7% improvement over SOTA LLM-based methods in direct recommendations. Our code is available online.

## introduction large language models (llms) with powerful reasoning capabilities have attracted considerable attention in recommendations @xcite @xcite . they typically exploit prompt learning @xcite to integrate user and item ids into llms, including discrete prompts, which leverage alternative words (e.g., a movie title) to represent ids @xcite , continuous prompts to directly fine-tune id vectors @xcite , and hybrid prompts @xcite . furthermore, several attempts @xcite @xcite convert multiple recommendation tasks into generation tasks via textual prompts for training llms. as illustrated in fig. 1 (a), these tasks are (i) direct recommendation, (ii) sequential recommendation, and (iii) explanation generation based on user reviews. this enables llms to understand diverse perspectives in behavioral semantics for recommendations. however, there still remain two major drawbacks for these llm-based recommenders. one is their inability to learn the high-order collaborative signal within the user-item interaction graph, even though this signal could greatly assist models which identify the target from millions of candidates. as depicted in fig. 1 (b), graph neural networks (gnns), by propagating node embeddings via user-item interaction edges, have been widely utilized to capture these signals for recommendations @xcite @xcite @xcite . more recently, researchers have exploited llms to enhance graph learning for recommendations @xcite @xcite . nevertheless, integrating the learning process of gnns into llms for recom-mendations still poses an unresolved challenge. another issue with llm-based recommenders is their struggle with token decomposition, where an id token (e.g., "user_1234") is often split into multiple tokens by llms (e.g., "user", "_", "12", and "34"). to address the issue, @xcite and @xcite propose representing each user or item (e.g., "user_1234") with both its id tokens (e.g., "user", "_", "12", and "34") and wholeword embeddings. each id's tokens are covered by one whole-word embedding vector, while all other non-id tokens share an identical embedding vector. this can emphasize the self-attention correlations among tokens representing the same user or item. however, such whole-word embeddings cannot independently represent users and items, failing to address spurious relatedness among their ids, e.g., "1234" and "8912" are independent, while they share the same token "12". motivated by the issues, we propose an enhanced llm-based recommender model @xcite . we present novel whole-word embeddings using random feature propagation @xcite , which remarkably enhance high-order interaction awareness of llms for recommendations. this is inspired by the fact that gnns propagate node embeddings to their neighbors, leading them to convey similar features @xcite , which can reflect their relationships in the interactive graph. this also mitigates the spurious relatedness between ids, as each id is represented by both its tokens and their corresponding whole-word embeddings. our approach allows llms to seamlessly absorb high-order interactive signals for recommendations without requiring graph pre-training. furthermore, llm-based recommenders may prioritize predicting the next items based on earlier interactions rather than recent ones. we attribute this to two factors: (1) during training, they repeatedly sample random segments from interaction sequences to predict the last item in each segment, potentially causing them to overly emphasize past interactions; and (2) the transformer @xcite in llms excels at capturing long-term dependencies among historical interactions, which potentially leads llms to pay less attention to recent ones. hence, we introduce a simple, effective, and training-free reranking approach. the main contributions of our work are summarized as follows: (1) we present a novel whole-word embedding approach, which significantly boosts the ability of llms to capture the relative positions of users and items within the interaction graph. this finding may inspire endeavors to incorporate knowledge graphs into llms via whole-word embedding for recommendations; (2) we demonstrate that llms might recommend items based on users' past interactions and propose a plug-andplay reranking solution; and (3) extensive experiments demonstrate that emlrec achieves stateof-the-art (sota) performance in both direct and sequential recommendations. ## approach we present our elmrec from both direct and sequential recommendation perspectives. ## conclusion this paper presents elmrec, an enhanced llmbased recommender model. we introduce novel whole-word embeddings to improve the interaction graph awareness of llms and a reranking approach to prevent llms from prioritizing earlier interactions. extensive experiments show the effectiveness of elmrec in both direct and sequential recommendations. future work involves (i) exploring computationally efficient gnns @xcite and (ii) investigating more effective and learnable reranking approaches.