Text-like Encoding of Collaborative Information in Large Language Models for Recommendation (2024)

Abstract: When adapting Large Language Models for Recommendation (LLMRec), it is crucial to integrate collaborative information. Existing methods achieve this by learning collaborative embeddings in LLMs’ latent space from scratch or by mapping from external models. However, they fail to represent the information in a text-like format, which may not align optimally with LLMs. To bridge this gap, we introduce BinLLM, a novel LLMRec method that seamlessly integrates collaborative information through text-like encoding. BinLLM converts collaborative embeddings from external models into binary sequences — a specific text format that LLMs can understand and operate on directly, facilitating the direct usage of collaborative information in text-like format by LLMs. Additionally, BinLLM provides options to compress the binary sequence using dot-decimal notation to avoid excessively long lengths. Extensive experiments validate that BinLLM introduces collaborative information in a manner better aligned with LLMs, resulting in enhanced performance. We release our code at https://github.com/zyang1580/BinLLM.

## introduction due to the remarkable power of large language models (llms), there is a growing focus on adapting them for recommender systems (llmrec), which has seen significant progress in the past year @xcite @xcite . in recommendation, collaborative information, which delineates the co-occurrence patterns among user-item interactions, has emerged as a pivotal component in modeling user interests, especially for active users and items @xcite . however, this information exists in a different modality from textual data and thus presents a challenge in directly leveraged by llms like textual information @xcite @xcite . to enhance recommendation quality, it is undoubtedly crucial to seamlessly integrate collaborative information into llms. to date, two integration strategies have emerged. the first strategy resembles latent factor models @xcite by incorporating additional tokens and corresponding embeddings into llms to represent users and items, subsequently fitting interaction data to implicitly capture collaborative information within the embeddings @xcite . however, this approach suffers from low learning efficacy due to the inherent low-rank nature of the information, leading to tokenization redundancy within llms @xcite . to address these challenges, an alternative approach leverages an external latent factor model to capture the information, which is then mapped into the llm token embedding space @xcite @xcite , circumventing the need to learn it from scratch. while effective, this method introduces the additional overhead of training the mapping model. whether learning collaborative information directly from scratch in the llm token embedding space or mapping it from external models, the resulting representations diverge significantly from the llm's original textual-level encoding. this, to a certain extent, hampers the full utilization of llms' capabilities, as llms are initially trained on textual data and excel at processing textually encoded information. for instance, introducing new tokens alters the generative space of llms, potentially compromising their original functionalities, let alone capitalizing on their capabilities. therefore, exploring text-like encoding of collaborative information in llms holds immense promise. nevertheless, it poses challenges due to the inherent differences between textual and collaborative information modalities @xcite . in this study, we delve into the central theme of encoding collaborative information in llms for recommendation, an area of promise yet not explored in llmrec. the crux lies in transforming collaborative information into a sequence formatted like text. we believe this text-like sequence need not be comprehensible to humans; rather, it should be interpretable by llms for effective utilization, such as facilitating reasoning tasks like discerning user and item similarities through sequence comparisons. thus, this text sequence does not necessarily have to adhere to conventional natural language patterns. to this end, we introduce binllm, an innovative llmrec approach that integrates collaborative information into llms using a text-like encoding strategy. we transform the collaborative embeddings obtained from external models into binary sequences, treating them as textual features directly usable by llms. this design is motivated by two primary considerations: 1) the feasibility of binarizing collaborative embeddings without compromising performance @xcite ; 2) llms can naturally perform bitwise operations or do so after instruction tuning @xcite , enabling the comparison of similarities between binarized sequences. taking a step further, we explore representing the binary sequence in dot-decimal notation @xcite , resulting in shorter representations, akin to converting binary sequences to ipv4 addresses. by fine-tuning llms with recommendation instruction data containing such encoded collaborative information, we could leverage both textual semantics and collaborative data for recommendation without modifying the llms. the main contributions of this work are summarized as follows: ## methodology in this section, we introduce our binllm method, starting with presenting the model architecture and ## experiments in this section, we conduct experiments to answer the following research questions: rq1: does binllm effectively incorporate collaborative information into llms to improve recommendation performance? how does its performance compare with that of existing methods? rq2: how do our design choices influence the performance of the proposed method binllm? ## related work researchers have initiated endeavors to incorporate collaborative information into llms. some researchers attempt to look for id encoding methods to introduce new tokens through vocabulary expansion and train these tokens from scratch @xcite @xcite . among them, @xcite employ vector quantization techniques. however, this approach often faces with low learning efficacy. another group of researchers explores using a latent factor model to capture collaborative information @xcite @xcite , which is then mapped onto the semantic space of llms through a mapping layer. this method exhibits better learning efficacy but requires additional training of the mapping layer. moreover, due to the non-text-like format of collaborative information, both sets of methods face challenges in aligning with the information processing mechanism in llms, limiting their performance. ## conclusion in this study, we emphasize the importance of textlike encoding of collaborative information modeling to enhance recommendation performance for llmrec. we introduce binllm, a novel approach designed to incorporate collaborative information in a text-like format by binarizing collaborative embeddings for llmrec. this encoding allows the collaborative information to be utilized in a manner better aligned with how information is processed in llms. extensive results demonstrate the superiority of binllm.