A Multi-modal Large Language Model with Graph-of-Thought for Effective Recommendation (2025)

Abstract: Chain-of-Thought (CoT) prompting has been shown to be effective in guiding Large Language Models (LLMs) to decompose complex tasks into multiple intermediate steps, and constructing a rational reasoning chain for inferring answers. However, the linear nature of CoT falls short from enabling LLMs to effectively handle graph structures, which are essential for personalized recommendation tasks that rely on user-item interaction graphs. To bridge this gap, we introduce GollaRec, which leverages a Graph-of-Thought (GoT) prompting technique in a Multi-modal LLM, namely LLaVA, to effectively exploit the complex structure of the interaction graphs. GollaRec enhances the recommendation effectiveness by integrating both visual and textual “thoughts” into a graph-structured prompt, using both item images and descriptions to produce richer multi-modal user/item representations. In our proposed approach, GollaRec leverages text-graph alignment and graph instruction tuning to allow the Multi-modal LLM to capture complex graph structures. In addition, GollaRec leverages a graph adaptor to integrate user-item interactions into the resulting user/item embeddings, therefore effectively adapting the model to the recommendation task. Our extensive experiments on 6 benchmark datasets demonstrate the superiority of our proposed GollaRec model over 12 existing state-of-the-art models in various multi-modal recommendation tasks, including general and multi-domain recommendation tasks.

## introduction large language models (llms) have demonstrated a remarkable capability in language understanding and text generation in various realworld scenarios @xcite . however, their training on unstructured data often limits their capability in handling complex tasks necessitating complex, multi-step reasoning or a precise contextual understanding @xcite . this limitation becomes particularly important with graph-structured data, which is essential in fields such as social network analysis, drug discovery, and notably recommender systems @xcite . indeed, graph data, with its complex relational structures between node entities, poses a unique challenge for llms, which typically do not encounter structured data formats like columnindexed records during their pre-training, leading to difficulties in handling domain-specific knowledge inherent to such data @xcite . recent advances in prompting techniques have enhanced the lmms' capability to address complex reasoning tasks @xcite . for example, @xcite employed few-shot in-context learning to enhance the reasoning capabilities of an llm by using input and output examples as prompts. @xcite and @xcite enhanced the llms' effectiveness by using chainof-thought (cot) prompting, which involves a series of demonstrations where each step of the detailed, step-by-step explanation, serves as an instructive example to guide reasoning processes. despite its effectiveness in linear textual reasoning, cot does not inherently extend to tasks involving structural graph data, which necessitate mining complex relational structures @xcite . this identified limitation, henceforth denoted as insufficient graph mining, emphasises the need for enhancing llms to effectively tackle graph-related tasks. the graph data, especially user-item interaction graphs in recommender systems, encapsulates unique patterns that contain domain-specific knowledge. to bridge this gap, we integrate graph data into the prompt by linearising the structured data into textual sentences, thereby addressing the insufficient graph mining problem in cot. in addition to the problem of insufficient graph mining in cot, the language-based reasoning process is often overly complex and abstract (huang and chang, 2023). instead, the use of the item images can be an intuitive medium in recommendation tasks. indeed, integrating an image in a prompt not only enriches the modelling of the user profiles but also enables the used model to generate more coherent outputs. in this paper, we use a multi-modal large language model (mllm), namely llava @xcite , to ensure that the user/item embeddings are semantically coherent. to prompt effective reasoning in presence of multi-modality and a graph structure, we propose a multi-modal graph-of-thought (got) prompting technique, as illustrated in figure 1 , which can be used in two distinct recommendation tasks: a general recommendation task and a multi-domain recommendation task. to enable an effective understanding of the graph knowledge in the used mllm, we first perform a text-graph alignment to align the item textual embeddings and the corresponding item node embeddings in the semantic space and perform a graph instruction tuning on the mllm so as to match each graph token with its textual description. on the other hand, recent recommendation approaches employing llms for various recommendation tasks encounter a key challenge, namely the constraint of input token length in these llms @xcite . this limitation also restricts the amount of graph information that can be integrated within a got prompt. hence, in order to adequately integrate sufficient graph information within a got, we pre-train a recommender @xcite , specifically designed to select the maximum number of high-potential items within the input token limit. our contributions can be summarised as follows: (1) we propose graph-of-thought llava for recommendation (gollarec), a new recommendation model integrating the user-item graph information within graph-of-thought (got). to the best of our knowledge, this is the first work to apply got prompting specifically tailored to enhance multi-modal recommendation tasks, so as to effectively address the problem of insufficient graph mining in cot; (2) to effectively integrate the user-item graph information within got, we use a text-graph alignment method and graph instruction tuning to effectively capture the graph patterns; (4) to address the limited token length problem in the mllm, we use a pre-trained recommender to feed a maximum number of high-potential items into got; (5) our extensive experiments on six benchmark datasets show that gollarec significantly outperforms 12 strong baselines across two recommendation tasks. ## gollarec we first describe in section 2.1 the two top-k multimodal recommendation tasks we tackle in this paper, namely a general recommendation task and a multi-domain recommendation task. section 2.2 presents our proposed got technique, which includes three key parts (adaptive graph truncation, text-graph alignment and text-image alignment). next, we introduce the architecture of gollarec in section 2.3. the model is illustrated in figure 2 . ## related work there are two main bodies of related work, namely the use of llms in recommender systems, and leveraging chain-of-thought prompting approaches. llm for recommendation: recent recommendation approaches have used llms as inference models by designing prompts tailored to recommendation tasks @xcite @xcite . for example, p5 @xcite employed a pre-trained t5 model to adapt the recommendation tasks into natural language processing scenarios using personalised prompts. lmrecsys @xcite converted the user-item interactions into textual prompts using item indexes. by doing so, they reformulated the recommendation task as a language modelling task. unlike existing approaches, we construct a prompt that incorporates the user-item graph and item images in an mllm, thereby providing richer contextualised information. we also devise cot-like step-by-step demonstration prompts, each serving as an example for the reasoning process in the recommendation tasks. however, the use of llms for recommender systems often encounter difficulties in integrating the abundant user-item interactions. this is mainly due to the constraint of fixed input token length in the llms, which hinders their effectiveness in recommendation scenarios @xcite . to address this problem, we propose an adaptive truncation method that selectively maintains the most informative interactions within our devised got prompt. in particular, we integrate an initial ranking list, determined by a pre-trained recommender, into the step of the got's demonstration, thereby ensuring that the list length fits the token length constraint. chain-of-thought: chain-of-thought (cot) is a prompting technique that improves large language models (llms) in domains requiring logical reasoning @xcite , for example in mathematical reasoning @xcite . while chain-of-thought (cot) is effective in linear textual reasoning, it becomes less useful with graph-related tasks @xcite , such as in recommendation tasks, which involve complex user-item interaction graphs. to address this problem of insufficient graph mining in typical cot setups, we introduce the graph-of-thought (got) technique in our proposed gollarec model, which is specifically tailored for recommender systems. in particular, we leverage a text-graph alignment method to capture the relationships inherent between the item textual embeddings and the corresponding node embeddings. additionally, we apply graph instruction tuning on an mllm to effectively integrate user-item graph data in the recommendation tasks. ## conclusions in this paper, we introduced gollarec, a novel multi-modal large language model (mllm) for both general and multi-domain recommendation tasks. specifically, gollarec consists of an mllm (i.e. llava), which handles the interaction graph information within a newly designed graph-of-thought (got) prompt, and a graph adapter to incorporate domain-specific knowledge for both recommendation tasks. additionally, we devised an adaptive graph truncation method to maximize the number of high-potential items inputted into the got, thereby addressing the limited token length in the used mllm. our extensive experiments on 6 public benchmark datasets showed that gollarec significantly outperformed 12 strong existing baselines for both tacked recommendation tasks. the performance improvement reaches up to 18.2% in comparison to the strongest baseline model, bm3, on the used datasets. we also conducted an ablation study, which confirmed the significant contributions of the components of gollabrec, namely the got technique, the graph adapter, the text-graph and text-image alignment methods, to a more effective multi-modal recommendation.