VIP 5: Towards Multimodal Foundation Models for Recommendation (2023)

Abstract: Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys) are three prominent AI applications that have traditionally developed independently, resulting in disparate modeling and engineering methodologies. This has impeded the ability for these fields to directly benefit from each other’s advancements. With the recent development of foundation models, large language models have emerged as a potential general-purpose interface for unifying different modalities and problem formulations. In light of this, we propose the development of a multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks. This will enable the processing of multiple modalities in a shared architecture for improved recommendations. To achieve this, we introduce multimodal personalized prompts to accommodate multiple modalities under a shared format. Additionally, we propose a parameter-efficient training method for foundation models, which involves freezing the P5 backbone and fine-tuning lightweight adapters, resulting in improved recommendation performance and increased efficiency in terms of training time and memory usage. Code and data of VIP5 are available at https://github.com/jeykigung/VIP5.

## introduction with rapid growth, recommender systems have gradually become an indispensable element in people's daily lives. with more time spent on the web, people reveal their interests through richer modalities than before. in response to the trend, current recommendation systems @xcite @xcite consider more diverse contents when making recommendation decisions to users. historically, the technical developments for processing different types of information (such as personalization, visual, and textual) are mostly spread across different research communities. fortunately, recent advances in foundation models (fms) such as large language models (llms) unfold a promising route for building generalpurpose models and unifying diverse modalities, so that one single architecture can handle visual, textual and personalized information at the same time, enabling the possible approach towards artificial general intelligence (agi) @xcite and artificial general recommender (agr) @xcite . as a pioneering work, gpt-3 @xcite can perform in-context learning, enabling it to solve brand-new problems given few-shot demonstration examples as prompts. similarly, clip @xcite ) maintains superior zero-shot generalization ability when shifting to an out-of-distribution visual domain if provided with appropriate prompt. with more and more emergent abilities @xcite revealed in foundation models, they become not only a popular backbone to finetune downstream tasks @xcite @xcite but also an effective training scheme for unifying multiple modalities in a shared interface @xcite @xcite . following the trend in language and vision domains, p5 @xcite and m6-rec @xcite put forward the concept of personalized foundation models for recommendation and propose to pretrain llms on instructional prompts to accommodate various recommendation tasks under a shared model and training objective. while there are large models for language @xcite , vision @xcite , graphs @xcite and recommendation @xcite domains separately, in this work, we take one step further and aim to unify the above foundation models to jointly process multi-modality information sources for personalization and recommendation. to this end, we follow the "pretrain, personalized prompt and predict paradigm (p5)" for recommendation @xcite and propose a multimodal foundation model (mfm) named vip5 (visual p5), which provides the following advantages for recommender systems: 1) vip5 provides multimodal personalized prompts for supporting all modalities' connections to the recommendation foundation model. specifically, to construct multimodal personalized prompts, vip5 employs a mapping network to transfer features from other modalities into the corresponding tokens. by this step, multimodal features are projected to the same manifold space of the backbone foundation model. 2) the vip5 framework provides the ability of parameter-efficient tuning rather than pretraining in existing recommendation foundation models such as p5 @xcite . different from the pre-training step of p5 that updates all the parameters in the backbone foundation model -which is impractical when the size of foundation model grows explosively -vip5 only finetunes a small proportion of extra lightweight adapter modules during training while maintaining the large language model backbone fixed. 3) with the ability of multi-modality learning and parameter-efficient tuning, vip5 further improves the performance of recommendation foundation models with both less training time and less memory usage, making it easier to train and deploy foundation models for recommendation. overall, our key contributions are outlined as follows: large recommendation models. motivated by the success of large language models (llms), the recsys community started to pay more attention to recommendation model's generalization ability and transferability @xcite . for example, inspired by the prompt learning paradigm, peter @xcite and pepler @xcite proposes to learn personalized continuous prompts to represent user and item ids and generates natural language explanations to justify recommendations. in contrast, m6-rec @xcite ) converts all user behavior information to plain text sequences and feeds them into a transformer encoder, and then designs a task-specific training loss for downstream tasks and finetuning. apart from previous efforts, p5 @xcite and openp5 @xcite leverage not only instruction-based finetuning on llms to represent personalized fields for users and items but also unifies various tasks via natural language instructions. hence, p5 is able to unify various recommendation tasks into a shared encoder-decoder architecture and a joint training objective. p5-id @xcite further explores different item id creation methods for llm-based recommendation models, such as sequential indexing, collaborative indexing, semantic indexing, and hybrid indexing. ## vip5 paradigm with multimodal personalized prompts we introduce the proposed vip5 paradigm in this section. in section 3.1, we incorporate multimodal signals into personalized prompts. in section 3.2, we elaborate how to conduct parameter-efficient tuning with adapters based on multimodal personalized prompts. ## experiments in this section, we provide the performance comparison between the vip5 framework and representative approaches for different task groups. we conduct extensive experiments and ablation studies to explore the following research questions: ## conclusions this paper presents vip5, a parameter-efficient multimodal foundation recommendation model unifying vision, language, and personalization information. we design multimodal personalized prompts to integrate visual signals with text and personalization information, enhancing recommendation across diverse modalities. our parameterefficient tuning strategy updates a small proportion of adapters, achieving a better trade-off between recommendation performance, training efficiency, and memory usage. through extensive experiments, we show the effectiveness of our vip5 framework and show that multimodality information is helpful for various recommendation tasks. future work includes further scaling up the backbone model, incorporating even more modalities, and exploring improved prompt strategies, such as chain-of-thought prompting.