Paper: Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond (ICLR-2024)

Reviewer comment (Novelty):
This paper has a lot of things, but lacks of novelty.

Author response:
Thank you for your valuable review and suggestions. We appreciate the opportunity to make the following clarifications.

> **Q1:** Good experimental study and results. Consider multi-modal multi-tasks. But lack of novelty. The components are shown in the literature.

We greatly appreciate your recognition of our work, particularly our comprehensive experimental study, the robust results obtained, and our proposal of a multi-modal, multi-task approach. In terms of novelty, we are the first to propose the problem of unified multi-modal personalization for various tasks. Our method addresses the inherent challenges through a comprehensive approach that encompasses three complementary aspects: the formulation of input data, the fusion of modalities in network modeling, and the design of optimization objectives. The novelty of our work lies in the unique solution to each perspective and the seamless integration of these components to formulate an overall effective strategy. Next we will elaborate on how each component has been designed and adapted to create a distinctive approach.

For user input modeling, we are the first to emphasize and verify the feasibility of utilizing raw multi-modal content, as opposed to relying solely on single-modal data or product IDs to represent user sequences. This choice of content acts as a bridge, facilitating transferability and generalization across new domains and users. The comparative analysis of generalization capabilities, presented in Table 4 of our paper, highlights the strengths of our method. 

For modal fusion within the network, while bridging pre-trained vision and language models and the concept of cross-attention is not new, our implementation and the specific challenges we address in our paper are unique. In previous works, CLIP [1] processes visual and textual data separately through modality-specific encoders and only combines them at the final layer to compute similarity scores. VIP5 [2], on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language modeling. We refer to these as "late fusion" and "early fusion" strategies, respectively. These methods focus on processing a single image-text pair. In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP to align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data during fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337$\rightarrow$0.0244). This fusion is also performed at every layer, allowing for a more nuanced and detailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've included an experiment titled 'w/o Fine-grained' in Table 3 of our submitted paper, showing our method's performance without the cross-attention design. Additionally, we have conducted comparisons with Early Fusion, Late Fusion, and without Exclusive Attention strategies respectively in the below table, which further validate the effectiveness and rationale behind our proposed fusion method.

|                         |     HR@5     |   NDCG@5   |    MRR@5   |
|:-----------------------:|:------------:|:----------:|:----------:|
|           VIP5          |    0.0262    |   0.0163   |   0.0127   |
|  UniMP w/o Exclusive Attention |   0.0244    |   0.0165   |   0.0132  |
|  UniMP w/o Cross-attention |   0.0257     |   0.0192   |   0.0161   |
| UniMP with Early Fusion |    0.0271    |   0.0185   |   0.0135   |
|  UniMP with Late Fusion |   0.0274     |   0.0193   |   0.0156   |
|          UniMP          | **0.0337** | **0.0231** | **0.0196** |


Furthermore, our approach to enhance multi-task optimization, which focuses on improving alignment and prioritization, coupled with the introduction of a new multi-modal personalization benchmark, provides critical insights into this emerging domain. 

In summary, the integration of all these components presents a holistic solution for addressing real-world challenges in achieving unified multi-modal personalization. We believe our paper contributes valuable insights and advancements to the field, and we hope that the reviewer will consider these aspects of novelty and innovation in our work.

Reviewer comment (Novelty):
All the components are already in the literature, bridging pre-trained vision and language models is not new, cross attention idea etc.

Author response:
Thank you for your valuable review and suggestions. We appreciate the opportunity to make the following clarifications.

> **Q1:** Good experimental study and results. Consider multi-modal multi-tasks. But lack of novelty. The components are shown in the literature.

We greatly appreciate your recognition of our work, particularly our comprehensive experimental study, the robust results obtained, and our proposal of a multi-modal, multi-task approach. In terms of novelty, we are the first to propose the problem of unified multi-modal personalization for various tasks. Our method addresses the inherent challenges through a comprehensive approach that encompasses three complementary aspects: the formulation of input data, the fusion of modalities in network modeling, and the design of optimization objectives. The novelty of our work lies in the unique solution to each perspective and the seamless integration of these components to formulate an overall effective strategy. Next we will elaborate on how each component has been designed and adapted to create a distinctive approach.

For user input modeling, we are the first to emphasize and verify the feasibility of utilizing raw multi-modal content, as opposed to relying solely on single-modal data or product IDs to represent user sequences. This choice of content acts as a bridge, facilitating transferability and generalization across new domains and users. The comparative analysis of generalization capabilities, presented in Table 4 of our paper, highlights the strengths of our method. 

For modal fusion within the network, while bridging pre-trained vision and language models and the concept of cross-attention is not new, our implementation and the specific challenges we address in our paper are unique. In previous works, CLIP [1] processes visual and textual data separately through modality-specific encoders and only combines them at the final layer to compute similarity scores. VIP5 [2], on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language modeling. We refer to these as "late fusion" and "early fusion" strategies, respectively. These methods focus on processing a single image-text pair. In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP to align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data during fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337$\rightarrow$0.0244). This fusion is also performed at every layer, allowing for a more nuanced and detailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've included an experiment titled 'w/o Fine-grained' in Table 3 of our submitted paper, showing our method's performance without the cross-attention design. Additionally, we have conducted comparisons with Early Fusion, Late Fusion, and without Exclusive Attention strategies respectively in the below table, which further validate the effectiveness and rationale behind our proposed fusion method.

|                         |     HR@5     |   NDCG@5   |    MRR@5   |
|:-----------------------:|:------------:|:----------:|:----------:|
|           VIP5          |    0.0262    |   0.0163   |   0.0127   |
|  UniMP w/o Exclusive Attention |   0.0244    |   0.0165   |   0.0132  |
|  UniMP w/o Cross-attention |   0.0257     |   0.0192   |   0.0161   |
| UniMP with Early Fusion |    0.0271    |   0.0185   |   0.0135   |
|  UniMP with Late Fusion |   0.0274     |   0.0193   |   0.0156   |
|          UniMP          | **0.0337** | **0.0231** | **0.0196** |


Furthermore, our approach to enhance multi-task optimization, which focuses on improving alignment and prioritization, coupled with the introduction of a new multi-modal personalization benchmark, provides critical insights into this emerging domain. 

In summary, the integration of all these components presents a holistic solution for addressing real-world challenges in achieving unified multi-modal personalization. We believe our paper contributes valuable insights and advancements to the field, and we hope that the reviewer will consider these aspects of novelty and innovation in our work.

Reviewer comment (Writing):
What is x (the visual input of the item), how do you provide it in eq 1? Apart from the special token [IMG]?

Author response:
> **Q2:** What is x (the visual input of the item), how do you provide it in eq 1? Apart from the special token [IMG]?


Apologies for any confusion caused. The variable $x$ represents the feature matrix of the visual input, in the shape $\mathbb{R}^{H\times W\times C}$, where $H$, $W$, $C$ represents the height, width, and channel count of the image, respectively. Equation 1 serves as the input for the language model and is formatted in plain text. Note that $x$ is not included in Equation 1. We use the special token [IMG] in **plain text** to denote the image's location. It allows us to establish a correspondence between the textual tokens and the image associated with each product. The key purpose here is to ensure that the visual information of each product is conditioned exclusively on its corresponding textual data during cross-modal attention, a factor critical to the success of our method.

Reviewer comment (Presentation):
If you are already providing the visual input in user's interaction history, you are using the same image in visual encoder again into cross attention module? (As far as I understand, no visual information is added to the user's history, the features come from visual encoder, if this is correct, I suggest that separate image from the user's history in Fig 1.)

Author response:
> **Q3:** Are you are using the same image in visual encoder again into cross attention module? What is s in page 3 Section 2.1 line 5?

As shown in the answer to Q1, we exclusively input the image feature into the visual encoder to derive its representation. Subsequently, this visual representation for each product is conditioned solely on its corresponding textual tokens during the cross-attention fusion process. The [IMG] is a special token in plain text, and we didn't input the image feature to the language model directly. $s$ in page 3 Section 2.1 should be the user’s interaction history $u_h$. We've made adjustments in our paper for clarity.

Reviewer comment (Writing):
What is s in page 3 Section 2.1 line 5?

Author response:
> **Q3:** Are you are using the same image in visual encoder again into cross attention module? What is s in page 3 Section 2.1 line 5?

As shown in the answer to Q1, we exclusively input the image feature into the visual encoder to derive its representation. Subsequently, this visual representation for each product is conditioned solely on its corresponding textual tokens during the cross-attention fusion process. The [IMG] is a special token in plain text, and we didn't input the image feature to the language model directly. $s$ in page 3 Section 2.1 should be the user’s interaction history $u_h$. We've made adjustments in our paper for clarity.

Reviewer comment (Writing):
Page 2, traditional methods suffers -> suffer

Author response:
> **Q4:** Typos.

Thank you for bringing this to our attention. We have corrected the typos in the paper and will persist in enhancing its overall presentation and structure, with the aim of improving readability.

Reviewer comment (Writing):
Page 4, line 2, infomration -> information

Author response:
> **Q4:** Typos.

Thank you for bringing this to our attention. We have corrected the typos in the paper and will persist in enhancing its overall presentation and structure, with the aim of improving readability.

Reviewer comment (Writing):
Page 4, Section 2.2, the visual input -> The visual input

Author response:
> **Q4:** Typos.

Thank you for bringing this to our attention. We have corrected the typos in the paper and will persist in enhancing its overall presentation and structure, with the aim of improving readability.

Reviewer comment (Novelty):
Regarding the technical novelty in data fusion and user modeling, the contributions of this work are not impressive. In particular, the strategies of data fusion and user modeling are kind of straightforward. The effectiveness is not validated. Please clarify or verify your choice.

Author response:
Thank you for raising the question. Regarding data fusion, for previous works, CLIP [1] processes visual and textual data separately through modality-specific encoders and only combines them at the final layer to compute similarity scores. VIP5 [2], on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language modeling. We refer to these as "late fusion" and "early fusion" strategies, respectively. These methods focus on processing a single image-text pair. In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP to align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data during fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337$\rightarrow$0.0244). This fusion is also performed at every layer, allowing for a more nuanced and detailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've included an experiment titled "w/o Fine-grained" in Table 3 of our submitted paper, showing our method's performance without the cross-attention design. Additionally, we have conducted comparisons with Early Fusion, Late Fusion, and without Exclusive Attention strategies respectively in the table below, which further validate the effectiveness and rationale behind our proposed fusion method. We have also incorporated the results in Appendix A.6 of the revision.


|                         |     HR@5     |   NDCG@5   |    MRR@5   |
|:-----------------------:|:------------:|:----------:|:----------:|
|           VIP5          |    0.0262    |   0.0163   |   0.0127   |
|  UniMP w/o Exclusive Attention |   0.0244    |   0.0165   |   0.0132  |
|  UniMP w/o Cross-attention |   0.0257     |   0.0192   |   0.0161   |
| UniMP with Early Fusion |    0.0271    |   0.0185   |   0.0135   |
|  UniMP with Late Fusion |   0.0274     |   0.0193   |   0.0156   |
|          UniMP          | **0.0337** | **0.0231** | **0.0196** |


In our approach to user modeling, we emphasize the use of raw multi-modal content, as opposed to relying solely on single-modal data or product IDs to represent user sequences. This choice of content acts as a bridge, facilitating transferability and generalization across new domains and users. The comparative analysis of generalization capabilities, presented in Table 4 of our paper, highlights the strengths of our method. Notably, UniMP demonstrates a substantial edge in transfer learning scenarios.

Reviewer comment (Presentation):
The presentation of this work should be improved. Typos can be found without too much effort.

Author response:
Thanks for pointing out the typos. We've addressed these corrections and polished the paper. We'll continue to improve the overall presentation and organization of the paper, aiming to enhance its readability.

Reviewer comment (Reproducibility):
How do you incorporate the behavioral data, like click, browse, and scroll?

Author response:
Thanks for the question. For the Amazon dataset, there are purchase and review behaviors available. In the case of Netflix, it provides data on viewing behavior. To accurately model these behaviors, we incorporate the items that have been purchased or viewed, along with their corresponding attributes, into the user sequence. Moreover, review texts are utilized to represent review behaviors. If additional behavioral signals like clicks, browsing, and scrolling are available, we could assign specific behavior-type tokens to distinguish these various activities and describe the duration of each behavior in a textual format.

Reviewer comment (Writing):
How do you define and characterize the multi-modal information? The category, brand, description, and price are all textual inputs; why are they defined as multi-modalities in the paper?

Author response:
Apologies for any confusion caused. In our paper, we refer to "multi-modal information" as the combination of textual and visual content. Meanwhile, elements like category, brand, description, and price are classified as "heterogeneous information", which can be represented in a textual format.

Reviewer comment (Reproducibility):
What are the choices of backbone LMs?

Author response:
In our paper, we utilized the 3B INCITE model, an instruction-tuned large language model, for its ease of use and open-source availability. It is important to note that our framework is adaptable to various large language models (LMs). To further explore this adaptability, we tested UniMP's performance with Mosaic ML's MPT-1B model, denoted as UniMP-Small. The results of these experiments, detailed in the table below, suggest that larger LMs can more effectively improve personalized user modeling. We have also incorporated the results in Appendix A.9 of the revision.

|                         |     HR@5     |   NDCG@5   |    MRR@5   |
|:-----------------------:|:------------:|:----------:|:----------:|
|          UniMP-Small          |    0.0315    |   0.0216   |   0.0179   |
|          UniMP          | **0.0337** | **0.0231** | **0.0196** |

Reviewer comment (Reproducibility):
How does a user handle the token limits of the LLMs when s/he aims to apply this framework?

Author response:
Thanks for the insightful question. There are several ways to address the problem. First, we, like prior studies [2,3,..], input the most recent interactions rather than the entire history, as this more accurately reflects current user interests while reducing computational workload. Additionally, the user can consider omitting certain product attributes from the sequence. As evidenced in Table 7 of our paper, our UniMP model demonstrates robustness with incomplete and missing textual and visual data. Furthermore, the problem can be handled from the perspective of language modeling itself. Recent research works [4,5] have shown that pre-trained language models can significantly increase token limits while maintaining performance, achieved by modifying the attention mechanism in a post-hoc manner.

Reviewer comment (Experiments):
The experimental setting needs clarification.

Author response:
Thank you for your question. For the recommendation task, we categorize the baselines into four distinct groups: general recommenders, sequential recommenders, generative model-based recommenders, and multi-modal recommenders. Within these, both MF and LightGCN are classified under general recommenders. These general recommenders operate differently compared to other types and are included primarily to ensure a thorough and comprehensive comparison. It's important to note that the other categories are our primary baselines for comparison. To clarify our methodology and avoid any confusion, we've included additional details about our experimental setup in Section A.2 of the paper.

Reviewer comment (Experiments):
Whether this method can perform well on other datasets except for Amazon datasets is unknown.

Author response:
Thank you for your valuable suggestion. We primarily selected the Amazon dataset as our experimental platform due to its extensive and varied content, including reviews, ratings, images, interactions, and product attributes. This comprehensiveness makes it particularly suited for multi-modal, multi-task evaluations. Recognizing the importance of validating our findings across different datasets, we have also applied our UniMP method to the Netflix [3] and HM [4] datasets. For Netflix, the images are obtained by crawling movie posters from the website. The results, as detailed in our tables, affirm the effectiveness and adaptability of UniMP across these varied datasets. We have also included the results in Section A.8 in our revision.

|              |               |          HM        |                 |          |        Netflix          |                 |
|:------------:|:---------------:|:----------------:|:---------------:|:---------------:|:----------------:|:---------------:|
|              |       HR@5      |      NDCG@5      |      MRR@5      |       HR@5      |      NDCG@5      |      MRR@5      |
|      P5      |      0.0101     |      0.0063      |      0.0046     |      0.0742     |      0.0413      |      0.0316     |
|     VIP5     |      0.0122     |      0.0118      |      0.0093     |      0.0936     |      0.0589      |      0.0395     |
|    S3-Rec    |      0.0185     |      0.0121      |      0.0102     |      0.1155     |      0.0632      |      0.0511     |
|    UniSRec   |      0.0196     |      0.0139      |      0.0107     |      0.1324     |      0.0856      |      0.0644     |
| UniMP (Ours) | **0.0313** | **0.0206** | **0.0172** | **0.1723** | **0.1196** | **0.1024** |

Reviewer comment (Novelty):
The authors introduce a cross-attention mechanism to merge visual and textual data. The rationale for this choice remains ambiguous. It is important to discuss how this approach compares to other vision-language fusion techniques such as CLIP[1] and other multi-modal recommendation systems like VIP5.

Author response:
Thank you for bringing out the question. Our fusion method's main innovation, compared to CLIP [1] and VIP5 [2], lies in our unique approach to fusion strategy and fusion position. CLIP processes visual and textual data separately through modality-specific encoders and only combines them at the final layer to compute similarity scores. VIP5, on the other hand, merges visual embeddings, obtained from a frozen visual encoder, with textual inputs right at the beginning for language modeling. We refer to these as "late fusion" and "early fusion" strategies, respectively. These methods focus on processing a single image-text pair.

In contrast, UniMP is designed to utilize a user's entire history, involving multiple products, for the generation. To handle this, we proposed UniMP to align and integrate the visual information into the language model through the specific cross-attention design (instead of concatenation). The visual information of each product is conditioned exclusively on its corresponding textual data during fusion. We found that removing exclusive attention leads to a substantial performance decrease (0.0337$\rightarrow$0.0244). This fusion is also performed at every layer, allowing for a more nuanced and detailed understanding of user preferences. To demonstrate the effectiveness of this approach, we've included an experiment titled "w/o Fine-grained" in Table 3 of our submitted paper, showing our method's performance without the cross-attention design. Additionally, we have conducted comparisons with Early Fusion, Late Fusion, and without Exclusive Attention strategies respectively in the table below, which further validate the effectiveness and rationale behind our proposed fusion method. We have also incorporated the results in Appendix A.6 of the revision.

|                         |     HR@5     |   NDCG@5   |    MRR@5   |
|:-----------------------:|:------------:|:----------:|:----------:|
|           VIP5          |    0.0262    |   0.0163   |   0.0127   |
|  UniMP w/o Exclusive Attention |   0.0244    |   0.0165   |   0.0132  |
|  UniMP w/o Cross-attention |   0.0257     |   0.0192   |   0.0161   |
| UniMP with Early Fusion |    0.0271    |   0.0185   |   0.0135   |
|  UniMP with Late Fusion |   0.0274     |   0.0193   |   0.0156   |
|          UniMP          | **0.0337** | **0.0231** | **0.0196** |

Reviewer comment (Theory):
The necessity of the context reconstruction loss is still unclear to me. I wonder the detailed design of this loss item and why it can benefit alignment. And about the token-level reweighting, I think it is important to give more definitions/explanations of easy/hard tokens and how to distinguish.

Author response:
We apologize for the missing details. For the context reconstruction loss, this term involves predicting product attributes for each historical product image in a user's sequence. Unlike our primary task in the paper (e.g., recommendation), which utilizes the entire user history encompassing multiple products, context reconstruction loss focuses on individual products. This approach aligns with the pre-training objective of the model, such as generating detailed descriptions for each specific image. The integration of context reconstruction loss serves as a bridge, harmonizing the optimization of our model. Importantly, the inclusion of context reconstruction loss doesn't incur extra computational costs, yet it substantially enhances training effectiveness by enriching the information learned.

As for token-level reweighting, we recognize that reconstructing context is generally simpler than accurately predicting user preferences, which is a key motivation for our approach. To address this, we propose a reweighting term designed to automatically differentiate between tokens that are easier or harder to predict. We've also included the details and discussion about the loss in Section A.7 of the paper.

Reviewer comment (Experiments):
In the evaluation part, it is crucial to offer clarity on the experimental settings. Some baseline models like MF and LightGCN operate differently from sequential models like S3Rec and BERT4Rec. Ensuring a clear distinction between these setups is essential to maintain fairness in comparisons.

Author response:
Thank you for your question. For the recommendation task, we categorize the baselines into four distinct groups: general recommenders, sequential recommenders, generative model-based recommenders, and multi-modal recommenders. Within these, both MF and LightGCN are classified under general recommenders. These general recommenders operate differently compared to other types and are included primarily to ensure a thorough and comprehensive comparison. It's important to note that the other categories are our primary baselines for comparison. To clarify our methodology and avoid any confusion, we've included additional details about our experimental setup in Section A.2 of the paper.

Reviewer comment (Experiments):
The experiments utilize sub-datasets from the Amazon dataset, which might possess similar data distributions. This might limit the generalizability of the results. It would be informative to observe the recommendation model's performance, especially in scenarios of zero/few-shot recommendations, on diverse datasets with visual content.

Author response:
Thank you for your valuable suggestion. We primarily selected the Amazon dataset as our experimental platform due to its extensive and varied content, including reviews, ratings, images, interactions, and product attributes. This comprehensiveness makes it particularly suited for multi-modal, multi-task evaluations. Recognizing the importance of validating our findings across different datasets, we have also applied our UniMP method to the Netflix [3] and HM [4] datasets. For Netflix, the images are obtained by crawling movie posters from the website. The results, as detailed in our tables, affirm the effectiveness and adaptability of UniMP across these varied datasets. We have also included the results in Section A.8 in our revision.

|              |               |          HM        |                 |          |        Netflix          |                 |
|:------------:|:---------------:|:----------------:|:---------------:|:---------------:|:----------------:|:---------------:|
|              |       HR@5      |      NDCG@5      |      MRR@5      |       HR@5      |      NDCG@5      |      MRR@5      |
|      P5      |      0.0101     |      0.0063      |      0.0046     |      0.0742     |      0.0413      |      0.0316     |
|     VIP5     |      0.0122     |      0.0118      |      0.0093     |      0.0936     |      0.0589      |      0.0395     |
|    S3-Rec    |      0.0185     |      0.0121      |      0.0102     |      0.1155     |      0.0632      |      0.0511     |
|    UniSRec   |      0.0196     |      0.0139      |      0.0107     |      0.1324     |      0.0856      |      0.0644     |
| UniMP (Ours) | **0.0313** | **0.0206** | **0.0172** | **0.1723** | **0.1196** | **0.1024** |