Paper: Barycentric Alignment of Mutually Disentangled Modalities (ICLR-2024)

Reviewer comment (Theory):
the work implicitly relies on the assumption of the relatedness between behavioral data and side information, which in general is not supported by prior empirical evidence

Author response:
**Q: the work implicitly relies on the assumption of the relatedness between behavioral data and side information, which in general is not supported by prior empirical evidence.** 

**A:** Recent studies [2, 3, 4] have shown that incorporating side information such as textual and/or visual content is beneficial to recommendation models, e.g., improving recommendation accuracy. As collected rating data is often sparse in nature, which could prevent learning expressive user/item representations, side information provides additional semantic information of users/items, which contributes to learn higher quality of user and item embeddings from both rating data and side information.

Reviewer comment (Experiments):
the work is focused on a specific type of side information - textual description, which limits its generality

Author response:
**Q: The title is too general. Only textual factors are considered in the work, which is not the general case of arbitrary modalities**

**A:** We consider two modalities ratings and texts. Ratings are the key modality in multi-modal recommender systems [3]. A more specific rephrasing of the title could be *Barycentric Alignment of Mutually Disentangled Rating and Text Modalities*.  

Other modalities can be incorporated in our model. The alignment method can work for two mutually disentangled modalities. For example, one can replace the text channel with an image channel of which the inputs are image pixels. We consider textual content as it is available in chosen datasets while images are not available in Citeulike-a or MovieLens.

Reviewer comment (Experiments):
an important testing scenario related to cold-start settings is missing

Author response:
**Q: On the argument with respect to cold-start scenario** 

**A:** We agree that side information, e.g., visual or textual content, can be used to resolve cold-start problem in recommender systems. However, the key research question in this paper is how to fuse disentangled user interest factors from texts with those learned from ratings to achieve a more accurate recommendation model. Recent studies on multi-modal recommender systems also focus on how to leverage side information to obtain more expressive representations, leading to higher recommendation accuracy. 

As our focus is not resolving the cold-start problem, we do not dive into cold-start scenario to keep this paper focused on disentangled representation learning with textual content.

Reviewer comment (Theory):
In many places in the text, there's an implicit assumption that modalities are mutually related, e.g., "transfer knowledge between interest factors from mutually related modalities". This assumption may not be true in practice, as modalities may be "non-alignable". This is especially true for side information, i.e., static descriptive features and attributes of users and items.

Author response:
**Q: Modalities may be non-alignable. … The work implicitly assumes that ratings are not expressive enough...
Focusing on side features mostly leads to the so-called overspecialization, meaning that side features are not descriptive enough to provide any additional knowledge on top of what's hidden in the behavioral data and can only limit the expressivity of models, not improve it** 

**A:** We would like to make two clarifications here. *First*, recent studies [2, 3, 4] have demonstrated that side information helps alleviate rating data sparsity problem to achieve higher recommendation accuracy.  

*Second*, in our model, textual content does not completely replace ratings to make recommendations. Rather, it acts as an accompaniment. We fuse user interest factors learned from textual content with those from ratings to make rating interest factors more expressive. That it achieves higher recommendation accuracy than baselines, which evidences that the modalities are aligned.

Reviewer comment (Experiments):
The authors state that "side information could complement those mined from user consumption behaviors", implying that ratings are not expressive enough. Verification of this claim would require a rigorous study.

Author response:
**Q: Discussion on a classical study on the *even few ratings are more valuable than metadata* effect by [1]** 

**A:** In [1], the authors concluded that *“... even 10 ratings of a new movie are more valuable than the best solely metadata-based representation. ... this is due to the large gap between the movie descriptions and the movies themselves: people rate movies, not their descriptions”* 

The experiment leading to this conclusion is that the authors of [1] compare the proposed model using ratings against another model that only uses metadata to make content-based recommendation. Again, we would like to emphasize that our model does not fully rely on textual content to make recommendations. We still rely on rating data to make recommendations, which shares the same intention with [1]. The novelty is we innovatively fuse user interest factors learned from texts with those learned from ratings in disentangled fashion using alignment method (Section 4.2.1) and mapping method (Section 4.2.2) to obtain more expressive user representations.

Reviewer comment (Theory):
In the general case, focusing on side features often leads to overspecialization, where side features are not descriptive enough to provide additional knowledge beyond behavioral data, and can limit model expressivity rather than improve it. This is supported by prior work such as Pilászy and Tikk (2009).

Author response:
**Q: Modalities may be non-alignable. … The work implicitly assumes that ratings are not expressive enough...
Focusing on side features mostly leads to the so-called overspecialization, meaning that side features are not descriptive enough to provide any additional knowledge on top of what's hidden in the behavioral data and can only limit the expressivity of models, not improve it** 

**A:** We would like to make two clarifications here. *First*, recent studies [2, 3, 4] have demonstrated that side information helps alleviate rating data sparsity problem to achieve higher recommendation accuracy.  

*Second*, in our model, textual content does not completely replace ratings to make recommendations. Rather, it acts as an accompaniment. We fuse user interest factors learned from textual content with those from ratings to make rating interest factors more expressive. That it achieves higher recommendation accuracy than baselines, which evidences that the modalities are aligned.

Reviewer comment (Experiments):
The cold-start scenario is an exception where side features can be useful, as only side features are available and a mapping must be made between collaborative and side information spaces. Application of hybrid models in this scenario is logical, but the authors provide no related experimental results and only test on standard collaborative filtering tasks, limiting understanding of the approach's utility.

Author response:
**Q: On the argument with respect to cold-start scenario** 

**A:** We agree that side information, e.g., visual or textual content, can be used to resolve cold-start problem in recommender systems. However, the key research question in this paper is how to fuse disentangled user interest factors from texts with those learned from ratings to achieve a more accurate recommendation model. Recent studies on multi-modal recommender systems also focus on how to leverage side information to obtain more expressive representations, leading to higher recommendation accuracy. 

As our focus is not resolving the cold-start problem, we do not dive into cold-start scenario to keep this paper focused on disentangled representation learning with textual content.

Reviewer comment (Evaluation):
A common observation is that increased interpretability of a model often leads to lower quality. This is confirmed in the current work, where smaller $\epsilon$ values favor interpretability but worsen recommendations, consistent with the overspecialization remark.

Author response:
**Q: Smaller $\epsilon$ that favors interpretability leads to worse recommendations. This is in line with the overspecialization remark**

**A:** We would like to clarify that the trade-off between interpretability and recommendation in our model is controlled by $\epsilon$ in Equation 4, which controls the sparsity of alignment probabilities in Algorithm 1. 

When the alignment probabilities are close to uniform distribution (large $\epsilon$), it is difficult to find a one-to-one mapping between rating and text factors of a single user, thus decreasing the interpretability because we do not know which rating interest factors align with which text interest factors. When $\epsilon$ is small, the alignment probabilities are highly skewed, we can easily find a one-to-one mapping between rating interest factors and text interest factors. Therefore, we can understand the meaning of a rating interest factor via the aligned text factor by visualizing the output of text decoder corresponding to the aligned text factor. 

We illustrate this concept in Figure 4 in supplementary materials.

Reviewer comment (Evaluation):
The proposed architecture puts strong emphasis on interpretability (e.g., "turning off effect of $\mathcal{L}^t_u$ hurts recommendation accuracy"), but this claim is not directly supported by statistical evidence. The reported improvements lack statistical significance and may be due to overfitting or data-splitting artifacts. Confidence intervals are missing.

Author response:
**Q: The proposed architecture still puts a lot of focus on interpretability**

**A:** For clarification, we focus on disentangled representation learning to improve recommendation accuracy as this direction has been shown to be effective [1].  The interpretability is embedded inside disentangled representations by nature [2] as it discovers factors of variation underlying data. We transfer this concept from disentangled representation learning to recommender systems to discover user interest factors from ratings and texts. Thus, interpretability is naturally incorporated. Interpretability is not a new feature, rather we analyze it to further understand the proposed model, i.e., how the proposed aligning and fusing method relates to interpretability.

**Q: Effect of $\mathcal{L}^t_u$ due to overfitting to specific factor (like popular words), or training/data-splitting artifacts that lead occasionally higher values** 

**A:** To clarify the understanding, turning off the effect of $\mathcal{L}^t_u$ means the model does not have the supervision signals from textual content. Then the model is free to embed anything into text factors. Table 5 demonstrates that when including $\mathcal{L}^t_u$, text factors are trained to capture interest factors from texts, which is more meaningful than freely embedding anything into them, leading to better recommendation accuracy. Regarding statistical significance, in Table 5, the numbers of Citeulike-a R@10 and MovieLens R@10 and N@10 are statistically significant. Table 5 brings two key takeaways. For one, textual signals are helpful to recommendation accuracy to a certain extent. For another, the improvement of BandVAE not only comes from simply reconstructing textual content but it is also (mainly) from the proposed aligning and fusing modules, which are the key contributions in this paper.

Reviewer comment (Experiments):
No comparison is provided against simpler models such as Factorization Machines (FM) by Rendle (2010), which can also analyze item-feature adherence via inner products. The interpretability aspect is thus not novel, although the representation learning differs. The utility of the interpretation component (text decoder) is not convincingly demonstrated empirically, and the model may perform similarly without it. Results for the cold-start case are absent.

Author response:
**Q: no comparison is made with simpler modes, i.e. Factorization Machines (FM) by [Rendle 2010]** 

**A:** We did not compare BandVAE against FM as one of the baselines MDCVAE has outperformed FM. As the main target is to improve recommendation accuracy via disentangled representation learning, we involve recently developed disentangled recommendation models (and these models already demonstrated state-of-the-art results), including both those using textual content and those that do not, to verify the advantages of our model.  

**Q: The interpretability aspect cannot be considered novel here, although, the approach to obtain representations to be analyzed is different** 

**A:** As stated in the contributions, the key novelty in this paper is how to align and fuse disentangled user interest factors uncovered from ratings and texts.  The interpretability is brought by disentangled representation learning by nature [2]. We analyze interpretability to further understand the inner working of the proposed model.

Reviewer comment (Writing):
The text contains excessive generalization of results and conclusions, starting from the title. The title is too general and promises more than the results deliver. Only textual factors are considered, not arbitrary modalities; no other features or user attributes are used. The title, introduction, and conclusion should be revised to reflect these limitations.

Author response:
**Q: The title is too general. Only textual factors are considered in the work, which is not the general case of arbitrary modalities**

**A:** We consider two modalities ratings and texts. Ratings are the key modality in multi-modal recommender systems [3]. A more specific rephrasing of the title could be *Barycentric Alignment of Mutually Disentangled Rating and Text Modalities*.  

Other modalities can be incorporated in our model. The alignment method can work for two mutually disentangled modalities. For example, one can replace the text channel with an image channel of which the inputs are image pixels. We consider textual content as it is available in chosen datasets while images are not available in Citeulike-a or MovieLens.

Reviewer comment (Novelty):
There already exists the term "hybrid models" for approaches combining behavioral and side information (e.g., FM, HybridSVD, hybrid EASEr). A clear connection to this family should be made, as the term "modalities" is not standard in that context, although the same general task is addressed.

Author response:
**Q: A clear connection must be made to family of hybrid models (FM, hybridSVD, hybrid EASEr)**

**A:** Our proposed model shares the intention with hybrid models (FM, hybridSVD, hybrid EASEr), combining techniques from content-based filtering and collaborative filtering to achieve higher recommendation accuracy. We would consider adding this discussion in the final version.

Reviewer comment (Presentation):
Figure 1 lacks notation; adding and labeling the different weight matrices would help clarify the approach.

Author response:
**Q:** **Figure 1 lacks notations**

**A:** We already provided key notations in Figure 1. Due to limited space, the detailed descriptions of weight matrices are presented in preliminary sections to ease the reading.

Reviewer comment (Writing):
The notation $(r^{uy}_k , o^{uy}_k )$ is unclear — it is not evident if this denotes concatenation.

Author response:
**Q: The notation $(r^{uy}_k, o^{uy}_k)$ is not clear, is it a concatenation?**

**A:** This notation means the output includes two terms. We follow MacridVAE [1] for consistent notations.

Reviewer comment (Experiments):
Only a single training epoch is analyzed for computational efficiency, which is insufficient. The number of epochs needed to train optimal models for each case should be reported. More extensive analysis, including a comparison with simpler models without side features, is required, as the marginal performance gains may not justify the increased computational cost.

Author response:
**Q: On the evaluation of computational efficiency**

**A:** For clarification, to verify time efficiency we report the training time to run a single epoch in our GPU server in the main text. The reported time is averaged over multiple runs. The main observation is that the proposed BandVAE only requires slightly higher training and inference time than the closest baseline ADDVAE.

The table below shows the number of training epochs to achieve the reported results. We run each model 5 times with different random seeds on each dataset then report the averaged number of required epochs. For contrasting, we compare BandVAE against ADDVAE which uses textual content and MacridVAE which does not use textual content. There are two key takeaways. *First*, models using textual content does not always require more training epoch than models not using text content, which is shown by contrasting MacridVAE and ADDVAE. *Second*, BandVAE only requires slightly more training epochs than both MacridVAE and ADDVAE yet achieves higher recommendation accuracy, demonstrating the effectiveness and efficiency of BandVAE.

|           | Citeulike-a | MovieLens | Cell Phones |
|:---------:|:-----------:|:---------:|:-----------:|
|  BandVAE  |     58      |    45     |     34      |
|  ADDVAE   |     50      |    38     |     33      |
| MacridVAE |     46      |    40     |     36      |

Reviewer comment (Theory):
Ratings are not usually regarded as one modality in recommendation.

Author response:
**A1:** Ratings are considered as the key modality in multi-modal recommender systems [1]. Textual content, visual content, etc., are considered as other modalities to alleviate data sparsity problem, which helps recommendation models learn more expressive embeddings. As rating data is often highly sparse in nature, learning user/item representations from other data-modalities, e.g., texts or images, then fusing them with those learned from ratings has been shown to alleviate the data sparsity problem and obtain higher quality of user and item representations [1, 2]. 

[1] Truong et al. Multi-Modal Recommender Systems: Hands-On Exploration. ACM RecSys 2021.  

[2] Zhou et al. A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions. https://arxiv.org/pdf/2302.04473.pdf

Reviewer comment (Experiments):
With reference to Table 1, it seems that the improvement over the tested baselines is not sufficiently large to justify the efficacy of the proposed approach; the authors may provide additional results (if any) to effectively test BandVAE’s performance against the other baselines.

Author response:
**A2:** The results from Table 1, which are averaged over 10 runs, show that the proposed BandVAE achieves statistically significantly higher recommendation accuracy than the closest performing model on Citeulike-a and Cell Phones datasets, demonstrating the efficacy of BandVAE.  

On MovieLens dataset, RecVAE achieves the second highest accuracy (after our method BandVAE). That RecVAE is based on the complex multi-layered encoder (which uses roughly 1.2M parameters (around 18% of RecVAE’s parameters) more than encoder of BandVAE) and the composite prior layer. On other datasets, RecVAE is much less competitive.

Reviewer comment (Experiments):
Why did the authors decide to adopt ratings as one of the modalities involved in recommendation? A common approach in recommendation leveraging multimodalities is to exploit visual and textual features of product images, which might represent an interesting direction to follow.

Author response:
**A1:**  Ratings serve as key information to make recommendations. Existing recommendation models adopt ratings to learn user and item embeddings then make recommendations based on the learned embeddings. Our paper follows this typical paradigm in recommender systems research.  

Textual content is widely available in popular benchmark datasets such as Citeulike, MovieLens and Amazon datasets (visual content is not available in Citeulike and MovieLens), so we adopt textual content to complement rating data, i.e., fusing user representations learned from texts with those learned from ratings to derive more expressive user representations. 

A future work could extend our proposed method to incorporate visual content by finding its projection onto rating space via barycentric mapping strategy then combining the projected embeddings of visual content with those from texts and ratings (as described in Section 4.2.2. Mapping and Fusing) to derive user embeddings, which could be more expressive representations and has the potential to improve recommendation performance.

Reviewer comment (Evaluation):
Can the authors provide justifications for the not-so-high improvement with respect to the tested baselines in Table 1?

Author response:
**A2:** The results from Table 1, which are averaged over 10 runs, show that the proposed BandVAE achieves statistically significantly higher recommendation accuracy than the closest performing model on Citeulike-a and Cell Phones datasets, demonstrating the efficacy of BandVAE.  

On MovieLens dataset, RecVAE achieves the second highest accuracy (after our method BandVAE). That RecVAE is based on the complex multi-layered encoder (which uses roughly 1.2M parameters (around 18% of RecVAE’s parameters) more than encoder of BandVAE) and the composite prior layer. On other datasets, RecVAE is much less competitive.

Reviewer comment (Experiments):
Lacking some real-world applications of this approach.

Author response:
Dear Reviewer **esB1**,

We would like to thank you for spending time reviewing our paper. We address your concern as following 

**Q:** **Lacking some real-world applications of this approach**

**A:** The proposed approach in this paper is tested on the recommender system, which is a real-world research problem. Not only is it applicable for rating and text modalities but the proposed method also works for alignment problems between disentangled factors from other mutually disentangled data modalities, e.g., ratings and images, images and texts.