Train Once, Deploy Anywhere: Matryoshka Representation Learning for Multimodal Recommendation (2024)

Abstract: Despite recent advancements in language and vision modeling, integrating rich multimodal knowledge into recommender systems continues to pose significant challenges. This is primarily due to the need for efficient recommendation, which requires adaptive and interactive responses. In this study, we focus on sequential recommendation and introduce a lightweight framework called full-scale Matryoshka representation learning for multimodal recommendation (fMRLRec). Our fMRLRec captures item features at different granularities, learning informative representations for efficient recommendation across multiple dimensions. To integrate item features from diverse modalities, fMRLRec employs a simple mapping to project multimodal item features into an aligned feature space. Additionally, we design an efficient linear transformation that embeds smaller features into larger ones, substantially reducing memory requirements for large-scale training on recommendation data. Combined with improved state space modeling techniques, fMRLRec scales to different dimensions and only requires one-time training to produce multiple models tailored to various granularities. We demonstrate the effectiveness and efficiency of fMRLRec on multiple benchmark datasets, which consistently achieves superior performance over state-of-the-art baseline methods. We make our code and data publicly available at https://github.com/yueqirex/fMRLRec.

## introduction recent advancements in language and multimodal modeling demonstrates significant potential for improving recommender systems @xcite @xcite . such progress can be largely attributed to: despite performance improvements, different recommendation scenarios (e.g., centralized or federated recommender systems) often require varying granularities (i.e., model / dimension sizes) in item representations to achieve the balance between performance and efficiency @xcite @xcite . for instance, larger dimensions are typically required to encode language and vision features for fine-grained understanding and generation tasks, although marginally lower performance can often be achieved using considerably smaller feature sizes @xcite . to identify the optimal granularity for specific use cases in recommendation systems, methods like grid search or adaptive search heuristics are frequently utilized in training @xcite . however, such searches can lead to substantial training expenses or fail to identify the optimal model, particularly when given a large configuration space and constrained computational resources. therefore, a train-once and deploy-anywhere solution is optimal for the efficient training of recommender systems, which should ideally meet the following criteria: inspired by matryoshka representation learning (mrl) @xcite , we introduce a lightweight multimodal recommendation framework named full-scale matryoshka representation learning for recommendation (fmrlrec). fmrl-rec embeds smaller vector/matrix representations in larger ones like matryoshka dolls and is only trained once without additional computation costs. different from mrl that only embeds smaller finallayer activations into larger ones during training, fmrlrec pushes the efficiency of mrl training by introducing an efficient linear transformation that embeds both smaller weights and activations into larger ones, thereby reducing memory costs associated with both aspects. this approach is particularly effective for training recommender systems on large-scale data, offering a highly efficient framework for multi-granularity model training. combined with further improvements in statespace modeling represented by @xcite @xcite , the linear recurrence architecture in fmrlrec delivers both effectiveness and efficiency in recommendation performance across various benchmark datasets. we summarize our contributions below foot_0 : ## fmrlrec memory efficiency in this subsection, we analyze fmrlrec modelseries memory efficiency by driving the number of parameters plus activations needed to train model sizes of @xmath0, 4, 8, . . . , d} or @xmath1, 2, . . . , k} as (1) a train-once fmrlrec modelseries and (2) independent models. define w case 1: for fmrlrec-based training, number of parameters needed 2k) . case 2: for independent training, the number of parameters needed this indicates a parameter saving rate r s of ≈ 0.33 against the fmrlrec model. empirically, for a common setting @xmath2, the weights saved are approximately 4(n) × 0.33(r) × 512(d) × 1024(2d) ≈ 700k, the number of activation saved for four layer is approximately 4(n) × 0.33(r) × 32(b) × 50(l) × 1024(2d) ≈ 2m . this is to a great extent saving memory usage if independent training is executed in parallel or saving training time if executed sequentially. ## conclusions in this work, we introduce a lightweight framework fmrlrec for efficient multimodal recommendation across multiple granularities. in particular, we adopt matryoshka representation learning and design an efficient linear transformation to embeds smaller features into larger ones. moreover, we incorporate cross-modal features and further improves the state-space modeling for sequential recommendation. consequently, fmrlrec can yield multiple model sizes with competitive performance within a single training session. to validate the effectiveness and efficiency of fmrlrec, we conducted extensive experiments, where fmrlrec consistently demonstrate the superior performance over state-of-the-art baseline models.