I - AM - G : Interest Augmented Multimodal Generator for Item Personalization (2024)

Abstract: The emergence of personalized generation has made it possible to create texts or images that meet the unique needs of users. Recent advances mainly focus on style or scene transfer based on given keywords. However, in e-commerce and recommender systems, it is almost an untouched area to explore user historical interactions, automatically mine user interests with semantic associations, and create item representations that closely align with user individual interests.In this paper, we propose a brand new framework called **I**nterest-**A**ugmented **M**ultimodal **G**enerator (**I-AM-G**). The framework first extracts tags from the multimodal information of items that the user has interacted with, and the most frequently occurred ones are extracted to rewrite the text description of the item. Then, the framework uses a decoupled text-to-text and image-to-image retriever to search for the top- K similar item text and image embeddings from the item pool. Finally, the Attention module for user interests fuses the retrieved information in a cross-modal manner and further guides the personalized generation process in collaboration with the rewritten text.We conducted extensive and comprehensive experiments to demonstrate that our framework can effectively generate results aligned with user preferences, which potentially provides a new paradigm of **Rewrite and Retrieve** for personalized generation.

## introduction if you love adventure movies, which poster in figure 1 would catch your eyes and make you itch to go to the cinema? or, if the monotonous uniformity of clothing has led to your aesthetic fatigue, but the platform generates a unique adventure-themed t-shirt based on your past interests, would you be excited and buy it without hesitation? in this era of information homogenization @xcite al., i-am-g pocahontas directed by mike gabriel & eric goldberg character elegance adventure objective item generate posters with difference preference 2019), a personalized t-shirt that resonates with your personal flair, a movie poster that highlights your favorite idol and style, or an illustration from a news article that piques your interest could all spark your desire to explore. in real-scenarios like recommender systems, personalized generation is a strategy to win over a diverse user base, because custom-generated images can precisely cater to user preferences for items. moreover, it facilitates users to discover the content they truly enjoy. despite the high potential and practical use in online advertisement platforms, exploring user preferences for personalized generation remains largely unexplored. most works focus on mining user interests and recommending based on existing item information @xcite , but do not involve content customization for users. early advances in generation have focused on using keywords for style transfer @xcite or facial transformations @xcite on given images, without considering user preference information. an intuitive idea is to let users input keywords based on their preferences to generate images they like. however, users often cannot precisely express their interests, making it difficult for the generator to create personalized representations. recent study @xcite has attempted to address this issue, but it is still tough to extract user interests for satisfactory generation results. specifically, the challenges are twofold. first is how to explore user interests. due to the presence of preference ambiguity @xcite , the interests expressed by users may be subjective and obscure, which is also a common preference elicitation problem in psychology. for example, you might know you like fast and furious, but it's hard to pinpoint what exactly you like about it. similarly, you enjoy watching forrest gump, but it's difficult to summarize what both movies have in common that attracts you. second is how to let generator better utilize the interests. although a user's interest can largely represent their preferences, there is still a gap for creating images that users truly like. therefore, it is necessary to mine the relationship between user interests and item semantic representations. for instance, cute might mean a bow in outfit generation, but in movie posters, it may mean a cartoon style or brilliant color. if user interests can be used to explore semantically similar items as supplementary features for the objective item, then their knowledge could guide the personalized generation process. to tackle the challenges above, we propose interest augmented multimodal generator (i-am-g) for item personalization. given the issue of user preference ambiguity, we propose the interest rewrite strategy, which uses pre-trained language models to extract various modal tags for items. for each user, we aggregate these tags from their historical interactions and select the most frequent ones as the user's explicit interests. these interests are then filled into templates to rewrite item descriptions. moreover, to compensate for the semantic relationships ignorance, we innovatively introduce the interest retrieve attention (ira) module. this module explores the relevance of interests based on embeddings from different modalities of items. it first uses retrieval techniques to find the top-k most similar items to the rewritten description in text, and top-k images similar to the objective item as references. then, the attention mechanism @xcite focuses on the relevant parts of text embeddings in the item pool, and integrates image modal information in a cross-modal manner, providing the generator with solid representational basis. based on the aforementioned features of rewritten text and the fused semantic information from single and cross-modalities, the generator is trained under these informative condition, and generates image representations that match user interests. we conduct extensive experiments of the generated results, including human study and gpt evaluation, to demonstrate the superiority of our framework. in summary, our contributions are threefold: 1) we highlighted the significant gap in existing generation models concerning personalized generation based on user interests, and we identified two major challenges: preference ambiguity and semantic correlation ignorance. 2) we proposed the i-am-g framework for item personalized generation, which includes interest rewrite and interest retrieve attention. these components aggregate user preference tags and deeply integrate semantic information, respectively, providing personalized features for model training and inference. 3) we conducted extensive experiments with various metrics. the generated results demonstrated the effectiveness of our method, which is expected to facilitate further development of personalized generation. our code is available at https://github.com/xqwustc/i_am_g . ## preliminaries and problem setup we conclude that the goals of personalized item generation are primarily divided into two categories. one category involves generating items that do not actually exist according to the preferences of the users, such as custom clothing and furniture. the other category is to generate different multimodal representations of existing subjects according to the diverse interests of users, such as poster creation and thumbnail generation. the common goal of these two types of generation is to mine and attract the interest of users. for each item i, it is defined as a user u interacts with numerous items, and for the user's objective item i, we utilize historical item features along with the current objective item information to personalized item representation. the personalized generation results are given by: the ultimate goal of the generation results is to attract a specific user; however, there is currently no definitive metric to gauge whether a user prefers a particular generation result. in practice, the mean squared error (mse) loss can be employed to guide the training process for the image modality like @xmath0, i i ). ## methodology this section presents our personalized generation framework. we use large language model (llm) @xcite and vision language model (vlm) @xcite to extract tags for interest rewrite, and detail the training process based on the widely used diffusion model. ## experiment our experiment focuses on the five questions. due to space constraints, we add two interesting experiment details included in appendix b. ## conclusion for the goal of item personalized generation, we developed the i-am-g framework. first, we proposed the interest rewrite strategy to tackle the preference ambiguity issue. this strategy leverages pre-trained language models to extract tags for each item's multiple modalities, and uses the tags to rewrite the description of a given item for the user. furthermore, we introduced ira, an attention mechanism that retrieves the most relevant other items, and adaptively integrates cross-modal features based on their semantic similarity to the rewritten sentence. extensive experiments demonstrated the effectiveness of our approach, potentially offering new insights for the field of item personalization in online advertisement.