Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark (2023)

Abstract: Existing multimodal task-oriented dialog data fails to demonstrate the diverse expressions of user subjective preferences and recommendation acts in the real-life shopping scenario. This paper introduces a new dataset SURE (Multimodal Recommendation Dialog with Subjective Preference), which contains 12K shopping dialogs in complex store scenes. The data is built in two phases with human annotations to ensure quality and diversity. SURE is well-annotated with subjective preferences and recommendation acts proposed by sales experts. A comprehensive analysis is given to reveal the distinguishing features of SURE. Three benchmark tasks are then proposed on the data to evaluate the capability of multimodal recommendation agents. Basing on the SURE, we propose a baseline model, powered by a state-of-the-art multimodal model, for these tasks.

## introduction building conversational agents that can communicate with people in multimodal situations is an attractive goal for the ai community. many different tasks and datasets for the multimodal dialog have been proposed in recent years. among them, moon et al. @xcite provided a multimodal task-oriented dialog dataset simmc 1.0 in two shopping domains. it aims to train interactive assistants which can handle multimodal inputs in a co-observed environment. the simmc challenge based on simmc 1.0 was held as part of dstc9 @xcite . many works @xcite @xcite @xcite have been done following simmc 1.0. since simmc 1.0 environment is simple and far from realistic stores, kottur et al @xcite proposed simmc 2.0 with closer-to-real-world shop-* corresponding authors. though these datasets facilitate research of conversational agents, they simplify some crucial problems in the real-life shopping dialog, which should be addressed for building multimodal recommendation agents. in previous tod datasets, most of the user descriptions for items (e.g., clothes or furniture) are referring expressions in the domains, such as "the white couch chair", and "the black hat in the middle of the long rack", which can be mapped to a slot value without ambiguity. while in practice, customers are not experts in the domains. they use lots of different words to describe what they want, such as "clothes designed for young ladies", and "color makes me feel quiet and calm" in fig. 1 . these words (or phrases) usually reflect customers' subjective cognition and preference for items they want. we call this kind of expression as subjective preference. to understand such expressions, the salesperson needs to map subjective preferences to standardized categorization concepts in the domains, and then use the concepts to filter candidate attribute values (slot values) of scene items. (fig. 1 shows the two-step mappings under the subjective preferences). as we can see, such subjective preferences often correspond to a set of slot values instead of a unique one, which is very different from that in previous datasets. facing this kind of customer requirement, a salesperson needs to communicate with the customers, utilize suitable strategies to narrow candidates progressively and give sound recommendations through multimodal context (fig. 1 shows some recommendation acts, such as ask preference). in a word, understanding subjective preferences, finding a way to clarify the subjective preferences, and finally giving good recommendations is the challenge not depicted in both simmc 1.0 and 2.0. none of the existing multimodal dialogs research focuses on subjective preference and item recommendation. to facilitate building conversational agents that can handle subjective preference and make shopping recommendations, we introduce a dataset for multimodal recommendation dialog with subjective preference (sure). specifically, we collect 12k salesperson ↔ customer goal-oriented recommendation dialogs in complex store scenes in two phases. dialog flows were first generated by self-playing between the carefully designed customer and salesperson simulators, which helps to ensure the flows are reasonable. crowd-sourcing is then employed for rewriting categorization concepts in dialog flows to diverse subjective preferences. the dataset contains well-annotated subjective preferences and diverse dialog acts proposed by experienced sales experts, which provides rich resources for evaluating subjective preferences understanding and dialog policies. we then propose three benchmark tasks for evaluating multimodal recommendation agents' capability on subjective preference understanding and item recommendation: subjective preference disambiguation, referred region understanding, and multimodal recommendation. we provide a baseline model for these tasks and highlight the key challenges and future research directions. our main contributions are: a strong baseline model mra is proposed for these tasks. textual conversational recommendation. the rise of e-books, music, and video websites has witnessed the development of conversational recommendation agents. existing agents all operate in the textual modal, which elicits user preferences via conversation and recommends items based on dialog history and structured attribute data. the redial (li et al., 2018) agent immediately recommends movies obtained from dbpedia (auer @xcite after the user expresses their preferences. tg-redial @xcite ) is developed by walking along conceptnet @xcite threads containing movies to collect user preferences and movie recommendations. gorec-dial @xcite dialogs are collected by game-play to recommend target movies from candidate sets. durecdial @xcite include chinese dialogs between movie seeker and conversational bot based on the knowledge graph. @xcite focuses on how social strategies adopted by the agent influence the final success rate of recommendation. although these works study conversational recommendation problems from different aspects including topic, strategy, and language, they are all based on textual modal. therefore, conversational agents built on these datasets cannot equip the abilities required recommendation strategy attribute categorization ask preference : color exclude preference : color answer preference : cold color answer preference : dark color ...... multimodal simulator interaction ...... dialogue generation subjective preference warm color color of passion cold color color makes people feel calm sport type clothes for yoga thick type warm clothes ...... ...... ...... ...... ...... ...... (1) multimodal dialogue flow simulation (2) manual paraphrase survey for sales experts leftmost shelf back table back display wall background item annotation text image in multimodal scenarios. besides, all of these researches ignore developing agents to respond to subjective preferences frequently appearing in real recommendation dialogs. ## sure dataset we build sure (multimodal recommendation dialog with subjective preference ) dataset to facilitate research on multimodal recommendation agents. in sure dialogs, customers express their preferences subjectively. to effectively recommend items, the salespersons perform: ❶ actively elicit customer preferences about item attributes; ❷ disambiguate subjective preferences according to the multimodal context; ❸ narrow the candidate set of items based on dialog history and scene; ❹ recommend target item from the situated scene. to collect sure dialogs, we design a two phrases pipeline (fig. 2 ) following popular machine ↔ human collaborative dialog collection approaches @xcite @xcite . in this section, we will introduce the sure two phrases collection process in order and then analyze the distinguishing features of sure. ## task formulation we propose three tasks on sure dataset to evaluate multimodal recommendation agents. the tasks of subjective preference disambiguation and referred region understanding evaluate the side of multimodal understanding, while multimodal recommendation evaluates the side of policy learning. ## modeling & empirical analysis dataset split. sure is randomly divided into 4 parts: train (65%), dev (5%), dev-test (15%), and test-std (15%). we leave test-std as a heldout hidden set for performing a fair comparison of models in future potential competition. baseline. we proposed mra(multimodal recommendation agent) model as the baseline model for subjective preference disambiguation, referred region understanding, and multimodal recommendation tasks. the backbone of the mra model is encoder-decoder based single-stream visual-language pre-training model, which are stacks of transformer @xcite layers. the scene image is split into p patches. and each patch is projected to the visual embedding of the model's hidden size. the flattened dialog history and non-visual metadata are converted to subword sequences by byte-pair encoding (bpe) and then embedded into textual embedding. all visual embedding and textual embedding are concatenated as model input. the mra model completes three benchmarks at the same time by generating next salesperson act, candidate attribute values / referred region items and agent response auto-regressively as fig. 5 shows. note that mra will predict candidate attribute value if the latest round relates to eliciting preference. when the latest round relates to referring region, mra predicts object ids in the region. implementation details. mra model is based on transformer (vaswani et al., 2017) structure with 12 layers, where every transformer block has 768 hidden units and 12 attention heads. textual and visual embedding are projected to features the same size as the hidden units. we initialize mra parameters from pretrained ofa-base (wang et al., 2022) model. during training, mra model is trained for 20 epochs with 18 batch sizes to optimize language modeling loss. at the end of every epoch, the model is evaluated on dev split to save the best model parameters. the hyperparameters are determined by area search. adam (kingma and ba, 2015 ) is adopted as the optimizer with a 4e-4 learning rate while the dropout rate is set to 0.2 to prevent over-fitting. the whole training costs around 36 tesla-v100 gpu hours. note that the bleu-4 score is calculated by nltk @xcite package in the evaluation. analysis and future work as tab. 4 shows, model mra, powered by an advanced multimodal backbone, fails to perform well on three sure benchmarks. from the case study (fig. 20 ), we can observe that it is difficult for mra to accurately understand subjective preferences (task 1) and referred region (task 2), which further hinders the model from adopting suitable acts to make correct recommendations (task 3). ablation of metadata and scene snapshot greatly weakens mra on the first two tasks. it indicates that effective utilization of metadata and scene snapshot plays an essential role in model performance. future work can be done by designing modules or proposing multimodal pretraining tasks to facilitate model's understanding of subjective preferences, perception of referred region, and ability to take suitable acts. ## conclusion we introduce multimodal recommendation dialog with subjective preference (sure) dataset with 12k salesperson ↔ customer dialogs and 3k subjective preferences to study how to recommend item from complex scene based on subjective preferences. our proposed three benchmarks and strong baseline model mra address the new challenges and directions in the multimodal recommendation dialog.