Multimodal Review Generation with Privacy and Fairness Awareness (2020)

Abstract: Users express their opinions towards entities (e.g., restaurants) via online reviews which can be in diverse forms such as text, ratings, and images. Modeling reviews are advantageous for user behavior understanding which, in turn, supports various user-oriented tasks such as recommendation, sentiment analysis, and review generation. In this paper, we propose MG-PriFair, a multimodal neural-based framework, which generates personalized reviews with privacy and fairness awareness. Motivated by the fact that reviews might contain personal information and sentiment bias, we propose a novel differentially private (dp)-embedding model for training privacy guaranteed embeddings and an evaluation approach for sentiment fairness in the food-review domain. Experiments on our novel review dataset show that MG-PriFair is capable of generating plausibly long reviews while controlling the amount of exploited user data and using the least sentiment biased word embeddings. To the best of our knowledge, we are the first to bring user privacy and sentiment fairness into the review generation task. The dataset and source codes are available at https://github.com/ReML-AI/MG-PriFair .

## introduction users generate digital footprints when "traveling" on the internet. modeling this behavioral data is useful to understand users' preferences. for example, amazon infers users' preferences based on their views, add-to-card, or purchase actions. likewise, online reviews explicitly manifest how users opine about business entities such as restaurants. figure 1 shows an example of online reviews on yelp.com, that expresses user's opinions about food and service of a sushi restaurant, along with images and rating score. containing invaluable information of personal opinions, online reviews become an essential data source that is modeled in diverse tasks to comprehend users @xcite , e.g., sentiment analysis or review generation. in this paper, we study the task of review generation using multi modalities including image, user and entity information while taking into account user privacy and sentiment fairness. specifically, we present a framework, namely mg-prifair, which includes privacy and fairness controllers to preprocess data and a neural-based generation model to generate personalized reviews. reviews are user-generated contents that may contain personal information leading to privacy concerns. for example, the content and images of the review in figure 1 signify sensitive information about the reviewer, i.e., j. h. in daly city has a son named wah who might be born on 8 may. this observes figure 1 : an example of online reviews on yelp with personal information. the problem of revealing personal information of an individual by observing outputs of the model trained using user-generated data (i.e. reviews) (rocher and de montjoye, 2019). to address this problem, we apply a privacy controller, which is a two-stage approach to minimize the use of personal information. first, we propose dpsenti, a novel approach to learn differentially private word embeddings (dp-embedding), from which we infer user and entity representations (uers). next, we freeze these representations during the training process, to avoid further use of personal information. there are two levels of privacy protection in this task: individual privacy and model privacy. in the scope of this work, we focus on the former one. fairness is "the absence of any bias" @xcite . with the rapidly increasing number of machine learning applications in daily life, developing learning models that are fair with respect to sensitive attributes (e.g., gender, race) of the training data has become important. in the context of writing review, sentiment fairness is an issue raising at the individual level such as a restaurant, a product, or a dish. sentiment bias can come from training reviews or external data. for example, when we use word embeddings trained using external data, some words describing a dish might be highly correlated to negative words causing the model to generate negative-sentiment reviews for the dish. in this paper, we focus on the bias causing by pretrained word embeddings models. specifically, we propose an evaluation approach to measure sentiment bias for the food-review domain, thus assisting to select the least bias pretrained model. our contributions are three-fold. first, we propose a new dp-embedding (i.e., dpsenti) approach for training privacy guaranteed word embeddings for the task of review generation. secondly, we propose an evaluation approach for sentiment fairness in food-review domain. we also run the evaluation across multiple pretrained language models to evaluate their sentiment fairness for the domain. thirdly, to the best of our knowledge, we are the first to introduce the notions of user privacy and sentiment fairness for the task of review generation. we evaluate extensively and present insights on multiple tasks ranging from dp-embeddings, sentiment fairness, to review generation. additionally, the novel dataset is released with initial benchmark results for this task. ## related work we conduct a literature review in text generation, review generation, user privacy and fairness topics. text generation. the closest tasks to review generation are image captioning and review generation. in image captioning, the objective is to automatically generate text to describe the content of an image via learning the correlation between vision and textual features @xcite . @xcite tackle the sequence generation problem, which applies neural machine translation and image captioning techniques with a new target-target attention mechanism on target sequences. in order to generate personalized captions, chunseong @xcite present context sequence memory network to take into account users' historical activities. generally, review generation is different from image captioning since it requires additional input (i.e., user and entity information), and the target is not only to capture what is inside an image, but also to "express" opinions toward the entity being reviewed. review generation recently has received more attention. @xcite propose a graph-based approach to identify representative review snippets supporting to construct a review. @xcite propose an encoder-decoder network architecture that takes user/product attributes and ratings as input for personalized review generation. @xcite seek to learn aspect-aware user and item representations to generate reviews based on short phrases as input. in comparison with our proposed model, these reviewed works do not deal with visual input. @xcite introduce a multimodal review generation (mrg) to simultaneously predict ratings and generate reviews using information from users, items, and images. their objective is to learn user preferences through predicting ratings and generate short reviews, whereas, we aim at generating relatively longer reviews. we compare our proposed model with mrg in section 4. user privacy. preserving user privacy has been studied for decades. the techniques of anonymization @xcite and sanitization @xcite have been widely applied. differential privacy later emerged as the key privacy guarantee by providing rigorous, statistical guarantees against any inference from an adversary @xcite . differential privacy has been applied in many research including text data @xcite . this motivates the use of differential privacy for review generation task. we propose to decrease the use of user information to reduce privacy leakage risk. there have been some works @xcite in learning differentially private language models. however, this paper aims at finding word representations for the review generation task, which has to preserve good sentiment. therefore, we propose a different neural model to learn differentially private word embeddings for the sentiment classification task. fairness. there is an increasingly important concern as machine learning models are utilized to support decision making in high-stakes applications, e.g., mortgage lending, hiring, and prison sentencing (r. k. e., 2019). @xcite present an empirical example for college admissions that the inclusion of variables (e.g., race) can increase both equity and efficiency. b fish (2016) investigate algorithmic fairness and maintained the high accuracy of three learning algorithms while reducing the degree to reduce discrimination against individuals. conceptnet numberbatch 17.04 @xcite ) (hereafter conceptnet) has been well known for having good semantic representation while addressing several word-embedding biases (e.g., gender bias and religious bias). however, it does not resolve sentiment bias for the food domain (section 4). in this paper, we propose an evaluation approach that measures sentiment bias for the food-review domain to select a "less-bias" pretrained word embeddings model for the task of review generation. de-biasing sentiment bias is not in the scope of this work. in this section, we present our proposed multimodal review generation with privacy and fairness awareness framework, mg-prifair. as shown in figure 2a , mg-prifair consists of three main components: privacy controller, fairness controller, and personalized review generation model (prgen). privacy controller manages the use of personal information by injecting noise while learning differentially private representations for users and entities. the fairness controller measures sentiment bias of different word embeddings sets to select the least biased one. the preprocessed data is then passed to prgen to train the generation model (figure 2b ). we formulate the problem of multimodal review generation as follows. ## multimodal review generation with privacy and fairness awareness problem definition. let us denote the sets of users and entities as u and e, respectively. the dataset x consists of tuples (m, y, u, e), where image m is associated with review document y written by user @xmath0. each review document is a set of review sentences. given dataset x , the objective is to build a model that learns to generate a review document given an image m and the associated information of user u and entity e. ## comparing with text generation baselines figure 5 shows examples of reviews generated by our model (prgen b -py) and the two baselines. showntell has grammatical issues affecting the readability of the generated reviews and the contents are also not relevant to the input images. mrg is able to generate grammatically correct reviews, but tend to generate similar contents for different images. our model, however, is able to generate relevant and grammatically correct reviews. sometimes, generated reviews describe other aspects instead of only describing the image, e.g. the 1 st image. with these results, our proposed review generation model demonstrates the efficiency on generating reviews with quality and readability. ## conclusion this paper proposed mg-prifair, a multimodal neural-based framework, to automatically generate personalized reviews to understand user behaviors. mg-prifair is aware of user privacy and sentiment fairness. our extensive empirical experiments show the efficiency of the proposed framework in generating plausible reviews while taking into account user privacy and sentiment fairness. to the best of our knowledge, we are the first to raise the concerns of user privacy and sentiment bias for the review generation task. as a future work, the privacy of images can be concerned. for example, a taken photo of a restaurant may capture human faces. one potential solution to protect image-level privacy is to detect regions in images having sensitive-personal information and exclude those before sending to the model.