User Factor Adaptation for User Embedding via Multitask Learning (2021)

Abstract: Language varies across users and their interested fields in social media data: words authored by a user across his/her interests may have different meanings (e.g., cool) or sentiments (e.g., fast). However, most of the existing methods to train user embeddings ignore the variations across user interests, such as product and movie categories (e.g., drama vs. action). In this study, we treat the user interest as domains and empirically examine how the user language can vary across the user factor in three English social media datasets. We then propose a user embedding model to account for the language variability of user interests via a multitask learning framework. The model learns user language and its variations without human supervision. While existing work mainly evaluated the user embedding by extrinsic tasks, we propose an intrinsic evaluation via clustering and evaluate user embeddings by an extrinsic task, text classification. The experiments on the three English-language social media datasets show that our proposed approach can generally outperform baselines via adapting the user factor.

## introduction language varies across user factors including user interests, demographic attributes, personalities, and latent factors from user history. research shows that language usage diversifies according to online user groups @xcite , which women were more likely to use the word weakness in a positive way while men were the opposite. in social media, the user interests can include topics of user reviews (e.g., home vs. health services in yelp) and categories of reviewed items (electronic vs kitchen products in amazon). the ways that users express themselves depend on current contexts of user interests @xcite that users may use the same words for opposite meanings and different words for the same meaning. for example, online users can use the word "fast" to criticize battery quality of the electronic domain or praise medicine effectiveness of the medical products; users can also use the words "cool" to describe a property of ac products or express sentiments. user embedding, which is to learn a fixed-length representation based on multiple user reviews of each user, can infer the user latent information into a unified vector space @xcite . the inferred latent representations from online content can predict user profile @xcite @xcite and behaviors @xcite @xcite . user embeddings can personalize classification models, and further improve model performance @xcite @xcite @xcite . the representations of user language can help models better understand documents as global contexts. however, existing user embedding methods @xcite @xcite ) mainly focus on extracting features from language itself while ignoring user interests. recent research has demonstrated that adapting the user factors can further improve user geolocation prediction @xcite , demographic attribute prediction @xcite , and sentiment analysis @xcite . @xcite ; @xcite treated the language variations as a domain adaptation problem and referred to this idea as user factor adaptation. in this study, we treat the user interest as domains (e.g., restaurants vs. home services domains) and propose a multitask framework to model language variations and incorporate the user factor into user embeddings. we focus on three online review datasets from amazon, imdb, and yelp containing diverse behaviors conditioned on user interests, which refer to genres of reviewed items. for example, if any yelp users have reviews on items of the home services, then their user interests will include the home services. we start with exploring how the user factor, user interest, can cause language and classification variations in section 3. we then propose our user embedding model that adapts the user interests using a multitask learning framework in section 4. research @xcite generally evaluates the user embedding via downstream tasks, but user annotations sometimes are hard to obtain and those evaluations are extrinsic instead of intrinsic tasks. for example, the mypersonality @xcite that was used in previous work @xcite @xcite is no longer available, and an extrinsic task is to evaluate if user embeddings can help text classifiers. research @xcite suggests that the intrinsic evaluation including clustering is better than the extrinsic evaluation for controlling less hyperparameters. we propose an intrinsic evaluation for user embedding, which can provide a new perspective for testing future experiments. we show that our user-factor-adapted user embedding can generally outperform the existing methods on both intrinsic and extrinsic tasks. ## data we collected english reviews of amazon (health product), imdb and yelp from the publicly available sources @xcite @xcite . for the imdb dataset, we included english movies produced in the us from 1960 to 2019. each review associates with its author and the rated item, which refers to a movie in the imdb data, a business unit in the yelp data and a product in the amazon data. to keep consistency in each dataset, we retain top 4 frequent genres of rated items and the review documents with no less than 10 tokens. 1 we dropped non-english review documents by the language detector @xcite , lowercased all tokens and tokenized the corpora using nltk @xcite . the review datasets have different score scales. we normalize the scales and encode each review score into three discrete categories: positive (> 3 for the yelp and amazon, > 6 for the imdb), negative (< 3 for the yelp and amazon, < 5 for the imdb) and neutral. ## exploratory analysis of user variations language varies across user factors such as user interests @xcite , demographic attributes @xcite , social relations @xcite . in this section, our goal is to quantitatively analyze whether the user interests cause user language variations, which can reduce effectiveness and robustness of user embeddings. we approach this by two analysis tasks, first by measuring word feature similarity based on user interests, and second by examining how classifier performance depends on the grouped user interests in which the model is trained and applied. ## multitask user embedding we present the architecture of our proposed model in figure 3 on the left. methods @xcite to train text-based user embedding mainly focus on the user-generated documents while ignoring user factors, the user interests. a close work to ours only trained user embeddings by predicting if users co-occurred with sampled words @xcite . we extend this line of work by adapting user interests into modeling steps. the proposed unsupervised model trains four joint tasks based on the skip-gram @xcite : word and word, user and word, item and word, and user and item. note that we do not use the categories of rated items and user interests in our training steps. then we can optimize the model by minimizing the following loss function: @xmath0, w) + l(u, w) + l(p, w) + l(p, u) where w, u, p are the notations of words, users and rated items respectively. considering the large size of the vocabulary, users and rated items, we approximate our optimization objectives by the negative sampling. then we can treat each task as a classification problem and calculate loss values by the binary cross-entropy. we present the details of each optimization task as following: word and word is a standard way to train word2vec @xcite models. the prediction task is to predict if the sampled words have co-occurred within the window context. the training process uses the negative sampling to approximate objective function. we choose 5 as the number of negative samples. we keep the top 20,000 frequent words and finally replace the rest with a special token, < unk >. user and word predicts if a user authored the sampled words by the contexts of user posts. the goal is to learn patterns of user language usage from user historical posts. given a document i, its author u i and the user's vocabulary v u @xmath1,i , ..., w j,i , ..., w n,i }, where n is the number of frequent words authored by the user. our objective is to minimize the following function: item and word follows the prediction task of user and word to classify if sampled words describe the selected item. this task is to use review documents to train representations of rated items. then we can have l(p, w) =w @xmath2,w @xmath3,w k / ∈vp i log(1 -𝜃 𝑢 • 𝑝 𝑖𝑓 𝑝𝑢𝑟𝑐ℎ𝑎𝑠𝑒 𝜃 𝑢 • 𝑤 𝑖𝑓 ℎ𝑎𝑠 𝜃 𝑤 𝑖 • 𝑤 𝑗 𝑖𝑓 𝑐𝑜𝑛𝑡𝑒𝑥𝑡 𝜃 𝑝 • 𝑤 𝑖𝑓 ℎ𝑎𝑠 docs embedding d 1 : w 1 , w 2 , …, w n d k : w 1 , w 2 , …, w n doc encoder + predictor word embedding product embedding user embedding multitask user embedding personalized document classifier θ(e user and item learns if a user commented on the sampled items. the prediction task aims to adapt latent user factors into the user embeddings. given a document i, its author u i and the reviewed item p i , we can optimize the task by minimizing for model settings, we used adam (kingma and ba, 2014) for the model optimization with a learning rate of 1e-5. we set the training epochs as 5. the model initializes embedding vectors randomly and learns 300-dimension representations for words, users and reviewed items. we empirically use 5 as the number of negative samples. for the other parameters, we keep the same as the defaults in the keras (chollet and others, 2015). ## experiments we evaluate the effectiveness of the user factor adapted embedding model by an intrinsic evaluation, user clustering task and an extrinsic evaluation, personalized classification task. the first task aims to measure the purity of clusters with respect to categories of user interests, and the second task uses the document classification as a proxy of qualifying quality of user embeddings. we conduct a qualitative analysis of the user embeddings comparing with our close work @xcite . ## conclusion in this study, we have proposed user factor adaptation for building user embedding under a multitask framework. our analyses show how the user factor causes semantic variations in relation to word usage and document classification, showing that the user factor is rooted in language. we have evaluated the proposed user embedding mod-els in both intrinsic and extrinsic tasks. the user factor adapted model has shown its robustness to language variations in both instrinsic and extrinsic evaluations, learning user representations and personalizing classifiers. we release our source code and instructions of data access at https: //github.com/xiaoleihuang/userembedding. our work in user factor adaptation highlights several future directions to explore. first, our method models latent user factors inferred from user posts. a combination of user embedding and explicit attributes (e.g., demographic factors) may improve model personalization. second, user behaviors shift over time. a time-adapted user embedding can jointly model temporality and user attributes in online social media and can be extended to other fields, such as public health.