CHARM : Inferring Personal Attributes from Conversations (2020)

Abstract: Personal knowledge about users’ professions, hobbies, favorite food, and travel preferences, among others, is a valuable asset for individualized AI, such as recommenders or chatbots. Conversations in social media, such as Reddit, are a rich source of data for inferring personal facts. Prior work developed supervised methods to extract this knowledge, but these approaches can not generalize beyond attribute values with ample labeled training samples. This paper overcomes this limitation by devising CHARM: a zero-shot learning method that creatively leverages keyword extraction and document retrieval in order to predict attribute values that were never seen during training. Experiments with large datasets from Reddit show the viability of CHARM for open-ended attributes, such as professions and hobbies.

## introduction motivation. personal knowledge bases (pkbs) capture individual user traits for customizing downstream applications like chatbots or recommender systems @xcite . a potentially automatic way to populate a pkb is to draw personal knowledge from the user's conversations in social media and dialogues on other platforms. these interactions are a rich source of personal attributes, such as hobbies, professions, cities visited, medical conditions (experienced by the user) and many more. each of these would consist of key-value pairs, such as cities visited:paris or symptom:dizziness. however, a large number of potential attributes and their respective values makes this a challenging task. in particular, there is little hope to have training data for each of these key-value pairs. moreover, the textual cues in user conversations are often implicit and thus difficult to learn. example. consider the user's utterance: "i just visited london, which was a disaster. my hotel was a headache and i spent half the time in bed with a fever... so glad to be back home finishing the masts on my galleon." as humans, we can infer the following attribute-value pairs: (a) cities visited:london, (b) symptom:fever, (c) hobby:model ships. capturing such user traits is a daunting task, however, with both implicit and explicit signals present. we need to consider the context "spent in bed with", to infer that fever relates to a disease (as opposed to headache). to predict the user's hobby model ships, we have to pay attention to the cues 'galleon' and 'mast'. proper inference requires both deep language understanding and background knowledge (e.g., about ships, cities, etc.). ## methodology overview. as illustrated in figure 1 , charm consists of two stages: cue detection and value ranking. as input charm receives a user's utterances @xmath0..u n that contain a set of terms t 0 ..t m , for example, @xmath1, "studied for the exam so i could have better grades than my classmates"}. in the first stage, the term scoring model assigns a score to each term in the user's utterances, yielding l 0 ..l m . the highest scoring terms are then selected to form a query @xmath2..q k characterizing the user's correct attribute value, e.g., @xmath3. in the second stage, q is evaluated against an external document collection @xmath4..d l ; each document in d is associated with possible attribute values. documents such as wiki:student and wiki:dean's list, which are associated with the attribute value student, would score high with the example query. the score aggregator then ranks the attribute values based on the documents' scores s 0 ..s l , for instance, yielding a high attribute score for student given our example utterances. the list of attribute values v is known in advance (e.g., taken from wikipedia lists); however, potentially only a subset of values s ⊂ v have instances seen during training. ## experimental setup we evaluate the proposed method's performance in two experimental settings. first, we consider a zero-shot setting in which the attribute values in the training and test data are completely disjoint (i.e., the test set only contains unseen labels). this setting evaluates how well charm can predict attribute values that were not observed during training. second, we consider the standard classification scenario in which all attribute values are seen as labels in both training and test sets.this demonstrates that charm's performance in a normal classification setting does not substantially degrade because of its proposed architecture. experimental setup details differ for these two evaluation settings, which will be discussed in the following subsections. all our models were implemented in pytorch; technical details are in appendix b. the code and labeled datasets will be made publicly available upon acceptance. training and test data. for the unseen experiments, we perform ten fold cross-validation with folds constructed such that each attribute value appears in only one test fold. each of the folds contains roughly the same number of users and approximately 2-4 unique attribute values. 6 we assigned the users having multiple attribute values to a fold corresponding to one of their randomly chosen values. for the experiments with seen values, we randomly split the users into training and test sets in a 9:1 proportion, respectively. hyperparameters. bert, the term selection component, generates a contextualized embedding for each input term, which we process with a fully connected layer to produce a term score for each word in its context. specifically, we use the pre-trained bert base-uncased model with 12 transformer layers. to reduce bert's computational requirements, we discard the last 6 transformer layers (i.e., 6 we used a greedy algorithm to approximate a solution to the np-hard bin packing problem. during training, we sample 5 negative labels (i.e., incorrect attribute values) to be ranked when calculating the ndcg reward. for each label, we sample a subset of 15 documents to represent the label (i.e., attribute value). if the document collection has fewer than 15 documents for a label (e.g., wikipage), we consider all the label's available documents. when making predictions, we consider all documents and all labels (values). in both settings, we truncate documents to 800 terms when using knrm for efficiency and use the full documents with bm25. we use ten fold cross-validation on the training data to optimize the following hyperparameters in a grid search: (i) document aggregation strategy (average vs max); (ii) length of query; and (iii) maximum number of epochs. further details on the hyperparameter search are in appendix b. baselines. for the unseen experiments, we evaluate charm's performance against an end-to-end bert ranking method and against a bm25 (robertson and zaragoza, 2009) ranker combined with two state-of-the-art unsupervised keyword extraction methods: textrank and rake. we additionally include a baseline giving the user's full utterances as input to bm25 (baseline: no-keyword). following related work @xcite , we train the bert ir baseline using a binary cross-entropy loss to predict the relevance of each document to the user's utterances (acting as queries). we use the same pre-trained bert model as in charm. to fit both utterances and documents into the input size of bert, we split both into 256-token chunks and run bert on their cartesian product. to obtain the final score for each utterances-document pair we average across all chunk pairs. given n utterances and m documents, this baseline processes n × m inputs with bert, whereas charm processes n inputs with bert and m inputs with an efficient ranking method. this makes the bert ir baseline very computationally expensive on the wiki-category and web search document collec ## conclusion we presented the charm method for inferring personal traits from conversations. charm differs from prior work by its zero-shot ability to predict attribute values that are not present in the training samples at all. we demonstrated the viability of charm for inferring users' unseen attribute values by comprehensive experiments with reddit conversations, leveraging document collections from wikipedia and web search results for charm's retrieval component.