DAPPER : Learning Domain-Adapted Persona Representation Using Pretrained BERT and External Memory (2020)

Abstract: Research in building intelligent agents have emphasized the need for understanding characteristic behavior of people. In order to reflect human-like behavior, agents require the capability to comprehend the context, infer individualized persona patterns and incrementally learn from experience. In this paper, we present a model called DAPPER that can learn to embed persona from natural language and alleviate task or domain-specific data sparsity issues related to personas. To this end, we implement a text encoding strategy that leverages a pretrained language model and an external memory to produce domain-adapted persona representations. Further, we evaluate the transferability of these embeddings by simulating low-resource scenarios. Our comparative study demonstrates the capability of our method over other approaches towards learning rich transferable persona embeddings. Empirical evidence suggests that the learnt persona embeddings can be effective in downstream tasks like hate speech detection.

## introduction with increasing human-machine hybrid technologies, the real-world interactions with ai systems are often stilted. this shortcoming can be attributed to the lack of shared common knowledge about how people will act, communicate and react under different circumstances. several studies in the field of psychology @xcite @xcite have established the role of personas in governing how people process information, attend to and interpret life-experiences, and respond to social situations. specifically, the relationship between personality and natural language have been widely studied @xcite . for example, a narcissistic person might make frequent use of first-person expressions (i, me, myself, for me, etc.). therefore, endowing machines with the persona information can lead to the development of psychologically plausible intelligent systems. though computational models of personality have generally followed prior psychological models or theories @xcite , multiple definitions of personas have been in use depending on the nature of the domain or task at hand. there has been considerable amount of interest in the past that used nlp tools to conduct persona analysis of fictional characters in literary texts @xcite . motivated by such works, we focus on deriving persona representations that explain human social behavior categorized according to their influences on language, conversations and actions in different social contexts. in this work, we define persona as the sum total of mental, emotional, and social characteristics of an individual @xcite . this broad definition, while basing on theoretical foundations, allows us to learn persona embeddings from annotated text that span across multiple domains and social contexts. often these persona-annotated domain data are either too small or not representative of all the domain aspects of persona. therefore, we address these challenges by formulating our representation learning problem through the lens of domain adaptation. we propose a model called dapper foot_0 to learn a domain-adapted persona embedding that promotes positive knowledge transfer across multiple text domains: movies dialogue, forum discussion posts and personal life stories or essays. towards this goal, we use a pretrained bert model to extract rich semantic features from text and finetune them by introducing adaptive knowledge transformer that serve as adaptive layers on top of the representations obtained from bert model. these adaptive layers enrich the representations with domain-related persona knowledge. we explore variants of transformer encoder layer as our adaptive layers. in our experiments, we compare our transformer-based dapper model with rnnbased techniques on data from three different text domains. finally, we showcase the advantages of using our representations in a downstream hate speech detection task. thus, our contributions are as follows: ## related work considering that personality compels a tendency on a lot of aspects of human behavior, there have been several studies intended to model personality traits from text. an earlier work by (pennebaker and king, 1999) compiled stream-of-consciousness essay dataset for an automated personality detection task. since the five factor model is widely accepted, several attempts have been made to detect personality from these essays including liwc features or deep learning techniques @xcite . @xcite compared different machine learning models to predict myers-brigg type indicator. another line of work @xcite related to personas focused on developing a language independent and compositional model for personality trait recognition for short tweets. additionally, there have been other efforts that model personas of movie characters and incorporate speaker persona in dialogue models based on speaking style characterized by natural language sentences @xcite . we observe that most of these works use different theories and definitions for modeling personasranging from widely accepted psychological tests to simple emotion states of people as means of ascertaining personality @xcite . however, there is very limited work @xcite focusing on persona embeddings that can be adapted to different domains. in this work, our goal is to produce general purpose persona embeddings computed using texts from various domains . ## datasets towards learning a domain-adapted persona embedding, we aggregate different forms of text data: (a) personal stories/essays, (b) dialogues and (c) discussion forum posts. each of these datasets have distinct persona categories. ## problem setup the overall goal of our model is to learn persona embeddings using documents from different domains d: dialogue utterances, forum posts and personal essays. this persona representation learning problem is formulated as a supervised classification problem. let us denote the ℎ input document as i ( ) = [i ( ) here, a document refers to a list of sentences from the personal essays or forum posts corpus and dialogue snippets in case of movies dialogue corpus (explained in section 3). each input i ( ) in our data is associated with their domain-specific persona label ( ) @xmath0, 2, ..|d |}, ( ) ∈ y and y is the personal categories related to the ℎ -domain. ## proposed model in this work, we explore the idea of leveraging a pretrained bert model towards our goal of learning domain-adaptive persona embeddings. instead of relying only on the domain-specific training data, we allow additional domain knowledge to be injected into our model using an external memory. our model architecture is illustrated in figure 2 . ## experiments in this section, we describe the various evaluations settings: datasets, baselines, our model variants, modes and metrics. our experiments are designed to study the following research questions: rq1: how well does our dapper model perform in comparison to baselines and its variants on domain-specific persona classification task? rq2: is our model capable of adapting to new domains with limited labeled data? rq3: how good are the learned persona embeddings? do they exhibit transfer capability to a downstream task? ## application: hate speech detection with concerns about hate crimes, harassment, and intimidation on the rise, the role of online hate in exacerbating such violence cannot be discounted. hence, there is an growing need to identify and counter the problem of hateful content on social media. while most prior modeling approaches have attempted to capture the semantics of hate from text, a few of them @xcite have used multi-modal information to detect hateful content. few attempts have been made to study the personality of targets and instigators of hate. since our dapper model learns persona embeddings from different forms of text such as dialogues, posts or personal essays, we deem it fit to explore how well our persona embeddings transfer knowledge to a hate speech detection task involving texts from a different domain (in our case, twitter). there are several publicly available labeled hate speech datasets @xcite but very few include author metadata or tweets. in this work, we take advantage of the models and datasets introduced by @xcite (hereafter referred as mm-hate). this weakly-labeled dataset contains author information and additional metadata about potential hate groups. instead of training a powerful hate speech system from the scratch, we augment their base architecture with our persona embeddings and evaluate the prediction performance on the task at hand. we compute persona representations (p) for an author based on their past tweets. we train mm-hate's best performing model, bigru+char+attn (bca), under the following settings: (a) bca + p, which combines our persona embeddings with the extracted text features, (b) bca + sc + p (> ) , which integrates the persona embeddings at the penultimate layer. note that the text and sociocultural (sc) features are already fused at that layer, and (c) bca + sc + p (< ) fuses the extracted text and socio-cultural features with persona embeddings using an attention layer (as in mm-hate). ## conclusion we proposed a dapper model that learns a domain adapted pretraining-based persona representation. our dapper model leverages pretrained bert model and fine-tunes it with additional domainadaptive layers. by introducing a knowledgeattention mechanism, we allow the domain knowledge to be integrated into our persona embeddings. the proposed model achieves significant gains across persona classification task in different domains. our evaluations validate that our model is capable of adapting to a new domain with limited labeled data. we also highlight the transferability of our persona embeddings in a downstream hate speech detection task.