Language as a fingerprint: Self-supervised learning of user encodings using transformers (2022)

Abstract: The way we talk carries information about who we are. Demographics, personality, clinical conditions, political preferences influence what we speak about and how, suggesting that many individual attributes could be inferred from adequate encodings of linguistic behavior. Conversely, conditioning text representations on author attributes has been shown to improve model performance in many NLP tasks. Previous research on individual differences and language representations has mainly focused on predicting selected attributes from text, or on conditioning text representations on such attributes for author-based contextualization. Here, we present a self-supervised approach to learning language-based user encodings using transformers. Using a large corpus of Reddit submissions, we fine-tune DistilBERT on user-based triplet loss. We show that fine-tuned models can pick up on complex linguistic signatures of users, and that they are able to infer rich information about them. Through a series of intrinsic analyses and probing tasks, we provide evidence that fine-tuning enhances models’ ability to abstract generalizable user information, which yields performance advantages for user-based downstream tasks. We discuss applications in language-based assessment and contextualized and personalized NLP.

## introduction language is not simply a means to communicate about events or inner states. what we talk about, with whom, and in what way says something about who we are -our demographics, our personality, our social and political identity. people with different political views are likely to describe political events in radically different ways; a teenager is more likely to engage in a conversation about video games than in one about retirement savings; social events are more likely to be a frequent topic of conversation for extroverted individuals than for introverts. conversely, while language carries information about who has produced it, knowing who has produced a given utterance can help decode its meaning. an abstract word like "freedom" can mean very different things if uttered by a convict versus a republican us senator, and ironic statements generally require some background knowledge on the speaker to be understood as such. the relationship between language and individual differences has been investigated in multiple fields. previous research has uncovered systematic associations between patterns of language use and demographics @xcite , personality traits @xcite @xcite @xcite , mood disorders @xcite @xcite , conditions such as schizophrenia @xcite @xcite @xcite or asd @xcite @xcite , and political affiliation @xcite . conversely, conditioning text representations on author attributes (e.g., gender or personality) has been shown to enhance performance in several nlp tasks, ranging from sentiment analysis to stance detection @xcite @xcite @xcite . conditioning on author attributes and states can also improve performance in language modeling and generation @xcite @xcite @xcite . however, most research in these domains has focused on developing predictive methods to infer individual attributes from text, or on investigating how conditioning text representations on such attributes can improve performance in downstream nlp tasks. little work @xcite has been devoted to exploring self-supervised approaches to language-based author encoding, where compre-hensive representations of authors' profiles -with applications in both author attribute prediction and contextualization -are inferred from unlabelled text. self-supervised approaches to author encoding would yield two significant advantages over supervised methods. first, these do not rely on the availability of labelled data. secondly, models trained through self-supervised methods may learn to describe users along dimensions of individual variation which are not captured by standard descriptors such as demographics and personality. in this paper, we present a self-supervised approach to fine-tuning transformers as languagebased user foot_0 encoders. building on insights from transfer learning @xcite and contrastive representation learning @xcite @xcite , we fine-tune a pretrained distilbert architecture @xcite on a variant of triplet loss @xcite using reddit submissions from more than 1.7m users. our training objective incentivizes the model to maximize similarity between aggregate representations of posts produced by the same author, and to minimize similarity between representations of posts produced by different authors. this objective tunes models to detecting linguistic signatures of individuals. if, as suggested by previous studies, linguistic styles carry information about individual traits, this will result in models implicitly learning to extract rich information about user characteristics. models trained through this contrastive approach could be deployed for text-based prediction of individuals attributes and related behaviors -with potentially impactful applications in language-based clinical and psychological assessment (see @xcite ) -or for author-based conditioning in the context of contextualized text classification, language modeling and language generation @xcite @xcite @xcite . in this paper, we describe the training methodology and analyze the behavior of trained models through a battery of intrinsic analyses and probing tasks. these analyses are designed to shed light on: a) which linguistic signatures models rely on in the contrastive task; b) which author attributes are encoded in their representations, and how they vary as a function of training parameters (number of input posts); c) whether learned user representations yield performance advantages in downstream tasks relative to standard pretrained models. to encourage experimentation and evaluation on additional tasks and datasets, we make our models available on the hugging face model hub: we also share our code on github: ## evaluation in this section, we evaluate models and probe their heuristics through a series of intrinsic analyses. first, we simply compare the contrastive attribution accuracy of fine-tuned models with that of baseline models and of pretrained distilbert, to evaluate to what extent self-supervised fine-tuning improves models' ability to identify markers of individual styles across the two training scenarios. secondly, we evaluate model performance as a function of the length of anchor posts, to investigate whether models' heuristics rely on sentence-level stylistic markers or on abstract signatures of individuals' profiles. if models rely on detecting specific sentence-level markers (e.g., particular lexical choices or syntactic constructions), performance should decrease as the length of input posts decreases. on the other hand, if models rely on robust, abstract signatures of user characteristics, performance should be significantly less affected by length. thirdly, we investigate the role of semantics in models' heuristics. if models rely overwhelmingly on semantics to encode and identify users, performance should be low for triplets where there is little semantic overlap between the anchor(s) and the positive example. finally, we analyze how attention weights for a large number of vocabulary tokens change between pre-trained and fine-tuned models, in order to gain qualitative insights on whether and how implicit encoding of user characteristics is enhanced by contrastive fine-tuning. ## classification tasks we also evaluate models on a battery of probing tasks designed to test whether fine-tuning yields performance advantages on user-based predictive tasks. probing tasks are designed as follows. for each of the 30 most popular subreddits in the database, we train a classifier to predict whether a given user has posted at least once in that subreddit based uniquely on posts produced by the same user in unrelated subreddits. since no reference post from the target subreddit is provided and choice of input posts is in principle asystematic, performing this task relies on models being able to infer useful and generalizable information on users' profiles from their posts, and to use it to predict unknown preferences and behaviors. if fine-tuning facilitates this process, performance of fine-tuned models should be consistently higher than performance of pretrained models. for each subreddit, we build a training, a validation and a test dataset. to build the dataset for a given subreddit s, we first extract from the database the ids of all users who have posted in s at least once. for each user, we sample up to 10 submissions drawn from other subreddits. we then sample an equal number of users who have never posted in s, and extract up to 10 posts for each of them. we split the resulting dataset into 75/15/15 training/validation/test splits. aggregate encodings 3 of users' posts are used as inputs to simple classifiers (one per model and subreddit), which are optimized to predict whether a given user has posted in s at least once based uniquely on these. all classifiers are trained for 3 epochs. similar to the contrastive learning task, we 10-anchor model 1-anchor model offers, obsessed, crypt, xiao, sponge, leaked, suicidal, keen, pathetic, https, diverse, downloaded, ##lika, psychedelic, tran, purchasing, gambling, tents, banned, desperately, breed, bribe, ##grapher, ugly, wits, ##folk, divorced, tia, ##km, bc, abusive, folks, trance, worthless, wanna, husband, tee, disco, karma, jungle, rory, nyc, toni, probation, buddy, inexpensive, quantity, ##tive, http, prom, brass, bois, mir, fraternity, encouraging, tempered, [sep], cheers nephew, perfection, army, dated, abortion, lads, ##rang, uncle, wireless, nyc, historically, dashboard, comrade, article, ##riation, trained, japanese, profession, daddy, journalist, title, scientists, kidding, thanksgiving, albuquerque, hacker, bard, euro, shane, jai, roman, beautifully, lipstick, linear, ##pile, ito, pee, fragrance, width, tia, neighbors, rig, united, unpopular, bride, lease, ##rse, margarita, buffy, husband, toni, linux, ##ame, pardon, aaa, notebook, [sep], boyfriend, breast, fiance figure 3 displays classification accuracy for all subreddits. results corroborate our predictions. fine-tuned models perform better than all baselines in all classification tasks, both in the single-post and in the multi-post scenario. performance gains relative to pretrained distilbert are larger in the single-post scenario than in the multi-post scenario, arguably reflecting the fact that availability of multiple posts increases chances of topic overlap with the target subreddit, thus increasing the effectiveness of semantics-based heuristics and reducing the need for user-based ones. note that the magnitude of performance gains varies widely across subreddits (see figure 5 in the supplementary materials for details). performance differences are large for subreddits where likelihood to participate is arguably influenced by personality or demographics (e.g., teenagers, jokes, relationship_advice), but we also observe moderate to large gains for subreddits focused on specific video games (e.g., rocketleagueexchange). this may reflect some degree of overfitting to redditspecific discourse. training a truly generalizable language-based author encoder in a self-supervised fashion will require training on data drawn from multiple sources, thus avoiding overrepresentation of platform-specific axes of individual variation in e.g., topics, styles and demographics. ## discussion we introduced a self-supervised approach to training generalized language-based author encoders. we showed that models fine-tuned on user-based triplet loss learn to infer generalizable information on user profiles from complex patterns of linguistic behavior. author encoders may have impactful applications in a variety of domains. two particularly important examples are language-based clinical, psychological and personality assessment @xcite . language-based encodings of individuals could potentially be used to predict personality, psychological traits or even clinical diagnoses and symptoms from spontaneous textespecially for disorders, such as depression, that have previously been associated with consistent patterns of linguistic behavior. for both psychological and clinical applications, complementing traditional methods with naturalistic text-based techniques could not only yield general performance advantages, but also help increase scalability and generalizability @xcite @xcite , and reduce subjective biases @xcite . at the moment, however, no large-scale datasets are publicly available which make it possible to benchmark our models on tasks relevant to these applications. given the large potential for positive societal impact, we believe that the nlp community should promote interdisciplinary efforts aimed at collecting and safely sharing such resources @xcite @xcite . language-based user encodings learned through self-supervised methods could also have a significant impact on contextualized and personalized nlp @xcite . previous work has shown that conditioning representations of text sequences on author traits is beneficial for both downstream tasks and language modeling and generation. contextualization through user embeddings encoding rich information about user characteristics and their linguistic styles could prove a valid alternative, especially for tasks -such as irony or stance detection -that are particularly challenging in absence of complex contextual information @xcite . as a follow-up to this work, we are currently exploring the effect of author-based self-supervised contextualization during pre-training of bidirectional transformers. ## conclusion in this paper, we presented a self-supervised approach to training a language-based user encoder. we showed that models trained on user-based triplet loss can learn compact user representations that encode information about individual traits and yield performance benefits in downstream userbased tasks. future directions include scaling this approach to larger and more diverse data, developing resources for direct evaluation on societally important predictive tasks (e.g., psychiatric assessment), and exploring their potential to empower novel approaches to contextualized natural language modeling, understanding and generation.