Human Language Modeling (2022)

Abstract: Natural language is generated by people, yet traditional language modeling views words or documents as if generated independently. Here, we propose human language modeling (HuLM), a hierarchical extension to the language modeling problem where by a human- level exists to connect sequences of documents (e.g. social media messages) and capture the notion that human language is moderated by changing human states. We introduce, HaRT, a large-scale transformer model for solving HuLM, pre-trained on approximately 100,000 social media users, and demonstrate it’s effectiveness in terms of both language modeling (perplexity) for social media and fine-tuning for 4 downstream tasks spanning document- and user-levels. Results on all tasks meet or surpass the current state-of-the-art.

## introduction language use, like any human behavior, is moderated by underlying human states of being @xcite . indeed, different ways of incorporating human information into nlp models have recently been shown to improve accuracy on many nlp tasks @xcite @xcite . at the same time, while language modeling has proven itself fundamental to nlp, it is typically absent the notion of a human producing the natural language. from a statistical modeling perspective, this absence of human state can be seen as an instance of the ecological fallacy -the treatment of multiple observations (i.e. text sequences) from the same source (i.e. human) as independent @xcite . to address this, we introduce the task of human language modeling (hulm), which induces dependence among text sequences via the notion of a human state in which the text was generated. in particular, we formulate hulm as the task of estimating the probability of a sequence of tokens, w t,1∶i , while conditioning on a higher order state (u 1∶t-1 ) derived from the tokens of other documents written by the same individual. its key objective is: to build a language model that effectively addresses the hulm task, we develop hart, a human-aware recurrent transformer. hart is built using a new user-state based attention layer, that connects standard word sequence transformer layers in order to incorporate the human context. the recurrent user state allows hart to effectively model long contexts necessary to handle all the previous messages written by an individual. we train hart on the hulm task defined over a large collection of social media texts spanning 100k users and apply it (fine-tuning) on 2 downstream message-level tasks: stance detection @xcite , and sentiment analysis @xcite as well as 2 human-level tasks: age estimation and personality assessment @xcite . contributions. our contributions are threefold: (1) we introduce the task of human language modeling (hulm), providing a mathematical definition and relation to traditional language modeling; (2) we propose hart, a novel transformer-based model for performing hulm and capable of being fine-tuned to specific tasks including user-level tasks for which traditoinal language models cannot be applied without architectural alterations; (3) we evaluate hart, demonstrating state-of-the art performance on five tasks: social media language modeling (perplexity), two document-level tasks (sentiment analysis and stance detection), and two user-level tasks (personality-openness assessment, and age estimation). ## related work recent advances in language model pre-training have led to learned representation of text. pretraining methods have been designed with different training objectives, including masked language modeling @xcite and permutationbased auto-regressive language modeling @xcite . these have contributed in building deep autoencoding architectures, allowing the same pre-trained model to successfully tackle a broad set of nlp tasks. while pre-training over large collections of text helps models acquire many forms of linguistic and world knowledge @xcite @xcite , they are still devoid of the information about the text creator. recently, it has been suggested that the nlp community address the social and human factors to get closer to the goal of human-like language understanding @xcite . this call builds on a series of studies suggesting that integrating the human context into natural language processing approaches leads to greater accuracy across many applications in providing personalized information access @xcite and recommendations @xcite @xcite . the idea of contextualizing language with extra linguistic information has been the basis for multiple models: hovy (2015) learn ageand gender-specific word embeddings, leading to significant improvements for three text classification tasks. @xcite proposed a domain adaptaion-inspired method for composing userlevel, extra-linguistic information with message level features, leading to improvements for multiple text classification tasks. @xcite propose a new form of personalized word embeddings that use demographic-specific word representations. in addition to addressing to social and human factors, recent work has also focused on personalized language models @xcite learning author representations @xcite and personalized word embeddings @xcite pointing out the importance of personalized semantics in understanding language. @xcite explore personalized versus generic word representations showing the benefits of both combined. while these models are trained for singular user, @xcite trains a single shared model for all users for personalized sentiment analysis. however, the approach is not scalable as it is still user specific and expects a unique user identifier. while not the primary goal, human language modeling may yield effective approaches to extend the context during language modeling. thus, an aspect of this work can be seen as part of the recent pursuit of sequence models that cap-ture dependencies beyond a fixed context length @xcite . for example, @xcite and @xcite propose controllable language generation using one or more attribute classifiers or control codes. @xcite propose augmented language model pretraining with a latent knowledge retriever which allows the model to retrieve and attend over documents from a large corpus. these models extend context limits, but they do not model the higher order structure capturing a notion of the common source of documents i.e., the author. on the other hand, yoshida et al. ( 2020 ) fits a hierarchical model extension to language modeling by adding recurrence to a pretrained language model. this idea forms a basis for our proposed hulm architecture, hart, but yoshida et al. do not exploit the inherent higher order structure (i.e. the model was not used for hulm). ## human language modeling (hulm) our goal is to re-formulate the language modeling task into one that directly enables a higher-order dependence structure that represents a human generating the language. language modeling formulations pose probabilistic questions over text represented as sequences of tokens. the main goal is to model the probability of observing a given token sequence in the language as a whole. in particular language models (lms) estimate the joint probability of the tokens in the string, defined in terms of the probabilities of each token in the sequence conditioned on the previous tokens. we pose the human language modeling problem (hulm), where the goal is to model the probabilities of observing a sequence from the language as generated by a specific person. an initial idea might be to pose this task as conditioning the probability of a string, w i on a static representation of 3 traditional lms provide estimates of the conditional probabilities often relying on further simplifying assumptions (e.g. markovian assumptions to handle long sequences.). in one extreme, u 1∶t-1 could model all previous tokens in all previous documents by one person. in the opposite extreme, u 1∶t-1 can be the same for all values of t, giving a static representation for a user (equivalent to equation 2) or even static across users which reduces to a standard language modeling version (equivalent to equation 1). still, modeling a user via their previous documents provides a seamless way to integrate the user information into language models -the only change is that the models will now have to incorporate more text when they are making predictions. note that this problem formulation does not directly require explicit modeling of the identity of a user. this makes it easier to handle new users in downstream tasks and test instances, or creating models that can be further fine-tuned to both document-and user-level tasks. ## human-aware recurrent transformer this section introduces, hart, a human-aware recurrent transformer that trains on the human language modeling (hulm) formulation. hart is designed to produce human-aware contextual representations of text at multiple levels. hart's design is motivated by two goals: (i) we want to support hierarchical modeling, i.e., to hierarchically represent the set of all-messages written by a user and at the same time have human-aware contextual word representations. this implicitly entails modeling large context size. for example, gpt-2 @xcite uses a context size of 1024 tokens, whereas our estimate of the average context size for a twitter user is more than 12000 tokens. (ii) to support user-level tasks (e.g. personality assessment @xcite )), we need representations of the entire set of messages written by a user capturing the inherent human states that broadly encompasses the user representation. hart addresses the hierarchical language modeling issue by processing all messages written by a user in a temporally ordered sequence of blocks. it uses a recurrence structure to summarize information in each block into a user state vector, which is then used to inform the attention between tokens in the subsequent block. for human-level tasks the aggregate of user states can be used as the representation of the entire context for the user. the idea of adding recurrence to pre-trained transformers builds on yoshida et al. (2020)'s method for handling long contexts. however, the main difference is that hart models the input data (language) in the context of its source (user) along with inter-document context, thus enabling a higher order structure representing human context. ## evaluation: human language modeling we seek to compare hart with a standard language model that is exposed to the same data but without modeling the notion of a user. thus, we compare hart's human language modeling performance to the model it was based, gpt-2. for calibration we report performance on gpt-2's original pre-trained version (gpt-2frozen), and a version of the lm that was fine-tuned on the hulm-corpus (gpt-2hlc). we train and evaluate the models using the train and test splits of the hulm-corpus described in section 4.2. for hyperparameter search, we use the full development set of both seen and unseen users. each training instance for hart is capped to 8-blocks of 1024-tokens each. following previous work fine-tuning transformer language models for social media (v ganesan et al., 2021), gpt-2 was trained over individual messages. we train both for five epochs and set the learning rate, batch size, and stopping patience based on the development set (see appendix a.3). for hart, we initialize all gpt-2 self-attention layers with the corresponding weights in the pre-trained gpt-2. the user-state based self-attention layer weights (query, key, and value) are normal initialized with 0 mean and 0.02 standard deviation. perplexity here, we evaluate the utility of fine-tuning hart for document-and user-level tasks. just as standard transformer language models are fine-tuned for tasks, we take our pre-trained hart model and fine-tune it for stance detection, sentiment classification, age estimation, and personality (openness) assessment tasks. for both sets of tasks we compare fine-tuning the gpt-2hlc as a nonuser-based lm baseline and also report previously published results from other task specific models, most of which employ historical context for re- ## conclusions language is deeply human. yet, language models in wide-spread use today lack a notion of the human that generates the language. motivated by other advances in human-centered language processing and psychological theory that suggest language is moderated by human states, we introduced human language modeling. hulm extends lms with the notion of a user and their states via their previous messages. in this first step toward large human language models, we developed a human-aware transformer (hart) that uses a recurrence mechanism to model user states and show that pre-training this transformer on the human language modeling task yields significant gains in both generation and fine-tuning for multiple downstream document-and user-level tasks. overall, state-of-the-art results with hart, a model neither trained on substantially larger data nor adding many parameters, suggests progress for transformers not based on massive increases in data or parameters but on a task grounded in language's "natural" generators, people.