Empirical Evaluation of Pre-trained Transformers for Human-Level NLP : The Role of Sample Size and Dimensionality (2021)

Abstract: In human-level NLP tasks, such as predicting mental health, personality, or demographics, the number of observations is often smaller than the standard 768+ hidden state sizes of each layer within modern transformer-based language models, limiting the ability to effectively leverage transformers. Here, we provide a systematic study on the role of dimension reduction methods (principal components analysis, factorization techniques, or multi-layer auto-encoders) as well as the dimensionality of embedding vectors and sample sizes as a function of predictive performance. We first find that fine-tuning large models with a limited amount of data pose a significant difficulty which can be overcome with a pre-trained dimension reduction regime. RoBERTa consistently achieves top performance in human-level tasks, with PCA giving benefit over other reduction methods in better handling users that write longer texts. Finally, we observe that a majority of the tasks achieve results comparable to the best performance with just 1/12 of the embedding dimensions.

## introduction transformer based language models (lms) have quickly become the foundation for accurately approaching many tasks in natural language processing @xcite . owing to their success is their ability to capture both syntactic and semantic information @xcite , modeled over large, deep attention-based networks (transformers) with hidden state sizes on the order of 1000 over 10s of layers @xcite . in total such models typically have from hundreds of millions @xcite to a few billion parameters @xcite . however, the size of such models presents a challenge for tasks involving small numbers of observations, such as for the growing number of tasks focused on human-level nlp. human-level nlp tasks, rooted in computational social science, focus on making predictions about people from their language use patterns. some of the more common tasks include age and gender prediction @xcite , personality @xcite , and mental health prediction @xcite @xcite . such tasks present an interesting challenge for the nlp community to model the people behind the language rather than the language itself, and the social scientific community has begun to see success of such approaches as an alternative or supplement to standard psychological assessment techniques like questionnaires @xcite . generally, such work is helping to embed nlp in a greater social and human context @xcite . despite the simultaneous growth of both (1) the use of transformers and (2) human-level nlp, the effective merging of transformers for humanlevel tasks has received little attention. in a recent human-level shared task on mental health, most participants did not utilize transformers @xcite . a central challenge for their utilization in such scenarios is that the number of training examples (i.e. sample size) is often only hundreds while the parameters for such deep models are in the hundreds of millions. for example, recent human-level nlp shared tasks focused on mental health have had @xmath0, @xmath1, 146 @xcite and @xmath2. such sizes all but rules out the increasingly popular approach of fine-tuning transformers whereby all its millions of parameters are allowed to be updated toward the specific task one is trying to achieve @xcite . recent research not only highlights the difficulty in fine-tuning with few samples @xcite but it also becomes unreliable even with thousands of training examples @xcite . on the other hand, some of the common transformer-based approaches of deriving contextual embeddings from the top layers of a pretrained model @xcite still leaves one with approximately an equal number of embedding dimensions as training size. in fact, in one of the few successful cases of using transformers for a human-level task, further dimensionality reduction was used to avoid overfit @xcite , but an empirical understanding of the application of transformers for human-level tasks -which models are best and the relationship between embedding dimensions, sample size, and accuracy -has yet to be established. in this work, we empirically explore strategies to effectively utilize transformer-based lms for relatively small sample-size human-level tasks. we provide the first systematic comparison of the most widely used transformer models for demographic, personality, and mental health prediction tasks. then, we consider the role of dimension reduction to address the challenge of applying such models on small sample sizes, yielding a suggested minimum number of dimensions necessary given a sample size for each of demographic, personality, and mental health tasks 1 . while it is suspected that transformer lms contain more dimensions than necessary for document-or word-level nlp @xcite , this represents the first study on transformer dimensionality for human-level tasks. ## related work recently, nlp has taken to human-level predictive tasks using increasingly sophisticated techniques. the most common approaches use n-grams and lda @xcite to model a person's language and behaviors @xcite . other approaches utilize word embeddings @xcite and more recently, contextual word representations @xcite . our work is inspired by one of the top performing systems at a recent mental health prediction shared task @xcite that utilized transformer-based contextualized word embeddings fed through a non-negative matrix fac-1 dimension reduction techniques can also be pre-trained leveraging larger sets of unlabeled data torization to reduce dimensionality @xcite . while the approach seems reasonable for addressing the dimensionality challenge in using transformers, many critical questions remain unanswered: (a) which type of transformer model is best? (b) would fine-tuning have worked instead? and (c) does such an approach generalize to other human-level tasks? most of the time, one does not have a luxury of a shared task for their problem at hand to determine a best approach. here, we look across many human-level tasks, some of which with the luxury of having relatively large sample sizes (in the thousands) from which to establish upper-bounds, and ultimately to draw generalizable information on how to approach a human-level task given its domain (demographic, personality, mental health) and sample size. our work also falls in line with a rising trend in ai and nlp to quantify the number of dimensions necessary. while this has not been considered for human-level tasks, it has been explored in other domains. the post processing algorithm @xcite of the static word embeddings motivated by the power law distribution of maximum explained variance and the domination of mean vector turned out to be very effective in making these embeddings more discriminative. the analysis of contextual embedding models (ethayarajh, 2019) suggest that the static embeddings contribute to less than 5% to the explained variance, the contribution of the mean vector starts dominating when contextual embedding models are used for human-level tasks. this is an effect of averaging the message embeddings to form user representations in human-level tasks. this further motivates the need to process these contextual embeddings into more discriminative features. lastly, our work weighs into the discussion on just which type of model is best in order to produce effective contextual embedding models. a majority of the models fall under two broad categories based on how they are pre-trained -auto-encoders (ae) and auto-regressive (ar) models. we compare the performance of ae and ar style lms by comparing the performance of two widely used models from each category with comparable number of parameters. from the experiments involving bert, roberta @xcite , xlnet @xcite and gpt-2 @xcite , we find that ae based models perform better than ar style models (with comparable model sizes), and roberta is the best choice amongst these four widely used models. ## data & tasks we evaluate approaches over 7 human-level tasks spanning demographics, mental health, and personality prediction. the 3 datasets used for these tasks are described below. fb-demogs. (age, gen, ope, ext) one of our goals was to leverage one of the largest humanlevel datasets in order to evaluate over subsamples of sizes. for this, we used the facebook demographic and personality dataset of @xcite . the data was collected from approximately 71k consenting participants who shared facebook posts along with demographic and personality scores from jan-2009 through oct-2011. the users in this sample had written at least a 1000 words and had selected english as their primary language. age (age) was self-reported and limited to those 65 years or younger (data beyond this age becomes very sparse) as in @xcite . gender (gen) was only provided as a limited single binary, male-female classification. personality was derived from the big 5 personality traits questionnaires, including both extraversion (ext -one's tendency to be energized by social interaction) and openess (ope, one's tendency to be open to new ideas) @xcite . disattenuated pearson correlation foot_0 (r dis ) was used to measure the performance of these two personality prediction tasks. ## methods here we discuss how we utilized representations from transformers, our approaches to dimensionality reduction, and our technique for robust evaluation using bootstrapped sampling. ## error analysis here, we seek to better understand why using pretrained models worked better than fine-tuning, and differences between using pca and nmf components in the low sample setting (n @xmath3. ## discussion ethical consideration. we used existing datasets that were either collected with participant consent (fb and clpsych 2018) or public data with identifiers removed and collected in a non-intrusive manner (clpsych 2019). all procedures were reviewed and approved by both our institutional review board as well as the irb of the creators of the data set. our work can be seen as part of the growing body of interdisciplinary research intended to understanding human attributes associated with language, aiming towards applications that can improve human life, such as producing better mental health assessments that could ultimately save lives. however, at this stage, our models are not intended to be used in practice for mental health care nor labeling of individuals publicly with mental health, personality, or demographic scores. even when the point comes where such models are ready for testing in clinical settings, this should only be done with oversight from professionals in mental health care to establish the failure modes and their rates (e.g. false-positives leading to incorrect treatment or false-negatives leading to missed care; increased inaccuracies due to evolving language; disparities in failure modes by demographics). malicious use possibilities for which this work is not intended include targeting advertising to individuals using language-based psychology scores, which could present harmful content to those suffering from mental health conditions. we intend that the results of our empirical study are used to inform fellow researchers in computational linguistics and psychology on how to better utilize contextual embeddings towards the goal of improving psychological and mental health assessments. mental health conditions, such as depression, are widespread and many suffering from such conditions are under-served with only 13 -49% receiving minimally adequate treatment @xcite . marginalized populations, such as those with low income or minorities, are especially under-served @xcite . such populations are well represented in social media @xcite and with this technology developed largely over social media and predominantly using self-reported labels from users (i.e., rather than annotator-perceived labels that sometimes introduce bias @xcite ), we do not expect that marginalized populations are more likely to hit failure modes. still, tests for error disparities @xcite should be carried out in conjunction with clinical researchers before this technology is deployed. we believe this technology offers the potential to broaden the coverage of mental health care to such populations where resources are currently limited. future assessments built on the learnings of this work, and in conjunction with clinical mental health researchers, could help the under-served by both better classifying one's condition as well as identifying an ideal treatment. any applications to human subjects should consider the ethical implications, undergo human subjects review, and the predictions made by the model should not be shared with the individuals without consulting the experts. limitations. each dataset brings its own unique selection biases across groups of people, which is one reason we tested across many datasets covering a variety of human demographics. most notably, the fb dataset is skewed young and is geographically focused on residents within the united states. the clpsych 2018 dataset is a representative sample of citizens of the united kingdom, all born on the same week, and the clpsych-2019 dataset was further limited primarily to those posting in a suicide-related forum @xcite . further, tokenization techniques can also impact language model performance @xcite . to avoid oversimplification of complex human attributes, in line with psychological research @xcite , all outcomes were kept in their most dimensional form -e.g. personality scores were kept as real values rather than divided into bins and the clpsych-2019 risk levels were kept at 4 levels to yield gradation in assessments as justified by @xcite ## conclusion we provide the first empirical evaluation of the effectiveness of contextual embeddings as a function of dimensionality and sample size for human-level prediction tasks. multiple human-level tasks along with many of the most popular language model techniques, were systematically evaluated in conjunction with dimension reduction techniques to derive optimal setups for low sample regimes characteristic of many human-level tasks. we first show the fine-tuning transformer lms in low-data scenarios yields worse performance than pre-trained models. we then show that reducing dimensions of contextual embeddings can improve performance and while past work used non-negative matrix factorization @xcite , we note that pca gives the most reliable improvement. auto-encoder based transformer language models gave better performance, on average, than their auto-regressive contemporaries of comparable sizes. we find optimized versions of bert, specifically roberta, to yield the best results. finally, we find that many human-level tasks can be achieved with a fraction, often 1 6 th or 1 12 th , the total transformer hidden-state size without sacrificing significant accuracy. generally, using fewer dimensions also reduces variance in model performance, in line with traditional bias-variance tradeoffs and, thus, increases the chance of generalizing to new populations. further it can aid in explainability especially when considering that these dimension reduction models can be pre-trained and standardized, and thus compared across problem sets and studies.