Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers (2023)

Abstract: Demographic factors (e.g., gender or age) shape our language. Previous work showed that incorporating demographic factors can consistently improve performance for various NLP tasks with traditional NLP models. In this work, we investigate whether these previous findings still hold with state-of-the-art pretrained Transformer-based language models (PLMs). We use three common specialization methods proven effective for incorporating external knowledge into pretrained Transformers (e.g., domain-specific or geographic knowledge). We adapt the language representations for the demographic dimensions of gender and age, using continuous language modeling and dynamic multi-task learning for adaptation, where we couple language modeling objectives with the prediction of demographic classes. Our results, when employing a multilingual PLM, show substantial gains in task performance across four languages (English, German, French, and Danish), which is consistent with the results of previous work. However, controlling for confounding factors – primarily domain and language proficiency of Transformer-based PLMs – shows that downstream performance gains from our demographic adaptation do not actually stem from demographic knowledge. Our results indicate that demographic specialization of PLMs, while holding promise for positive societal impact, still represents an unsolved problem for (modern) NLP.

## introduction demographic factors like social class, education, income, age, or gender, categorize people into specific groups or populations. at the same time, demographic factors both shape and are reflected in our language (e.g., @xcite . a large body of work focused on modeling demographic language vari-ation, especially the correlations between words and demographic factors @xcite @xcite . in a similar vein, @xcite and @xcite demonstrated that explicitly incorporating demographic information in language representations improves performance on downstream nlp tasks, e.g., topic classification or sentiment analysis. however, these observations rely on approaches that leverage gender-specific lexica to specialize word embeddings and text encoders (e.g., recurrent networks) that have not been pretrained for (general purpose) language understanding. to date, the benefits of demographic specialization have not been tested with transformer-based @xcite pretrained language models (plms), which have been shown to excel on the vast majority of nlp tasks and even surpass human performance in some cases @xcite . more recent studies focus mainly on monolingual english datasets and introduce demographic features in task-specific fine-tuning @xcite , which limits the benefits of demographic knowledge to tasks at hand. in this work, we investigate the (task-agnostic) demographic specialization of plms, aiming to impart the associations between demographic categories and linguistic phenomena into the plms parameters. if successful, such specialization could benefit any downstream nlp task in which demographic factors (i.e., demographically conditioned language phenomena) matter. for this, we adopt intermediate training paradigms that have been proven effective for the specialization of plms for other types of knowledge, e.g., in domain, language, and geographic adaptation @xcite @xcite . to this effect, we perform (i) continued language modeling on text corpora produced by a demographic group and (ii) dynamic multi-task learning @xcite , wherein we combine language modeling with the prediction of demographic categories. we evaluate the effectiveness of the demographic plm specialization on both intrinsic (demographic category prediction) and extrinsic (sentiment classification and topic detection) evaluation tasks across four languages: english, german, french, and danish, using a multilingual corpus of reviews @xcite annotated with demographic information. in line with earlier findings @xcite , our initial experiments based on a multilingual plm (mbert; @xcite , render demographic specialization effective: we observe gains in most tasks and settings. through a set of controlled experiments, where we (1) adapt with in-domain language modeling alone, without leveraging demographic information, (2) demographically specialize monolingual plms of evaluation languages, (3) carry out a metaregression analysis over dimensions that drive the performance, and ( 4 ) analyze the topology of the representation spaces of demographically specialized plms, we show, however, that most of the original gains can be attributed to confounding effects of language and/or domain specialization. our findings indicate that specialization approaches, proven effective for other types of knowledge, fail to adequately instill demographic knowledge into plms, making demographic specialization of nlp models an open problem in the age of large pretrained transformers. our research code is publicly available at: https://github. com/umanlp/socioadapt . ## demographic adaptation our goal is to inject demographic knowledge through intermediate plm training in a taskagnostic manner. to achieve this goal, we train the plm in a dynamic multi-task learning setup @xcite , in which we couple masked language modeling (mlm-ing) with predicting the demographic category -gender or age group of the text author. such multi-task learning setup is designed to force the plm to learn associations between the language constructs and demographic groups, if these associations are salient in the training corpora. ## experimental setup here we describe evaluation tasks and provide details on the data used for demographic specialization and downstream evaluation. evaluation tasks. we follow @xcite and measure the effects of demographic specialization of plms on three text-classification tasks, coupling intrinsic demographic attribute classification (ac) with two extrinsic text classification tasks: sentiment analysis (sa) and topic detection (td). as an intrinsic evaluation task, ac directly tests if the intermediate demographic specialization results in a plm that can be more effectively fine-tuned to predict the same demographic classes used in the intermediate specialization: plms (vanilla plm and our demographically specialized counterpart) -are fine-tuned in a supervised fashion to predict the demographic class (gender or age) of the text author. sa is a ternary classification task in which the reviews with ratings of 1, 3, and 5 stars represent instances of negative, neutral, and positive class, respectively. td classifies texts into 5 different topic categories. we report the f 1 -measure for each task following @xcite . data. we carry out our core experimentation on the multilingual demographically labeled dataset of reviews @xcite , created from the internationally popular user review website trustpilot. 2 for comparison and consistency, we work with exactly the same data portions as hovy (2015): to avoid any information leakage, we ensurefor each country-demographic dimension collection (e.g., us, gender) -that there is zero overlap between the portions we select for intermediate demographic specialization and portions used for downstream fine-tuning and evaluation (for ac, sa, and td). for td, we aim to eliminate the confounding effect of demographically-conditioned label distributions (e.g., female authors wrote reviews for clothing store more frequently than male authors; vice-versa for electronics & technology). to this effect, we select, for each country, reviews from the five most frequent topics and sample the same number of reviews in each topic for both demographic groups (i.e., male and female for gender; below 35 and above 45 for age). for the intrinsic ac task (i.e., fine-tuning to predict either gender or age category), we report the results for two different review collections: the first is the set of reviews that have, besides the demographic classes, been annotated with sentiment labels (we refer to this as ac-sa) and the second are the reviews that have topic labels (i.e., product/service category; we refer to this portion as ac-td). for these fine-tuning and evaluation datasets, we make sure that the two demographic classes (male and female for gender under 35 and above 45 for age) are equally represented in each dataset portion (train, development, and test). for intermediate specialization of the multilingual model, we randomly sample 100k instances per demographic group from the gender specialization portion and 50k instances each from the texts reserved for age specialization concatenated across all 5 countries. for the specialization of monolingual plms, we randomly sample the same number of instances but from the specialization portions of a single country. following the established procedure (e.g., @xcite , we dynamically mask 15% of the tokens in the demographic specialization portions for mlm. @xcite used in our experiments. for each country (denmark, france, germany, uk, and us), we report the size of the specialization and fine-tuning portions, the latter for each of the two extrinsic tasks: sentiment analysis (sa) and topic detection (td). note that we use the same sa and td reviews for the intrinsic ac tasks of predicting the demographic categories (denoted ac-sa and ac-td, respectively). numbers are shown separately for the two demographic dimensions: gender and age. for fine-tuning datasets (for sa/ac-sa, and for td/ac-td), we indicate the number of instances in each category (which is the same for both categories: f and m for gender, <35 @xmath0. we split the fine-tuning datasets randomly into train, validation, and test portions in the 60/20/20 ratio. pre-trained language models. given that we experiment with trustpilot data in four different languages, in our core experiments, we resorted to multilingual bert (mbert) foot_2 @xcite as the starting plm. this allows us to merge the (fairly large) specialization portions of trustpilot in different languages (see training and optimization. in demographic specialization training, we fix the maximum sequence length to 128 subword tokens. we train for 30 epochs in batches of 32 instances and search for the optimal learning rate among the following values: {5 • 10 -5 , 1 • 10 -5 , 1 • 10 -6 }. we apply early stopping based on the development set performance: we stop if the joint mtl loss does not improve for 3 epochs). for downstream finetuning and evaluation, we train for maximum 20 epochs in batches of 32. we search for the optimal learning rate between the following values: {5•10 -5 , 1•10 -5 , 5•10 -6 , 1•10 -6 } and apply early stopping based on the validation set performance (patience: 5 epochs). we use adamw @xcite as the optimization algorithm. ## results and discussion we first discuss the results of multilingual demographic specialization with mbert as the plm ( §4.1). we then provide a series of control experiments in which we isolate the effects that contribute to performance gains of demographically specialized plms ( §4.2). ## related work intermediate training (adaptation). intermediate language modeling on texts from the same or similar distribution as the downstream data has been shown to lead to improvements on various nlp tasks (e.g., @xcite . during this process, the goal is to inject additional information into the plm and thus specialize the model for a particular domain (e.g., @xcite @xcite or language (e.g., @xcite or to encode other types of knowledge such as common sense knowledge (e.g., @xcite , argumentation knowledge (e.g., @xcite , or geographic knowledge (e.g., @xcite . for instance, @xcite propose a computationally efficient approach by employing domain-specific adapter modules. they show that their domain adaptation approach leads to improvements in task-oriented dialog. @xcite and @xcite perform language adaptation through intermediate mlm in the target languages with filtered text corpora, demonstrating substantial gains in downstream zero-shot crosslingual transfer for abusive language detection and dialog tasks, respectively. these specialization approaches mainly rely on a single objective (e.g., masked language modeling on "plain" text data). instead, @xcite conduct geoadaptation by coupling mlm with a token-level geolocation prediction in a dynamic multi-task learning setup. in this work, we adopt a similar approach and perform continued language modeling on the text corpora of a specific demographic dimension. demographic specialization. language preferences vary with user demographics @xcite . accordingly, several studies have leveraged demographic information (e.g., gender, age, education) to investigate the effect of encoded sociodemographic knowledge in the representations of plms @xcite or obtain better language representations for various nlp tasks @xcite . recent research studies on demographic adaptation mainly focus on (1) learning demographicaware word embeddings and do not work with large plms @xcite or (2) leveraging demographic information with special plm architectures specifically designed for certain downstream tasks (e.g., empathy prediction @xcite ). the latter, however, do not consider a task-agnostic approach to injecting demographic knowledge into language models, and also focus on a monolingual setup only. further, what roles the different factors (i.e., domain, language, demographic aspect) in the specialization really play remains unexplored. ## conclusion in this work, we thoroughly examined the effects of demographic specialization of transformers via straightforward injection methods that have been proven effective for other types of knowledge. initial results on intrinsic and extrinsic evaluation tasks using a multilingual plm indicated the usefulness of our approach. however, running a series of additional experiments in which we controlled for potentially confounding factors (language and domain) and a meta-analysis indicated that the demographic aspects only have a negligible impact on the downstream performance. this observation is supported by additional qualitative analysis. overall, our findings point to the difficulty of injecting demographic knowledge into transformers: we hope that our in-depth analysis and findings catalyze future research on the topic of truly humancentered nlp, especially in multilingual settings.