Classist Tools: Social Class Correlates with Performance in NLP (2024)

Abstract: The field of sociolinguistics has studied factors affecting language use for the last century. Labov (1964) and Bernstein (1960) showed that socioeconomic class strongly influences our accents, syntax and lexicon. However, despite growing concerns surrounding fairness and bias in Natural Language Processing (NLP), there is a dearth of studies delving into the effects it may have on NLP systems. We show empirically that NLP systems’ performance is affected by speakers’ SES, potentially disadvantaging less-privileged socioeconomic groups. We annotate a corpus of 95K utterances from movies with social class, ethnicity and geographical language variety and measure the performance of NLP systems on three tasks: language modelling, automatic speech recognition, and grammar error correction. We find significant performance disparities that can be attributed to socioeconomic status as well as ethnicity and geographical differences. With NLP technologies becoming ever more ubiquitous and quotidian, they must accommodate all language varieties to avoid disadvantaging already marginalised groups. We argue for the inclusion of socioeconomic class in future language technologies.

## introduction the relationship between social class and language was first examined by @xcite . he discovered that new yorkers with greater socioeconomic status pronounce the /r/ sound after vowels, whereas individuals with lower socioeconomic position drop it. he quantified this observation by asking employees in various department shops (a proxy for socioeconomic rank) for things found on 1 we make our artifacts available on our github repository. the fourth floor. then he kept track of how many rs were dropped in those two words. he discovered a clear anti-correlation between the store's (and presumably the speakers') socioeconomic standing and the quantity of dropped rs: the higher the status, the fewer dropped rs. since @xcite analysis of social stratification and language, linguistics has made concerted efforts to understand how different sociodemographic factors influence language production and perception, and how speakers use them to create identity @xcite @xcite ; using both naturally produced language and language from media like tv shows and films @xcite to study sociolects. despite a considerable body of evidence demonstrating links between language and demographic characteristics (the "first wave" of sociolinguistic variation studies), comparatively few socio-demographic factors have been studied in the context of natural language processing (nlp) technology @xcite . but nlp is a product and a tool for understanding other phe-nomena and must represent all language varieties. existing research on socio-demographic characteristics has primarily concentrated on the signaling effect of certain linguistic variables like age, ethnicity, regional origin, and gender @xcite . most of these publications focus on any bias toward the particular variable they study. however, nlp hardly interacts with the second and third waves of sociolinguistics, i.e., how variation 1) shapes local identity and 2) drives language development. to narrow this gap, we use nlp to focus on socioeconomic position. using a dataset of 95k utterances from englishlanguage television episodes and movies, we empirically investigate performance as a function of socioeconomic class. we study its effect on vocabulary, automated speech recognition (asr), language modelling, and grammatical error correction. in all applications, we see strong correlations. movie characters are usually typecast to present as a certain class, providing a representative sample without the privacy issues associated with regular subjects. the authenticity of these representations is backed by work in sociolinguistics such as @xcite and @xcite . we discover that the performance of nlp tools is associated with socioeconomic class, geographic variety (us vs. uk), and race across all tasks. our findings highlight an important lack of flexibility of nlp tools, and a more fundamental issue: what does it mean if nlp technologies only reliably work for a limited segment of society? ## social class and language social stratification is the grouping of people based on their socioeconomic status (ses), which is determined by factors such as income, education, wealth, and other characteristics. different groups are distinguished in terms of power and prestige. there are various social stratification systems, such as the indian caste system, indigenous american clans or tribes, and the western hierarchical class system. the exact number of social strata is unknown and varies by country and culture. however, at least three strata are commonly used to refer to different groups in western societies: upper, middle, and lower-class people. other systems distinguish between blue collar and white collar jobs. recently, the great british class survey (gbcs, @xcite has taken an empirical approach to understanding the different social strata, and they propose a seven-level system for the united kingdom. their stratification is based on economic, social, and cultural capital: elite, established middle-class, technical middle-class, new affluent workers, traditional working class, emergent service workers, and precariat. they derive these classes from a survey conducted over british citizens that received more than 160k responses. social class influences people's daily lives by granting or limiting access to resources. beyond power and prestige, social stratification has a significant impact on people. for example, lower socioeconomic status has been linked to poorer health outcomes and higher mortality. @xcite . ses also affects language use from the very early stages of development. @xcite theorised that language takes on a different role in families of different socioeconomic status, where middleclass parents encourage language learning to describe more abstract thinking leading to broader vocabularies. in working-class families, parents are limited to more concrete and descriptive concepts. parents from lower ses tend to interact less with their children, with fewer open-ended questions than parents from higher ses, which shapes language development @xcite . @xcite show that education reduces this language gap, education has traditionally been one of the most important factors in determining social class and potential for upward social mobility. given the well-documented effects of socioeconomic status on language development and use, it stands to reason that social class should be carefully considered as a variable in nlp. ## dataset here, we want to assess the impact a user's social class has on the performance of nlp systems. although there have been concerted efforts to document wide varieties of english, existing datasets do not currently capture all social strata or other sociodemographic aspects sufficiently. for example, the national speech corpus @xcite provides education information for the talker and their parents but consists of almost exclusively white and educated speakers; foot_0 the coraal corpus @xcite on the other hand is focused on african american english (aae), not capturing other ethnicities. to the best of our knowledge, no datasets are available that would suit our purposes: a large enough sample that is annotated with the speakers' socieconomic background. we thus collect our own. we identify an initial sample of english language television series by selecting the 250 highestrated series on the international movie database (imdb). 3 we annotate each show according to attributes, foot_2 i.e., class, race, gender, dialect, and geography of the main characters, as well as the genre and time period of the show. three annotators familiar with the shows agree on the labels unanimously. we only consider shows that are scripted to ensure that the language production of each character is controlled by the script-writers and actors to produce a particular vernacular and to avoid collecting highly sensitive information about private individuals. we therefore exclude cartoons, documentaries. we further exclude science fiction and historical fiction, as these are likely to operate without strict adherence to existing social structures and language variation. after our filtering and selection, 63 shows remain in consideration. of those, 34 are predominantly about white men, and 29 feature a mixture of white women and men. only three predominantly feature female characters, and none black or lower-class female characters. based on this initial sample, we identified additional shows to augment gaps (e.g., the lack of black stories) in class and dialect. because we want to measure the effect of class separately from gender, race and regional accent, we select the tv shows from the list representing each category. in our final selection of movies and television shows, we include a total of 19 shows that are based in the united states of america and the united kingdom. to ensure racial diversity, we emphasise movies and shows that predominantly represent black characters in addition to movies and shows that represent white characters. 5 we include the u.s.-based shows the wire and when they see us (predominantly lower-class black characters), fargo (middle-class black characters), the fresh prince of bel-air (upper-class black characters). for the shows' centre white characters, we include trailer park boys (lower-class white), the sopranos (lower-class white), breaking bad (middle-class white), and arrested development (upper-class white). the list of shows above, however, predominantly feature male characters. for this reason, we also include shows whose characters represent a wider array of genders. the shows that we include are pose (lower-class black and latinx), big little lies (middle-class white), and sex and the city (upper middle-class white). finally, to compare with other regional varieties of english, we include movies and shows from different regions in the united kingdom. the movies and shows that we include are train spotting and t2 (lower-class white scottish, predominantly male characters), derry girls (lower-class white irish, primarily female), downton abbey (english), the it crowd (white, middle-class) and the crown (upper-class white english). we report full details in we collect the first season for each of the shows on our list, except for fargo and the wire. for fargo, we collect the fourth season, which includes black characters. for the wire, we collect seasons 1 and 3, which feature black characters but have a minority representation of white characters. we deliberately chose this approach to data selection from "artificial sources" since collecting class information is highly sensitive and can introduce privacy risks. fictional characters, however, are often scripted into particular roles with explicit ses. a drawback of this methodology is the question of authenticity -the extent to which actors are representative of particular sociolects. actors reportedly undergo significant training to learn different lects. 6 by selecting highly rated shows, we hope to capture good, realistic performances. moreover, tv show and film data is increasingly used in sociolinguistics @xcite and @xcite found important similarities between real and tv show dialogue. in nlp, media data is broadly used in tasks like emotion analysis (plaza-del arco et al., 2024) and dialogue @xcite . we therefore have reason to believe our dataset provides a reasonable proxy for real speakers. in total, our dataset contains 95k utterances, text and speech, from 19 tv shows and movies (see ## social class and speech beyond written language, our dataset affords testing whether speech models understand and process accent unevenly. therefore, we test disparities in asr word error rates across social classes. we calculate the wer for all shows and movies to measure how well asr technologies capture different variants of english in our sample. models: we use wav2vec2 @xcite , xls-r @xcite , and whispermedium @xcite as implemented in 9 we use scikit-learn's tfidfvectorizer collecting uniand bi-grams for tf-idf, and https://huggingface.co/ sentence-transformers/all-mpnet-base-v2 . processing pipeline: we use movies and tv shows which are available online. we first collect the audio and subtitles from opensubtitles foot_6 and use them as gold reference. 11 then, we used one asr model to transcribe the audio. the timestamps from the model-generated transcription and the subtitles often do not match, and neither source provides information about which character is speaking. we therefore group all utterances from a single episode or movie together into one long string and calculate the wer using jiwer @xcite . ideally, we would like to separate by character to get a more fine-grained analysis. however, this approach would require us to keep mappings from utterance to character consistent across episodes. despite several approaches, we were not able to achieve this goal with the computational power we had. our analysis is therefore group-based, rather than character-based. results: figure 2 shows the differences in worderror-rate by race and class in us tv shows. we find clear trends across models, with effects from both race and class: lower asr error rates are associated with higher ses and with whiteness. these trends are stronger for the wav2vec2 model. ## language modelling language modelling plays an essential role in downstream applications in nlp, so it is imperative that language models can accurately and equally represent different speakers' lects. language models are evaluated through perplexity, a measure of how expected a given sentence is: the higher the perplexity, the more unexpected. perplexity can give us an indication of how well a model might perform in downstream tasks: gonen et al. @xcite show that the lower the perplexity of a prompt, the better the prompt can perform the task. here, we calculate perplexity as a measure of linguistic acceptability, that is, how 'expectable' a particular language variant might be to a given language model. should models display differing perplexities for different groups, they would put such groups at a disadvantage. ## grammar correction finally, we consider the grammaticality of each sentence. we suspect that grammar features correlate strongly with class, as "correct" grammar is typically a hallmark of signalling higher ses. we calculate the edit distance between the original sentence and the corrected one. lower edit distance means higher grammaticality. models: since we are concerned with potential end-user experience, we choose to evaluate the four most downloaded models on huggingface since they will reach the largest audience. we use the following models for grammar correction: instead, we consider whether models are more likely to produce edits for different classes. in figure 5 we plot the count of utterances with corrections per model and per class. from this, we can see a clear pattern: models produce corrections more frequently for those of lower ses. in addition, we notice that often some of these corrections are performed on in-group slang or regional linguistic phenomena, see ## discussion following established work in sociolinguistics, our results support our hypothesis that nlp systems display biases towards different sociolects, beyond geographical and ethnic differences. on the difference between life and art: the use of media (if not synthetic) data as a stand-in for real-world examples is not uncommon in nlp (e.g. plaza-del @xcite . our work presents a starting point to study an aspect of linguistic variation that has thus far received little attention in the nlp literature. to substantiate our call to action, the characters' language need not be an exact replica but a good approximation of a given sociolect. however, we do not claim that all media portrayals are accurate representations: tv shows and films are written by certain people, for specific audiences, and the quality of acting varies. through our selection process, we selected shows that would not caricature certain groups, including tv shows like the wire and pose, which feature a cast representative of the characters. nevertheless, these results should be validated with real speakers. nlp as an enforcer of "standard" language: native speakers of any language have different registers available to them, including when interacting with nlp systems. while this may help mitigate some of the potential harms shown in this paper, our position is that humans should not have to adapt their language to interact with technology, but rather technology should adapt to humans. by dint of their use, nlp systems are setting a standard of language: nlp systems are becoming more common, not only in social contexts calling for formal (i.e. standardised) language, or in fields like social science @xcite , but in everyday sce going forward while reporting on the socioeconomic background of study participants and dataset contributors is more complicated than other factors such as age or ethnicity (as it is not a one-point measurement and it is culture-dependent), we encourage researchers to report on this to draw a more accurate picture of the language varieties nlp will capture and to better benchmark systems across sociolects. given the limitations of our study (most notably, restriction to english and group rather than individual scores, see the limitations section below), we envision several possible avenues for future research in this area: (1) the findings of this study should be validated for other languages where such differences have been observed; (2) the collection of a (possibly larger) corpus of unaggregated speakers; and (3) while we find support from previous research such as @xcite for social media data, the findings should also be validated for other data forms such as long-form written texts. ## related work works in nlp considering social class in the context of language use are few and far between (cercas @xcite , mostly focusing on predicting social class from texts (e.g. @xcite . @xcite ascribe the dearth of work in bias and social class to the lack of self-reported explicit labels in online user profiles. the difficulty in obtaining such data hinders efforts in addressing this issue. more recently, some work has focused on representational harms with regards to social class: @xcite investigate bias against the poor in nlp in terms of the language that surrounds poor people, considering the associations of 'poor' and 'rich' in embeddings and language models; @xcite instead focus on aporophobia, classist attitude stigmatising the poor, in the context of toxic language. to the best of our knowledge, we are the first to investigate how nlp systems perform with speakers from different social strata. ## conclusion social class plays an important role in people's identity construction, and is consequently strongly reflected in their language use. despite its central status as a variable in sociolinguistics, there is so far little work in nlp engaging with social class and its impact. our contributions are twofold: following methods from sociolinguistics, we use a data set of 95k movie transcripts that we annotate for portrayed class, race, and geographical variety (uk vs. us). we use this data to show that linguistic class markers greatly affect the performance of various nlp tools. our findings suggest that speakers of low-prestige sociolects experience lower application performance on a range of tasks. these results suggest that we should actively incorporate social class as a variable in nlp systems if we want to make them more equitable and fair. our data set also provides a starting point for more explorations of social class in nlp.