The Echoes of the ‘ I ’: Tracing Identity with Demographically Enhanced Word Embeddings (2024)

Abstract: Identity is one of the most commonly studied constructs in social science. However, despite extensive theoretical work on identity, there remains a need for additional empirical data to validate and refine existing theories. This paper introduces a novel approach to studying identity by enhancing word embeddings with socio-demographic information. As a proof of concept, we demonstrate that our approach successfully reproduces and extends established findings regarding gendered self-views. Our methodology can be applied in a wide variety of settings, allowing researchers to tap into a vast pool of naturally occurring data, such as social media posts. Unlike similar methods already introduced in computer science, our approach allows for the study of differences between social groups. This could be particularly appealing to social scientists and may encourage the faster adoption of computational methods in the field.

## introduction identity is central and one of the most commonly studied constructs in the social sciences, shaping our understanding human behaviour, and society more generally @xcite . while there is no universally accepted definition of identity, it generally refers to individual's selfperception that consists of self-ascribed personal traits, beliefs about themselves, as well as selfcategorization into particular social groups and roles. research on identity spans disciplines from psychology to sociology, and from linguistics to political science offering rich theoretical insights into identity @xcite . however, measuring identity and related constructs remains challenging, which is why there is still a clear need for empirical studies that would allow to validate and refine existing theories @xcite . established methods typically require the annotation of survey data by experts who have to be specially trained. take, for instance, loevinger theory of ego development @xcite which is generally considered as one of the most empirically supported theories of personality development @xcite . traditionally, ego development is measured via the washington university sentence completion test (wusct) @xcite . that is a projective technique where participants are asked to complete sentence stems such as "what gets me in trouble..." or "a girl has a right to...". while wusct has been shown to be a reliable and valid method of measuring ego development @xcite , its administration is resource-intensive and requires a specialized training for raters. at the same time, recent developments in computational methods suggest that social media data, at least at a macro level, can aid in assessing psychological constructs @xcite . this could pave the way for alternatives to traditional survey-based assessments. computational approaches and natural language processing have been previously applied to study identity. in particular liwc (tausczik and pennebaker, 2010) -a popular dictionary-based method -has been used to analyze responses to wusct @xcite or to identify salient identity in social media posts @xcite . in our work, we propose using word embeddings as they allow capturing more complex semantic relationships in the text by considering the context in which words are used. the common approach to using word embeddings in social science is to consider projections on semantic axes in word-vector space. it has been previously demonstrated that this technique could effectively recover human sentiments, judgments, and perceptions @xcite . this enabled computational social scientists to extract insights from large text corpora. the potential of this approach was most notably demonstrated in studies on stereotypes @xcite @xcite . typically, a word embedding model is trained on a specific corpus of interest. then, the distance between target words and predefined reference poles, represented by opposing words or sets of words, is considered. this distance is interpreted as the semantic closeness between target words and reference poles, providing insights into underlying associations and relationships. more concretely, it has been shown that certain occupational terms, e.g. 'mechanic', are closer to words representing men ('man', 'boy', 'he', etc.), while other terms, e.g. 'nurse', are closer to words representing women ('woman', 'girl', 'she', etc.), indicating a gender bias @xcite . by training separate word embedding models on time-segmented historical texts, it has been further demonstrated that the changes in word distances over time reflect realworld changes in women's occupations. another study has found that words representing men are closer to words related to intelligence, while words representing women are closer to 'studying', reflecting a common stereotype in education: "boys are successful at school because they are smart and girls because they study a lot" @xcite . training separate word embedding models on texts produced at different time points further showed that this stereotype emerged at specific point in time, consistent with sociological explanations of the phenomenon. simply computing word similarities in a given corpus is often not very informative. therefore, researchers typically segment the corpus for comparative analysis. these segments might represent different time periods, as in the examples above, or the corpus could be split by other criteria, such as training distinct models on texts authored by republicans versus democrats @xcite . this approach, however, has a disadvantage as it reduces the amount of data available for training individual models, which could impair their performance. it also requires the alignment of resulting models in a common space, which could complicate the interpretation of the results @xcite . in our work, we build upon these ideas by enhancing word embeddings with socio-demographic information and focusing on studying the self. more specifically, we replace every occurrence of the word 'i' in a large corpus of social media posts with i g,a tokens, where g represents the gender of the post author and a their age. we then train a word embedding model on the altered corpus. projecting the resulting enhanced vectors on semantic axes allows exploring identity as expressed in social media posts. by incorporating socio-demographic information into the i-tokens, it also becomes possible to compare different social groups without splitting the original corpus. in the remainder of this paper, we provide a more detailed description of our method. we then characterise the obtained enhanced i-tokens and verify whether they meet the criteria for face validity. to further validate our approach, we check if it can reproduce established findings on gendered selfviews. next, we investigate the robustness of the results with respect to model specifications and corpus size. finally, we discuss how our approach can be applied in different contexts and compare it with existing methods. ## results we found that the variation between i g,a tokens is largely explained by gender and age variables. in particular, the first principal component extracted from these vectors corresponds to gender, clearly separating i woman, * from i man, * tokens (figure 1a and 1b ). the point-biserial correlation coefficient between gender and the first component is 0.986 (@xmath0. we expected that the second principal component would correspond to age. however, the results are more nuanced: the second component corresponds to younger age (figure 1a ) with spearman's ρ = 0.965, @xmath1, while the third component corresponds to older age (figure 1b ) with spearman's ρ = 0.928, @xmath2. the interaction between these components and age is shown in figure 1c . we hypothesise that this might be explained by graduation from university and the transition to working life, as the curve's turning point (25-26 years in figure 1c ) roughly matches the age when students typically complete their degrees in russia. ## discussion in this paper, we introduced a novel approach that leverages readily available data sources, such as social media, to study identity. unlike traditional methods that rely on self-report surveys, our method allows for the study of identity in natural settings and on a larger scale. while we used data from vk, the same technique can be applied to other datasets as well. for example, self-reported gender and age have been extracted from posts on popular platforms such as reddit and twitter @xcite , making it possible to apply our method directly to these datasets. attributes that can be used to construct enhanced i-tokens are not limited to gender and age. for instance, with datasets containing profession information on reddit @xcite or educational outcomes on vk @xcite , it becomes possible to study differences between various socio-economic groups. moreover, this approach can be extended beyond social media data. our experiments demonstrate that the corpus does not need to be exceptionally large for the method to be effective. therefore, it could be applied to tv scripts to analyse the representation of different groups on television, building upon previous research in this area @xcite . as a proof of concept, we applied our method to study gendered self-views. we found that the approach not only reproduces established results but also allows for new findings by covering a wider age range than is typically available in surveys. this method can similarly be applied to other phenomena using curated word lists. alternatively, an open dictionary approach can be used to identify and examine words that are especially close to certain enhanced i-tokens in a corpus of interest. the introduced method relies on natural language processing techniques that are admittedly no longer considered state-of-the-art. since the introduction of word2vec @xcite , more advanced models have emerged, particularly fasttext @xcite , which operates at the character n-gram level and potentially offers superior embeddings for morphologically rich languages such as russian. later, contextual word embedding models were developed, most notably bert @xcite , which outperformed static models in a wide range of tasks. however, we believe that advances in machine learning outpace their adoption in social sciences, and there are still many opportunities for new insights to be obtained from using static continuous representations of words. while newer models have led to remarkable performance gains in machine learning applications, we believe that the higher interpretability and computational efficiency of simpler models might still make them preferable for analytical purposes and applications in social science. the idea of using semantic projections traces its origins back to at least 2016 @xcite , when a gender axis was constructed to reveal biases in word embeddings. this methodology was later formally introduced in @xcite , re-introduced in @xcite , and re-introduced again in @xcite . it was further extended to contextual word embeddings @xcite . despite these developments within the computer science literature, their adoption in social sciences has been relatively slow. one possible explanation is that these methods enable the identification of biases at an aggregated level of entire corpora, which, while interesting, has limited applications. in our paper, we build upon previous ideas and show how they can be extended to study differences between social groups. we believe this opens up many new possibilities that would be particularly appealing to social scientists.