Gender Bias in T urkish Word Embeddings: A Comprehensive Study of Syntax, Semantics and Morphology Across Domains (2024)

Abstract: Gender bias in word representations has emerged as a prominent research area in recent years. While numerous studies have focused on measuring and addressing bias in English word embeddings, research on the Turkish language remains limited. This work aims to bridge this gap by conducting a comprehensive evaluation of gender bias in Turkish word embeddings, considering the dimensions of syntax, semantics, and morphology. We employ subword-based static word vectors trained on three distinct domains: web crawl, academical text, and medical text. Through the analysis of gender-associated words in each domain, we not only uncover gender bias but also gain insights into the unique characteristics of these domains. Additionally, we explore the influence of Turkish suffixes on word gender, providing a novel perspective on gender bias. Our findings reveal the pervasive nature of gender biases across various aspects of the Turkish language, including word frequency, semantics, parts-of-speech, and even the smallest linguistic unit - suffixes. Notably, we demonstrate that the majority of noun and verb lemmas, as well as adverbs and adjectives, exhibit masculine gendering in the general-purpose written language. This study is the first of its kind to offer a comprehensive examination of gender bias in the Turkish language.

## introduction the rise of pretrained language models, such as bert @xcite , has significantly improved various natural language processing (nlp) tasks. however, these models are often trained on large amounts of web-based data, which can contain social stereotypes and biases that may be inherited by the models. this raises concerns as such biases can be perpetuated in downstream applications @xcite . the advent of large language models (llms) @xcite has further highlighted the importance of understanding and evaluating training data quality, including the presence of toxicity and gender bias @xcite . while previous studies have primarily focused on gender bias in embeddings, particularly in english, research on other languages has been limited to a few multilingual projects. for instance, @xcite ) evaluated gender bias in machine translation by translating gender-neutral languages using the google translate api, while (lewis and lupyan, 2019) examined gender stereotypes across 25 natural languages. however, languages other than english have received minimal attention in this research domain. in the case of the turkish language, the existing research is sparse. @xcite investigated overt and covert gender bias in machine translation models by examining gender-neutral turkish and gendered english. @xcite explored gender bias in turkish transformer models. despite the advancements in llms, several crucial research gaps related to the turkish language still remain. firstly, there is a lack of studies focusing on the fundamental form of word embeddings, namely pretrained word vectors. secondly, we strongly believe that turkish morphology warrants an extensive linguistic study that delves into the intrinsic nature of the language itself. this unique aspect of turkish sets it apart from english and other welllstudied western languages and adds an additional dimension to the research on gender bias. to address these gaps, our work aims to fill the research void by conducting a comprehensive evaluation of gender bias in turkish word embeddings, considering the dimensions of syntax, semantics, and morphology. we employ static embeddings, specifically floret vectors, trained on three distinct domains: web data, academic data, and medical data. through our analysis, we investigate the frequency of words associated with men and women, examine the parts of speech associated with each gender, and explore the conceptual clusters of words associated with men and women. additionally, we provide an in-depth exploration of the relationship between morphology and the gendering of words by dissecting the semantic aspects of suffixes. our main contributions are as follows: this paper is organized as follows: we present our data and methodology, followed by domainspecific results related to the pretrained embeddings in each domain. the final section focuses on morphology. our code and data are available in our github foot_0 and huggingface foot_1 repositories. ## methodology in this section, we provide an overview of our methodology, including details about the datasets used, the choice of word vectors, and the process of training and calculating gender-related metrics. ## training and evaluation of word embeddings in our study, we considered the agglutinative nature of the turkish language with its rich morphology. to effectively represent the complex word forms that can be generated through the addition of numerous inflectional and derivational suffixes, we chose to use floret embeddings @xcite , an extended version of fasttext @xcite that incorporates bloom embeddings @xcite . floret combines word and subword information, allowing for more compact vector tables with enhanced representation of the morphological structure. compared to traditional word vectors, floret's subwords are up to 10 times smaller. we utilized the floret library code by spacy @xcite to train our word vectors. the training was conducted using the continuous bag-of-words (cbow) algorithm @xcite , with each word vector having a dimension of 300. for the subwords, we considered 2-grams to 5-grams. to reduce the size of the vocabulary, we used a compact vocabulary of 250,000 entries for the web crawl corpus (mc4) and the academic crawl corpus. for the medical crawl corpus, which has a smaller size, we used a vocabulary of 80,000. the choice of the subword window range [2, 5] was determined heuristically, considering that the length of most common turkish suffixes varies from 1 to 5. to evaluate the quality and effectiveness of the produced embeddings, we compared them to the floret vectors of the pretrained spacy model tr_core_news_lg @xcite on morphology and syntax tasks. we initialized a spacy model with our floret vectors and then trained syntactic 5 each dataset exists with their original turkish name in our huggingface repository. we used english translations to reach a broader audience. names of the datasets we used are clean-mc4, akademik-makaleler, akademik-ozetler, medikal-ozetler and medikal-makaleler, respectively. parser components, including the pos tagger, dependency parser, and morphologizer components, on the boun treebank @xcite . testing was performed on the test division of this treebank. the results, shown in to assess the gender bias encoded in the trained embeddings, we employed the method introduced by @xcite . for each word, we calculated a gender bias score by computing the dot product between its vector and the vector representation of the concept of "woman" subtracted by the vector representation of "man." in our experiments, we used the turkish translations of "woman" (kadın) and "man" (erkek). a positive score indicates a closer association with masculinity, while a negative score implies a stronger association with femininity. the magnitude of the score reflects the degree of bias, with higher absolute values indicating greater bias. a score of 0 indicates neutrality. unlike many other studies, our approach is unsupervised, and we did not employ the word embedding association test (weat) scores. ## results and discussion in this section, we thoroughly examine each data genre individually. we first train the floret word vectors for each genre and analyze the gender distribution of the vocabulary. we then provide statistics on the vocabulary based on different syntactic categories. additionally, we conduct unsupervised clustering separately for each gender using all the words and explore the topics associated with each gender, focusing specifically on the web domain due to the more diverse range of topics compared to academic and medical papers. finally, we delve into the relationship between morphology and gender by investigating how word genders change based on suffixes. in a spacy model, each pipeline component assigns relevant attributes (pos tag, morphological analysis string, lemma, dependency tag, and head in the dependency tree) to tokens. accuracy for pos tag, lemma, and morphology is determined by collecting the attributes for each token and comparing them to the ground truth list. the last two attributes are evaluated at the syntax tree level, assessing the structure of the dependency tree, correct head, and dependency arcs. uas measures structure accuracy, while las additionally evaluates the accuracy of dependency labels on each arc. ## morphology and gender this section of our research is not specific to any particular domain; instead, we focus on exploring the role of morphology. specifically, we investigate how certain types of suffixes influence the gender of words. we examine each type of suffix in detail within this subsection. as mentioned in previous sections, our choice to use subword-enriched word vectors is motivated by a significant factor: we aim to generate representations for suffixes as well. in turkish, a typical word is composed of morphemes, a lemma, and a list of suffixes, with each suffix carrying its own meaning. in this section, we demonstrate the semantic impact of suffixes on the gender dimension. we discuss various groups of suffixes, providing further explanations and examples for each group. for a comprehensive understanding of turkish morphology and detailed explanations of these suffix groups, refer to @xcite . ## conclusion this paper examines the presence of gender bias in static word embeddings of the turkish language. the findings indicate that gender biases are prevalent across various aspects, including word frequency, parts-of-speech, clustered concepts, word meaning dimensions, and even in the smallest units of the language, such as suffixes. overall, the results reveal a pervasive association of words and concepts with men rather than women in turkish pretrained embeddings. furthermore, the study demonstrates how gender associations are differentiated based on parts-of-speech and clusters of concepts, with women being more associated with nouns and domestic content, while men are more associated with "serious matters." these findings raise concerns about the amplification of gender biases in ai and society through social, cultural, and digital mechanisms. komplikasyonlarıdır, kuruluşlardır, kuruluştur, kültürüdür, lezyonlarıdır, olaydır, oluşmaktaydı, oluşmasıdır, oluşmuştur, oluşturabilmektedir, oluşturmaktadır, oluşturulamamıştır, oluşturulmaktadır, oluşturulmasıdır, oluşturur, radyoterapidir, sendromudur, şikayetlerdir, tedavidir, tedavisiydi, teknikidir, varlıkıdır, vitamindir, yöntemleridir, yöntemlerdir. adjective klinik, kadın, ana, medikal, dişi, finansal, jinekolojik, farmakolojik, salgın, cinsel, epidemiyolojik, kozmetik, estetik, kardiyak, kardiyovasküler, istatistiki, finanse, memeli, lösemili, kliniksel, lezbiyen. adverb evdeyken, konunca, bilgilendirilip, kasten, inince.