Did the Names I Used within My Essay Affect My Score? Diagnosing Name Biases in Automated Essay Scoring (2024)

Abstract: Automated essay scoring (AES) of second-language learner essays is a high-stakes task as it can affect the job and educational opportunities a student may have access to. Thus, it becomes imperative to make sure that the essays are graded based on the students’ language proficiency as opposed to other reasons, such as personal names used in the text of the essay. Moreover, most of the research data for AES tends to contain personal identifiable information. Because of that, pseudonymization becomes an important tool to make sure that this data can be freely shared. Thus, our systems should not grade students based on which given names were used in the text of the essay, both for fairness and for privacy reasons. In this paper we explore how given names affect the CEFR level classification of essays of second language learners of Swedish. We use essays containing just one personal name and substitute it for names from lists of given names from four different ethnic origins, namely Swedish, Finnish, Anglo-American, and Arabic. We find that changing the names within the essays has no apparent effect on the classification task, regardless of whether a feature-based or a transformer-based model is used.

## introduction artificial intelligence is being deployed in highstakes situations, such as automated grading of second language essays in proficiency assessment. while ai can improve the opportunities students have in education, the job market, etc., such systems often display human-like biases @xcite . @xcite notes that human graders have a slight bias based on names appearing in essay texts. in this paper we aim to identify whether the same pattern holds in automated systems. the broad question for our study is: are there any implicit biases that models have learnt from the training data that can influence automated essay scoring in a negative way? in particular, we are interested in uncovering potential biases that can be associated with use of names representing different ethnic groups -and how this can be reflected in the domain of automatic essay scoring (aes). for the purposes of this work, we say that there is bias in aes when an essay is scored not only by its contents but also by the assumed demographic characteristics of its author. we use this definition as we are looking for biases in a downstream application (i.e. extrinsic biases) as opposed to biases either in the training data or in any intermediate representations (i.e. intrinsic biases). even though we know that biases in deep learning models cannot be removed in absolute terms @xcite , we can attempt to minimize their impact. because of this, we have set out to create a novel paradigm of diagnostic benchmarks for identifying hidden biases in aes models as a safety gatekeeping before they are approved for use in real-life scenarios. in such a dataset each essay is duplicated (several times), artificially altering given names appearing in the text to identify if such perturbation affects how an essay is scored. since the essays are identical as far as linguistics, language complexity, and content are concerned, we expect them to be graded similarly. thus, we would say that our model for this task presents bias if it systemically assigns lower grades when using versions of the essays with names coming from specific ethnicities. our research questions are the following: for this, we use a de-anonymized (i.e. origi-nal) version of the swell-pilot corpus of second language swedish learner essays @xcite , which consists of 502 essays annotated with cefr levels 1 (council of europe, 2001), as our source data. first, we compile four lists of given names inspired by those in @xcite : traditional swedish names; modern swedish names of anglo-american origin; finnish names (due both to the close sociocultural links between finland and sweden and to swedish being an official language of finland being learnt by the population that does not speak it as their first language); and names of arabic origin (the most prominent group of learners in the corpus). second, we create a diagnostic dataset to identify biases in the classification task. we select swellpilot essays in which a given name appears only once. then, we generate an essay version for each name on the lists by substituting the name in the original text with one from the list. all of the essays chosen have the names in their base form. third, we fine-tune a bert @xcite ) model on the original swell-pilot data to predict the cefr level of a given essay and compare it to an existing feature-based model @xcite . finally, we test the two models and compare the equality of opportunity between the different given name groups on the diagnostic dataset, as described by @xcite . as mentioned previously, we would expect an unbiased or a fair model (in terms of given names) to not show systemic misclassification for the ethnic groups considered. it does not mean that it will be unbiased towards names from other ethnic groups or that different names would not elicit unexpected responses from our model @xcite . it is important to note that a model being fair for a downstream application does not mean that the model itself, the data, or the annotation lack biases (and vice versa). social biases are a very complex phenomenon and they can be embedded in a variety of ways, as illustrated by @xcite . moreover, @xcite note that the presence or absence or intrinsic biases (e.g. in language models) does not necessarily correlate with the presence or absence of extrinsic biases (e.g. in downstream applica-1 cefr stands for common european framework of reference for languages. it is a framework to evaluate foreign language learning and assigns one of six reference levels to determine the proficiency level of a second language speaker. tions). because of this, it is important to monitor and to audit aes models regularly regardless of whether they are fair. and, given that this is a high-stakes task, it is essential to always have a human-in-the-loop approach. the rest of the paper is structured as follows: section 2 reviews some of the related work both in terms of automated essay assessment and of bias and fairness in nlp. section 3 presents our methodology, the models and data we used, as well as how they were evaluated. in section 4 we show and discuss the results from our experiments, while in section 5 we present some ideas for future work. ## related work language assessment and subsequent documented language proficiency, be it for citizenship, university admission or a job application, are extremely influential, if not life-changing, both on the individual, societal and political levels @xcite . assessment should therefore be guaranteed to be fair and unbiased, and assessors should be kept accountable for the results, i.e. be able to motivate the assigned scores (e.g. aslha, 2023; alte, 2020). this is a non-trivial requirement even for human assessors, and is clearly a much greater challenge for automated language assessment. ## experiment setup, materials and methods our major question for the experiment is whether algorithms for essay classification are sensitive to names (or pseudonyms) used in essays. if we need to pseudonymize research data on a constant basis to protect writer identities, which is a gdpr requirement (eu commission, 2016), we should find ways to do so that do not affect students. this means that it is our responsibility to check the effect the replacement candidates may have on the data and its downstream tasks and research applications. in this experiment we study the effects that replacing given names in learner essays might have on essay assessment in terms of cefr 4 level, as described in the introduction (section 1). the cefr levels are a six-level scale to gauge the proficiency of an individual on a foreign language (i.e. not their first language or languages) and they range from a1 to c2, with a1 being the lowest. ## results and discussion we can notice from when looking at the performance on the diagnostic set in tables 3 and 4 , we noticed that changing the names in the text of the essays yielded no change in performance with either of the models. that is, the equality of opportunity of the different groups and subgroups is zero, indicating that the model is not unfair under this metric. testing with a wider array of names yielded no differences either in terms of class assigned. on a similar note, when checking for biases regarding whether the names were male or female we found no difference in performance. as mentioned in section 3.3, we did not expect the feature-based model to show much bias, if at all. this is due to it not using features directly related to the vocabulary. on the other hand, we expected the transformerbased model to display some sort of bias considering the previous literature on name biases in nlp (see section 2). this means that ultimately neither the distribution of the demographics in the training set nor the biases in the base bert model (i.e. intrinsic biases) had any effect on the fairness of the model (i.e. extrinsic bias). a possible direction on which this study could be expanded to would be a thorough analysis of given names present in the vocabulary of the bert model and seeing whether there is any correlation between how the model behaves for each of these. these results are consistent with what we would expect from a fair model for aes for second language assessment. that is, we expect it to score the students in terms of their linguistic skills and proficiencies as opposed to other unrelated things. one of the possible issues that we could have run into were the essays used for the diagnostics dataset. while they represent different cefr lev- ## conclusions and future work in this work we examined how changing given names within the text of second language learner essays of swedish affects the cefr level they are assigned to by the models. we found that changing the names did not change the performance of the model in any noticeable way across four different name groups with twenty names each. this points to our models learning to differentiate the level of an essay based on linguistic characteristics, as opposed to the kind of personal identifiable information found within the essays, such as given names. because of this, we think that pseudonymization should be considered as a viable method to allow for research data to be used and shared. however, it is important to note that these results could vary from language to language and from dataset to dataset. there is no silver bullet to solve the bias issue in nlp, as it is deeply ingrained within human perception and the data we generate, which can lead to unexpected results @xcite . moreover, it is possible that the chosen given names and ehtnic groups could have had an impact on our results, as argued by @xcite . this would be particularly important when considering people coming from regions under-represented in our data, as they are the most at risk of being the most affected by discrimination, be it from humans or from machines. there are several directions in which our work could be expanded to. one would be to use more essays for the diagnostic dataset. as mentioned in section 4, this would require either acquiring new essays or generating synthetic data, both of which can be challenging tasks. another possible direction to expand our work to would be to do an in-depth analysis of the given names appearing both in different corpora as well as in the training data of the different models. this would allow us to verify that the lack of perceptible bias we found was not due to the names not appearing on the data. both of these could be used as a paving stone to create guidelines on how to generate diagnostic datasets to identify biases in automated essay scoring of second language learner essays. it would be particularly interesting to analyze whether the same patterns hold for different kinds of personal identifiable information, such as other kinds of personal names and places. moreover, it would be good to check whether this apparent lack of bias is maintained when dealing with several pieces of private information at the same time.