Identifying Fairness Issues in Automatically Generated Testing Content (2024)

Abstract: Natural language generation tools are powerful and effective for generating content. However, language models are known to display bias and fairness issues, making them impractical to deploy for many use cases. We here focus on how fairness issues impact automatically generated test content, which can have stringent requirements to ensure the test measures only what it was intended to measure. Specifically, we review test content generated for a large-scale standardized English proficiency test with the goal of identifying content that only pertains to a certain subset of the test population as well as content that has the potential to be upsetting or distracting to some test takers. Issues like these could inadvertently impact a test taker’s score and thus should be avoided. This kind of content does not reflect the more commonly-acknowledged biases, making it challenging even for modern models that contain safeguards. We build a dataset of 601 generated texts annotated for fairness and explore a variety of methods for classification: fine-tuning, topic-based classification, and prompting, including few-shot and self-correcting prompts. We find that combining prompt self-correction and few-shot learning performs best, yielding an F1 score of 0.79 on our held-out test set, while much smaller BERT- and topic-based models have competitive performance on out-of-domain data.

## introduction large language models (llms) have become ubiquitous in the space of natural language generation (nlg) due to recent advances in model capability @xcite . however, these improvements come with the potential for various negative societal impacts. these negative impacts include even native english speakers would likely not be able to identify the correct response if they were not familiar with taylor swift. in (b), the generated question does not require specific background knowledge, so test takers would not need to use specialized knowledge to identify the correct answer. our goal is to identify and filter content like (a) to help ensure fair testing. the generation of misinformation/propaganda, allocation harms of systems providing benefits only to certain groups of people, and representational harms revolving around bias and stereotyping. natural language processing (nlp) models-including llms-are known to reflect and repeat harmful biases and stereotypes @xcite @xcite , and research into how the community addresses the societal harms engendered by nlp technology is critical @xcite @xcite . many of these types of bias in language generation are well-studied. biases based on gender @xcite @xcite , race @xcite , nationality @xcite , and disability @xcite have been identified in language models, and many modern llms incorporate deliberate safeguarding measures in an attempt to alleviate these issues @xcite . in the area of language assessment, there exists a tangential set of issues regarding fairness to test takers and score users @xcite . these issues are particularly dangerous when applied to language learning and assessment; tests with inherent biases have the potential to compromise the validity of the test. therefore, content that is irrelevant to the skills and abilities the test is intended to measure should be avoided (figure 1 ). this includes content that could disadvantage anyone based on their culture, location, or experiences (e.g., focusing on barbeques on the 4th of july could disadvantage test-takers who are unfamiliar with u.s. culture); their emotions (e.g., health hazards and diseases can evoke negative emotional responses among some people); their worldviews (e.g., luxury cruises or designer clothing may make some people feel excluded); and other factors. we refer to these types of issues as fairness issues. knowing how to better understand, detect, and mitigate bias related to fairness in nlg not only raises awareness of the issue but also enables researchers and developers to create more fair and inclusive nlp systems, evaluation metrics, and datasets in the language assessment space. our goal is to build a system for identifying fairness-violating content in automatically generated texts. it is of course still necessary to have human review and revision of the content, but by adding a filtering process after generation and before manual review, we can significantly reduce the time taken for reviewing and the chance that fairness-related content is mistakenly allowed. to accomplish this goal, we explore four different approaches: fine-tuning, topic-based classification, few-shot prompting, and prompt-self correction. our methods need to adapt to new contexts: our definition of fairness is operationally defined by the particular testing context, and may not apply to others, so the guidelines, prompts, and models may not apply generally to new contexts. for this reason, we assess our methods on two held-out test sets and analyze how our methods could be applied to new contexts. we release our resulting dataset, consisting of 601 samples, of which 19.4% contain fairness issues 2 , to facilitate improvements in the fairness-detection community. 2 each sample we used was rejected for deployment in actual tests. using rejected samples for our experiments allows us to release the dataset: accepted stimuli cannot be made public. our contribution consists of the following: ## related work bias, fairness, and responsible ai has been at the forefront of education technology, with contemporary research focusing on automated scoring, writing assistance, and other nuances of applying nlp technology to this sensitive domain @xcite . @xcite find that assisted writing tools may exhibit moderate bias depending on the task, while @xcite found no significant gender bias difference in writing done with and without automated assistance. @xcite explore bias in educational tools for german peer review, and @xcite propose novel methods for detecting bias in automated scoring algorithms. we are specifically interested in applications to language generation, and there is also substantial work in using llms and other nlp technology to generate content for educational assessments @xcite @xcite @xcite . however, this work largely fails to address bias and fairness issues in content generation. our work is specifically focused on fairness issues in automatically generated language testing content. in the context of language models, fairness and bias have emerged as critical concerns. existing detection and mitigation tools generally diverge from our work: some are overly domain-specific like the focus on news articles in @xcite , while others are focused on assessing issues within the language models and datasets @xcite , rather than the outputs. other works rely on retrospective metrics that assess a model's fairness through aggregated predictions and subgroup analysis, and/or focus on classification rather than generation problems @xcite @xcite . although these tools enhance transparency and accountability for evaluating language model issues, they fundamentally differ from our bias detection approach tailored for evaluating generated text in real-time for a production environment. ## problem motivation in the language testing context, we face a unique set of fairness challenges in generating content. specifically, fair testing requires content that does not contain irrelevant factors that negatively impact the assessment of a test taker. a primary concern is to ensure that the test content measures only what it is intended to measure. for english-language proficiency tests, this means that the test must measure only the skills and abilities needed to communicate effectively in english, and not other constructs such as background knowledge of specific jobs, events, or cultures. consider the following question and an example of a response to that question: eliminating the fairness issue for this type of question would result in the following revision: in addition to avoiding testing outside knowledge, it is also important that language proficiency tests do not include content that is offensive or disturbing. for example, the following question and response refer to serious health issues, which have the potential to evoke deep negative emotions. content like this that could prompt strong feelings of anger, sadness, or anxiety should be avoided because it could derail a test taker's concentration, resulting in lower performance on the test. how a test taker interacts with this test content may tell more about their ability to concentrate under emotional strain than about their ability to identify a response's linguistic appropriateness. eliminating this construct-irrelevant content helps to ensure that the test measures only the skills and abilities it is intended to measure. ## methods our goal is to detect whether a generated stimulus contains an issue as a binary classification task. we build a dataset of texts labeled for potential fairness issues and explore potential detection methods. ## test results the previous experiments describe our attempts to identify the best-performing model for fairness classification on our validation set. our goal is to develop a system that generalizes. for this, we evaluate the best-performing of the above model types on two held-out test sets: best performance combining the data-driven prompt with self-correction and few-shot learning performs the best on the in-domain test. this shows this is the best approach if there is available data and expertise to support hand-crafting a data-driven prompt and running self-correction. on the out-of-domain data, the smaller initial prompts, generic (short) and guideline (short) both outperform the data-driven prompt, perhaps due to their more generic nature: the data-driven prompt is too specific to this dataset, and understandably doesn't generalize well. the self-correct+few-shot methodology performs the best in both cases: few-shot learning alone is better than self-correction alone, but the combination is typically the best. strong results from small models traditional transformer-based classification performs remarkably well, especially in generalizing to the out-ofdomain data. on the in-domain data, the best performing model deberta-base performs on par with the best base prompting model (0.58 compared to 0.60 f1 score), although this is a significant drop from the validation performance of 0.77, and performs quite poorly on out-of-domain data (0.20), indicating the model may overfit during training. on the out-of-domain data, roberta-base per-forms nearly as well as the best-performing overall model, just 0.04 behind the generic (short) prompt with self-correction and few-shot learning. if the goal is to quickly and cheaply build a system that is applicable to a wide variety of domains, there appears to be significant value in relying on these relatively small transformer-based classification models. the topic (data) approach is also competitive on out-of-domain data, and does not even require model training; it lags only slightly behind the roberta-base model. self-correction we found significant success in our proposed self-correction mechanism. while it typically does not outperform few-shot learning in isolation, the methods are naturally complementary, and the combination often yields the best-performing model. in examining the models' self-corrections, we find that when asked to become more restrictive, the model tends to add sentences with new constraints, which nicely reflect the issue that was missed. when asked to become less restrictive, the model tends to add hedges to currently existing constraints. in our experiments, we noted some issues. first, when run using too many samples or batches, the prompts tend to degrade: once the llm makes an error and returns a prompt that doesn't match the specifications, the run needs to be aborted. even when the llm sticks to the instructions, after many iterations the prompts become unwieldy and selfcontradictory, and performance rapidly declines. we suggest using somewhere between six and 20 total samples for prompt self-correction; it is best to avoid making corrections indefinitely. ## conclusions this work delivers four key contributions: an exploration of a novel fairness detection task, a dataset of 601 samples annotated for fairness issues, evaluation of a variety of classification models for this task, including fine-tuning, topic-based approaches, and prompting, and a novel prompting strategy, which, combined with few-shot learning, achieves state-of-the-art performance on the task. this work is aimed to explore the space of fairness and bias issues in generated content, especially in the education context. we aim to highlight the difficulties of accounting for fairness, particularly in specific contexts unlikely to be accounted for by traditional model guardrails. as language model usage becomes more prevalent, the need for proper bias and fairness strategies from people training, deploying, and using these models is paramount.