AAVENUE : Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark (2024)

Abstract: Detecting biases in natural language understanding (NLU) for African American Vernacular English (AAVE) is crucial to developing inclusive natural language processing (NLP) systems. To address dialect-induced performance discrepancies, we introduce AAVENUE (AAVE Natural Language Understanding Evaluation), a benchmark for evaluating large language model (LLM) performance on NLU tasks in AAVE and Standard American English (SAE). AAVENUE builds upon and extends existing benchmarks like VALUE, replacing deterministic syntactic and morphological transformations with a more flexible methodology leveraging LLM-based translation with few-shot prompting, improving performance across our evaluation metrics when translating key tasks from the GLUE and SuperGLUE benchmarks. We compare AAVENUE and VALUE translations using five popular LLMs and a comprehensive set of metrics including fluency, BARTScore, quality, coherence, and understandability. Additionally, we recruit fluent AAVE speakers to validate our translations for authenticity. Our evaluations reveal that LLMs consistently perform better on SAE tasks than AAVE-translated versions, underscoring inherent biases and highlighting the need for more inclusive NLP models.

## introduction nlp systems have shown exceptional performance on various benchmarks, excelling in tasks such as sentiment analysis, machine translation, and question answering @xcite @xcite . however, these benchmarks feature mainly sae, often neglecting nonstandard dialects such * lead author † senior authors as aave @xcite @xcite . this oversight results in biased and inequitable nlp systems that do not adequately serve a significant portion of speakers. the advent of llms such as gpt-4 has driven advances in nlu tasks, achieving state-of-the-art results across various applications @xcite @xcite . despite these advancements, llms exhibit persistent biases against nonstandard dialects, including aave, particularly in tasks involving natural language generation and toxicity detection @xcite . these biases highlight the need for comprehensive benchmarks that evaluate model performance in different dialects, ensuring fair and reliable language technologies for all users @xcite . benchmarks such as glue and superglue have contributed significantly to the evaluation of nlp models, yet they focus mainly on sae, neglecting crucial dialectal variations @xcite . although value has sought to bridge this gap by using deterministic linguistic transformations to assess model performance in aave, these transformations are often contextspecific and difficult to generalize, limiting their broader applicability @xcite . to address these limitations, we introduce aav-enue, a benchmark specifically designed to evaluate llm performance across dialects, with a particular focus on aave. our goal is to promote inclusivity and equity in nlp tools by providing a more comprehensive assessment of dialectal fairness. our contributions are as follows: ## dataset to evaluate the performance of models on sae and aave, we selected five key tasks from the glue and superglue benchmarks, known for their diversity and relevance to natural language understanding tasks @xcite . these tasks include boolq, multirc, sst-2, copa, and wsc. ## results we evaluated the accuracy of the translations in five tasks using five llms. the accuracy scores show the performance of each model in sae and aave translations, highlighting consistent performance drops when handling aave translations across all models, as shown in ## related work the development of benchmarks such as glue and superglue has significantly advanced the evaluation of language models on a variety of nlu tasks, including sentiment analysis, natural language inference, and reading comprehension @xcite . while these benchmarks have become standard tools for assessing model performance, their primary focus on sae often leads to performance disparities when applied to non-standard dialects. to address these disparities, the value benchmark was introduced. value incorporates deterministic linguistic transformations to evaluate the performance of the model in aave @xcite . although value aims to provide a comprehensive evaluation of models processing dialectal variations, its deterministic nature can limit generalizability across various contexts, reducing broader application effectiveness. recent advances in llm, such as gpt-4, have shown substantial improvements in nlu tasks, including sentiment analysis, machine translation, and question answering @xcite @xcite . despite these advancements, research indicates that these models still exhibit biases against non-standard dialects like aave, particularly in tasks that involve natural language generation and toxicity detection @xcite . this bias underscores the need for more inclusive benchmarks and evaluation frameworks. the gem benchmark and other studies have highlighted the importance of evaluating and mitigating biases in nlp models to promote fairness and inclusivity @xcite . these works emphasize the necessity of developing benchmarks that evaluate model performance across a range of dialects, ensuring that language technologies are equitable and reliable for all users. by introducing aavenue, we aim to provide a comprehensive benchmark that evaluates llm performance on both aave and sae, promoting inclusivity in nlp tools and addressing the limitations of existing benchmarks. this approach aligns with recent research that emphasizes the need for more dialect-inclusive benchmarks to enhance the fairness and reliability of language technologies across diverse linguistic communities. ## conclusion this paper introduced aavenue, a benchmark designed to evaluate llms on aave and sae tasks. by leveraging gpt-4o-mini and few-shot prompting, we translated sae tasks to aave. our comprehensive evaluation, considering metrics like fluency, quality, and understandability, along with feedback from human validators, revealed that our translations were more superior then values translations. our evaluation results, revealed inherent biases in llms, highlighting a performance gap between sae and aave. these findings build on the foundational work of others in this field, emphasizing the need for more inclusive training data and improved model architectures to address dialectal variations. we plan to extend this work by exploring additional dialects and further refining our translation methods. by doing so, we aim to set a new standard for equitable and accurate language processing across diverse communities.