Dialectal Toxicity Detection: Evaluating LLM -as-a-Judge Consistency Across Language Varieties (2025)

Abstract: There has been little systematic study on how dialectal differences affect toxicity detection by modern LLMs. Furthermore, although using LLMs as evaluators (“LLM-as-a-judge”) is a growing research area, their sensitivity to dialectal nuances is still underexplored and requires more focused attention. In this paper, we address these gaps through a comprehensive toxicity evaluation of LLMs across diverse dialects. We create a multi-dialect dataset through synthetic transformations and human-assisted translations, covering 10 language clusters and 60 varieties. We then evaluate five LLMs on their ability to assess toxicity, measuring multilingual, dialectal, and LLM-human consistency. Our findings show that LLMs are sensitive to both dialectal shifts and low-resource multilingual variation, though the most persistent challenge remains aligning their predictions with human judgments.

## introduction toxicity and hate speech detection has become essential for creating safer online environments @xcite . the rise of large language models (llms) has advanced the detection of toxic content, but challenges remain in addressing implicit biases within these models @xcite . while llms are increasingly used as automated "judges" for bias and toxicity assessments, their judgments still reflect underlying biases @xcite . despite progress in multilingual and dialectal toxicity detection @xcite , a key gap persists in understanding how dialectal variations affect llms' toxicity judgments compared to standard languages. while these models often perform well, they tend to show low agreement with human evaluators on multilingual context-dependent content (de wynter et al., 2024). current benchmarks largely ignore dialectal complexities @xcite , underscoring the need for focused research on how dialects influence llm judgments. this work addresses these issues through the following contributions: by focusing on both synthetic and real-world dialectal data, this study provides a holistic view of how llms perceive and evaluate toxicity across diverse language varieties, contributing to the broader goal of creating fairer and more effective toxicity detection systems. the figure shows the process of creating a multilingual, multi-dialect toxicity dataset through machine translation, dialect synthesis and real-world speaker utterances. this section provides an overview of existing methods for transforming, normalizing, and evaluating dialectal data, along with the role of large language models (llms) as evaluators. ## dialectal toxicity evaluation framework our framework for evaluating the robustness of llms against toxicity in various dialects can be divided in two key steps: (i) dialectal dataset expansion (ii) llm-as-a-judge consistency evaluation. ## experimental setup we evaluate the performance of five llms to assess their capability in detecting toxicity across a diverse set of standard and dialectal language varieties. here we choose those models, that already exhibits their superior performance in multilingual benchmarks. our evaluation includes standard classification metrics such as accuracy and f1 score, followed by consistency-based analyses to assess the robustness of model predictions across multilingual and dialect-sensitive settings. it demonstrates strong performance on multilingual evaluation benchmarks (e.g., mmlu), particularly in european languages. for the remainder of this paper, we refer to mistral-nemo-instruct-2407 as nemo, gpt-4.1-2025-04-14 as gpt, llama-3.1-8b-instruct as llama, qwen2.5-7b-instruct as qwen, and gemma-3-12b-it as gemma. ## results and analysis in this section, we present our experimental findings. the original human-labeled toxicity intent scores range continuously from 1 to 5 and are discretized into five ordinal bins to standardize comparison across models (see section f). we evaluate model performance using two complementary metrics: rmse-based similarity, which measures the deviation between model predictions and binned human labels (normalized and inverted to yield a similarity score between 0 and 1), and macroaveraged f1, which assesses classification accuracy across toxicity levels. full metric definitions are provided in section g. broad model comparisons rmse similarity scores range from 57.6 to 65.8, indicating relatively low alignment with human annotations. gemma consistently achieves the highest performance across both metrics. nemo ranks second in f1, while qwen performs second-best in rmse-sim, suggesting that ranking can differ depending on the evaluation perspective. interestingly, gpt scores lowest on rmse-sim, indicating that larger model size alone does not ensure better alignment with human judgments. overall, the agreement remains modest across all models, pointing to a broader challenge in reliably capturing human-defined toxicity signals. results across language clusters model performance varies noticeably across language clusters. in higher-resource languages such as english, arabic, and chinese, models tend to perform better, with relatively higher f1 and similarity scores. in contrast, performance drops in lower-resource clusters like sotho and kurdish. for instance, the lowest rmse similarity score appears in gpt's predictions for kurdish (50.3), which is over 10 points lower than gemma's score on the same cluster. these differences highlight persistent disparities in model robustness across language varieties, especially for underrepresented or morphologically complex languages. ## conclusion and future work we propose a holistic llm robustness evaluation framework for handling toxicity across language varieties. our findings suggest, a notable gap remains between model predictions and human judgment, emphasizing the need for improvements in alignment. additionally, llms tend to be more sensitive to low-resource dialects, indicating that further advancements are required to enhance their consistency across diverse language varieties. we aim to further expand our dataset by incorporating more utterance-based dialects and introducing new perturbation methods, leveraging llms' understanding of dialectal variations.