Using Contextually Aligned Online Reviews to Measure LLM s’ Performance Disparities Across Language Varieties (2025)

Abstract: A language can have different varieties. These varieties can affect the performance of natural language processing (NLP) models, including large language models (LLMs), which are often trained on data from widely spoken varieties. This paper introduces a novel and cost-effective approach to benchmark model performance across language varieties. We argue that international online review platforms,such as Booking.com, can serve as effective data sources for constructing datasets that capture comments in different language varieties from similar real-world scenarios, like reviews for the same hotel with the same rating using the same language (e.g., Mandarin Chinese) but different language varieties (e.g., Taiwan Mandarin, Mainland Mandarin). To prove this concept, we constructed a contextually aligned dataset comprising reviews in Taiwan Mandarin and Mainland Mandarin and tested six LLMs in a sentiment analysis task. Our results show that LLMs consistently underperform in Taiwan Mandarin.

## introduction a language can have different varieties. of the world's 7,000 languages, sixty (60) million people speak british english, 23 million speak taiwan mandarin, and 10 million speak european portuguese, compared to 330 million, 900 million, and 200 million who speak american english, mainland mandarin, and brazilian portuguese, respectively. these varieties differ enough in accent, vocabulary, or syntax for native speakers to distinguish them. nlp technologies, including llms, are known to perform better in english varieties that are more widely represented in the internet data they are trained on, particularly mainstream american english (mae), compared to less represented varieties like african american english (aae) @xcite . specifically, llms more accurately predict sentiment scores in figure 1 : online review platforms can be data sources to build datasets that capture comments in different language varieties from similar real-world scenarios. these contextually aligned datasets can then be used to benchmark llms' performance across language varieties. mae @xcite , generate higher-quality texts in mae @xcite , and hold better conversations in mae @xcite . these comparisons were made possible by intensive, targeted efforts specific to each language variety, such as "translating" data instances from a standard variety (e.g., mae) to less widely represented varieties (e.g., aae), followed by validation from native speakers @xcite . what is not known is whether these performance gaps and biases extend to a broader range of languages and their numerous varieties, such as mainland mandarin versus taiwan mandarin. building effective benchmarking datasets for evaluating model performance across language varieties is expensivecreating "fair" comparisons between varieties often needs native speakers and language experts. using mandarin chinese as an example, we propose an approach that uses large-scale usergenerated reviews to construct benchmarking datasets across varieties of a given language. we argue that the international online review platforms with millions of users, like booking.com, when properly curated, can serve as effective data sources for constructing datasets that capture comments in different language varieties from similar realworld scenarios, like comments for the same hotel with the same rating using the same language (e.g., mandarin chinese) but different language varieties (e.g., taiwan mandarin, mainland mandarin). these datasets, being contextually aligned, can then be used to benchmark llms' performance across language varieties for tasks like sentiment analysis and text generation (figure 1 ). once a lowcost and generalizable approach becomes available, researchers can then compare model performance across a wide range of language varieties, enabling reliable benchmarking of progress in addressing performance gaps and moving toward an llm that performs equally well across all language varieties. ## related work beyond machine translation @xcite , researchers tried to benchmark nlp models across language varieties @xcite @xcite @xcite , but the focus on identifying gaps between these varieties varies widely. some prior work focused solely on a single less-representative variety, such as taiwan mandarin @xcite , without measuring performance gaps across multiple varieties. other studies that measured these gaps employed different levels of granularity. the most common approach, task-level comparison, benchmarks the same nlp task across language varieties @xcite , such as sentiment analysis, but datasets often differ in source or genre across varieties, making the reported performance numbers not directly comparable. for instance, sentiment analysis datasets for mainland mandarin and taiwan mandarin often used different sources @xcite . a more refined approach, scenario-level comparison, evaluates performance within the same dataset or scenario, such as essay grading @xcite or speech rating @xcite , across data partitions of different language varieties @xcite . while this method eliminates biases caused by differing data sources, it cannot fully address biases introduced during dataset construction. the most rigorous method, instance-level comparison, involves constructing parallel datasets with an item-by-item alignment between varieties @xcite @xcite , where each instance is converted between language varieties. however, creating such comparisons is very costly, requiring native speakers and language experts to ensure accuracy. our approach achieves instancelevel comparability with lower costs. ## constructing a contextually-aligned review dataset for language varieties data. we constructed a dataset of hotel reviews sourced from booking.com, foot_0 which has been used in prior research studies @xcite . this dataset consists of 4,447,853 reviews labeled by the platform as written in chinese. the reviews cover 149,879 hotels located in japan, mainland china, south korea, taiwan, thailand, and vietnam, and were collected from august 2021 to august 2024. these locations were selected to ensure a substantial volume of data, as they are popular destinations for mandarinspeaking travelers. each review comprises three main components: the review title, positive feedback, and negative feedback. additionally, it includes review ratings (ranging from 1 to 10 stars) and metadata such as hotel id, posting time, and more (see appendix a for an actual sample). booking.com claims to invest significant effort in ensuring that reviews are posted by real users and in maintaining review quality. we included only nonempty reviews, meaning reviewers provided input in at least one of the following: the review title, positive feedback, or negative feedback. in total, we collected 1,513,056 reviews written in chinese. ## experimental results to examine biases from review structure, we tested three settings: (i) structured review retains the original format with title, positive, and negative feedback. (ii) plain review concatenates all elements into a single paragraph. (iii) shuffled review includes all elements but in random order. for the analysis, we excluded pairs that lacked complete predictions or received predictions that did not follow the specified format (see appendix d). once the contextually aligned dataset was constructed and available, we tested it using six llms: gpt-4o, llama3 (8b, 70b, 405b), and gemma2 (9b, 27b). the task involved predicting a rating score (from 1 to 10, where 1 is the worst and 10 is the best) based on the review content. the prompt (appendix c) includes the task description, the review content, and the prediction scale (1-10). llms performed significantly worse in taiwan mandarin compared to mainland mandarin. among all 54 experiments with different models and prompt settings, 38 of them had significant group differences in accuracy ( when the reviews' structures are disrupted, the performance gap increases. shorter reviews tend to produce larger mse gaps. our pilot study (appendix e) found that shorter texts may lack information and often affect model performance and behavior. we thus categorized our dataset into two groups based on review's text length: short (1-49 chinese characters) and long (50+ chinese characters). ## examining confounding variables could the performance gap be due to mainland mandarin reviews having better writing quality or better alignment between content and ratings? rationale: better writing quality or better content-rating alignment could make it easier for llms to predict ratings. analysis & findings: no. our human validation (section 3.2) shows that mainland mandarin reviews had slightly worse writing quality and content-rating alignment. could the performance gap be due to more code-mixed usage in taiwan mandarin? rationale: nlp models often struggle with codemixed data @xcite . analysis & findings: no. the mainland mandarin reviews contain more mixed-language input (30.99%) than the taiwan mandarin reviews (25.26%, see appendix g and could the performance gap be due to mainland mandarin users systematically giving higher scores, which align better with llm-generated scores? rationale: llms tend to assign higher scores @xcite @xcite . analysis & findings: unlikely. in our dataset, taiwan mandarin and mainland mandarin reviews show no significant difference in scores (t(22917) = .160, @xmath0.873). are mainland mandarin reviews easier for humans to guess ratings? rationale: human performance is sometimes used as an indicator of a task's difficulty for llms @xcite . analysis & findings: plausible. we conducted a user study with 10 participants (5 native speakers from each variety) who reviewed 50 random cn-tw review pairs (100 total reviews) and predicted their rating scores. participants performed significantly better at predicting ratings for reviews in mainland mandarin. after excluding two tw native speakers whose accuracy was more than two standard deviations below the mean, 6 out of the 8 participants had better accuracy on cn reviews than tw reviews, and 7 had better (lower) mse on cn reviews than tw reviews (see appendix b.2 for more details). these results should be interpreted with caution. unlike question-answering, predicting hundreds of review scores from content is not a typical human task, and most nlp papers on sentiment analysis do not compare model performance to human performance. thus, it is unclear whether human performance gaps in such tasks reliably indicate task difficulty for llms, especially given the small differences between the two varieties. additionally, our participants may not represent the average mandarin speaker's ability in sentiment analysis, as the two participants performed notably poorly. finally, despite our efforts to examine confounding variables such as text length, code-mixing, and writing quality, we still lack a clear understanding of what causes the observed llms' performance gaps across language varieties. 6 discussion do users who self-label as being from taiwan always use taiwan mandarin? in this study, we use users' self-reported nationality/region to infer whether they are speakers of taiwan mandarin or mainland mandarin. the convention is that taiwan mandarin employs traditional chinese characters, while mainland mandarin uses simplified characters. however, analysis using predefined character sets revealed that 30.99% of samples in the cn group contained characters beyond simplified chinese, and 25.26% of samples in tw group included characters not limited to traditional chinese. this suggests that the relationship between self-reported nationality/region, language variety, and character usage is more complex in real-world data. in appendix g, ## conclusion and future work this paper introduces a cost-effective method for benchmarking model performance across language varieties using international online reviews from similar contexts. to validate this, we built a contextually aligned dataset of taiwan mandarin and mainland mandarin reviews and tested six llms on sentiment analysis, finding that llms consistently underperform in taiwan mandarin. we aim to extend this approach to more language varieties, with the ultimate goal of creating llms that perform equally well across them.