BESSTIE : A Benchmark for Sentiment and Sarcasm Classification for Varieties of E nglish (2025)

Abstract: Despite large language models (LLMs) being known to exhibit bias against non-mainstream varieties, there are no known labeled datasets for sentiment analysis of English. To address this gap, we introduce BESSTIE, a benchmark for sentiment and sarcasm classification for three varieties of English: Australian (en-AU), Indian (en-IN), and British (en-UK). Using web-based content from two domains, namely, Google Place reviews and Reddit comments, we collect datasets for these language varieties using two methods: location-based and topic-based filtering. Native speakers of the language varieties manually annotate the datasets with sentiment and sarcasm labels. To assess whether the dataset accurately represents these varieties, we conduct two validation steps: (a) manual annotation of language varieties and (b) automatic language variety prediction. We perform an additional annotation exercise to validate the reliance of the annotated labels. Subsequently, we fine-tune nine large language models (LLMs) (representing a range of encoder/decoder and mono/multilingual models) on these datasets, and evaluate their performance on the two tasks. Our results reveal that the models consistently perform better on inner-circle varieties (i.e., en-AU and en-UK), with significant performance drops for en-IN, particularly in sarcasm detection. We also report challenges in cross-variety generalisation, highlighting the need for language variety-specific datasets such as ours. BESSTIE promises to be a useful evaluative benchmark for future research in equitable LLMs, specifically in terms of language varieties. The BESSTIE dataset is publicly available at: https://huggingface.co/datasets/unswnlporg/BESSTIE .

## introduction benchmark-based evaluation @xcite of large language models (llms) is the prevalent norm in natural language processing (nlp). benchmarks provide labelled datasets @xcite sent. sarc. eng. var. 2019; @xcite for specific tasks so that llms can be evaluated against them. however, most nlp benchmarks do not contain text in language varieties such as national varieties, dialects, sociolects or creoles @xcite @xcite . as a result, despite the superlative performance of llms on standard american english, llms are not evaluated on other varieties of english. knowing that llms can be biased against certain varieties of english, as shown by @xcite and @xcite for african-american english and indian english respectively, little empirical evidence is found for their bias towards several other varieties. an exception is multi-value @xcite which creates synthetic datasets of language varieties using linguistically-informed syntactic transformations and reports a degraded performance on the glue benchmark. however, such synthetic texts do not accurately represent the spoken or written forms of the language variety because language varieties are more than syntax and encompass orthography, vocabulary, and cultural pragmatics @xcite . as a result, labelled datasets of natural text are crucial to measure the bias in llms towards these varieties. acknowledging the role of the sentiment classification task @xcite in glue, we introduce besstie, a benchmark for sentiment and sarcasm classification for varieties of english, namely, australian (en-au), indian (en-in), and british (en-uk). the relationship between sentiment and sarcasm classification as related tasks is well-understood in the context of sentiment analysis @xcite . besstie comprises a manually labelled dataset of textual posts along with performance results of fine-tuned llms on the two classification tasks. the dataset consists of text samples collected from two web-based domains, namely, google places reviews, and reddit comments, using two filtering methods: location-based and topic-based filtering respectively. we validate the representation of these varieties through a twostep process: (a) a manual annotation exercise and (b) automatic variety identification using models fine-tuned on the ice-corpora @xcite . following that, we obtain manual annotations for boolean sentiment and sarcasm labels from native speakers of the language varieties. we then use the dataset to model both the tasks as binary classification problems. we evaluate nine fine-tuned llms, spanning encoder/decoder and monolingual/multilingual models. our results indicate that current models exhibit a significant decrease in performance when processing text written in en-in, an outer-circle variety, in comparison with inner-circle varieties of english foot_0 (en-au and en-uk). the performance on sarcasm classification is consistently low across the three varieties, pro-viding pointers for future work. following that, we examine the aggregate performance for the three varieties, the two tasks, and model attributes (encoder/decoder and monolingual/multilingual). we also examine cross-variety performance on all models highlighting the need for besstie. besstie evaluation provides a solid baseline to examine biases of contemporary llms, and will help to develop new methods to mitigate these biases. the novelty of the besstie dataset can be seen in the contributions of this work are: (a) creation of a manually annotated dataset of user-generated content with sentiment and sarcasm labels for three varieties of english, called besstie; (b) evaluation of nine llms on besstie and analysing their performance to identify challenges arising due to language varieties. ## dataset creation figure 1 shows the process followed to create a manually annotated dataset for sentiment and sarcasm classification. ## experiment details we conduct our experiments on a total of nine llms. these include six encoder models, three pre-trained on english corpora: bert-large (bert) @xcite , roberta-large (roberta) @xcite , albert-xxl-v2 (albert) @xcite ; and three multilingual models: multilingual-bert-base (mbert), multilingual-distilbert-base (mdistil) @xcite , xlm-roberta-large (xlm-r) @xcite . we also evaluate three decoder models, one pretrained on english corpora: gemma2-27b-instruct (gemma) @xcite and two multilingual models: mistral-small-instruct-2409 (mistral) @xcite , qwen2.5-72b-instruct (qwen) @xcite . the modelspecific details including architecture, language coverage of pre-training corpus, and number of parameters are described in appendix c. while encoder models are fine-tuned with full precision, we fine-tune quantised decoder models using qlora @xcite adapters, targeting all linear layers. all models (including the variety predictor) are fined-tuned for 30 epochs, with a batch size of 8 and adam optimiser. we choose an optimal learning rate by performing a grid search over the following values: 1e-5, 2e-5, and 3e-5. all experiments are performed using two nvidia a100 80gb gpus. we fine-tune encoder models using crossentropy loss, weighted on class distribution (positive: '1' and negative: '0' labels). in contrast, for decoder models, we perform zero-shot instruction fine-tuning @xcite , using maximum likelihood estimation -a standard learning objective for causal language modeling @xcite . for decoder models, we use two task-specific prompts, one for each task. for sentiment classification, we the prompt the model with: given the text and a task-specific prompt, the expected behaviour of the llm is to generate either 1 (for positive) or 0 (for negative). the forms of these prompts are experimentally determined using a few test examples. finally, we report our task performances on three macro-averaged f-score, i.e., unweighted average, disregarding the imbalance in label distribution. ## results we present our results and subsequent discussions to address the following questions: (a) how well do current llms perform on the benchmark tasks? (section 4.1); (b) how do factors such as model properties and domain affect the task performance? (section 4.2); (c) can models trained on one language variety generalise well to others? (section 4.3). ## error analysis we randomly sample up to 30 examples misclassified by mistral, from each variety and domaintask configuration. we then manually analyse these misclassified examples to identify pervasive or obligatory dialect features, as defined by the en-in variety exhibits a high count of dialect features (97) relative to its sample size (90), suggesting that regional variations significantly challenge model performance. locale-specific colloquial expressions are prevalent in all varieties (en-au: 28; en-in: 33; en-uk: 15), highlighting the impact of regional lexicons on model performance. these findings motivate future research in sentiment and sarcasm classification to better accommodate such colloquial expressions for inner-circle varieties, and to additionally adapt to dialect features for outer-circle varieties of english. similarly, several sarcasm classification datasets for standard variety of english exist, consisting of reddit comments @xcite , amazon reviews @xcite , and tweets @xcite @xcite . while datasets like @xcite present annotations for sarcasm, sentiment, and the dialect of each tweet for arabic, there is a notable absence of sarcasm classification datasets that account for non-standard varieties of english. similar to their work, we also create a dataset that contains both sentiment and sarcasm labels. to the best of our knowledge, besstie is the first benchmark for sentiment and sarcasm classification specifically for varieties of english. ## conclusion this paper presents besstie, a benchmark for sentiment and sarcasm classification across three varieties of english: australian (en-au), indian (en-in), and british (en-uk). our evaluation spans nine llms, six encoders and three decoders, assessed on datasets collected from google places reviews (google) and reddit comments (reddit). the models perform consistently better on en-au and en-uk (i.e., inner-circle varieties) than on en-in (i.e., the outer-circle variety) for both tasks. we verify the variety representation of collected data using: (a) manual annotation and (b) automated validation. we also perform an additional annotation exercise to verify the reliability of the annotations. although models report high performance on sentiment classification (f-score of 0.81 and 0.75, for google and reddit respectively), they struggle with detecting sarcasm (f-score of 0.59 on red-dit), indicating that sarcasm classification is still largely unsolved. this is supported by our crossvariety evaluation which reveals limited generalisation capability for sarcasm classification, where cultural and contextual understanding is crucial. notably, monolingual models marginally outperform multilingual models, suggesting that language diversity in pre-training does not extend to varieties of a language. our error analysis provides potential considerations to drive future research in sentiment and sarcasm classification to better accommodate colloquial expressions for inner-circle varieties, and dialect features for outer-circle varieties of english. in conclusion, besstie provides a resource to measure bias in llms towards nonstandard english varieties for the two tasks.