The Skipped Beat: A Study of Sociopragmatic Understanding in LLM s for 64 Languages (2023)

Abstract: Instruction tuned large language models (LLMs), such as ChatGPT, demonstrate remarkable performance in a wide range of tasks. Despite numerous recent studies that examine the performance of instruction-tuned LLMs on various NLP benchmarks, there remains a lack of comprehensive investigation into their ability to understand cross-lingual sociopragmatic meaning (SM), i.e., meaning embedded within social and interactive contexts. This deficiency arises partly from SM not being adequately represented in any of the existing benchmarks. To address this gap, we present SPARROW, an extensive multilingual benchmark specifically designed for SM understanding. SPARROW comprises 169 datasets covering 13 task types across six primary categories (e.g., anti-social language detection, emotion recognition). SPARROW datasets encompass 64 different languages originating from 12 language families representing 16 writing scripts. We evaluate the performance of various multilingual pretrained language models (e.g., mT5) and instruction-tuned LLMs (e.g., BLOOMZ, ChatGPT) on SPARROW through fine-tuning, zero-shot, and/or few-shot learning. Our comprehensive analysis reveals that existing open-source instruction tuned LLMs still struggle to understand SM across various languages, performing close to a random baseline in some cases. We also find that although ChatGPT outperforms many LLMs, it still falls behind task-specific finetuned models with a gap of 12.19 SPARROW score. Our benchmark is available at: https://github.com/UBC-NLP/SPARROW

## introduction multilingual llms have recently transformed nlp, due to their powerful capabilities on a ⋆ equal contribution @xcite , utcc @xcite , nai-jasenti @xcite , afrisenti @xcite , sentieval @xcite . to facilitate evaluation of llms and enhance fairness of model comparisons and reproducibility, early work introduces evaluation benchmarks. most existing benchmarks, however, focus on the monolingual setting. these include glue @xcite , @xcite , and tweeteval @xcite for english, arlue @xcite for arabic, clue @xcite for chinese, and indonlu @xcite for indonesian. although xtreme @xcite and xglue @xcite introduce multilingual benchmarks, they only include a few sm tasks for a limited number of languages. they are also limited to standard language use (e.g., wikipedia). @xcite propose a multilingual sentiment analysis benchmark (umsab), but it solely contains tweet sentiment analysis datasets in only eight languages. as such, absence of a unified, diverse, and comprehensive benchmark and a fragmented evaluation landscape hamper nlp work for cross-lingual sm. another challenge for sm research is the issue of data inaccessibility @xcite . although many studies release the ids of posts (e.g., tweets), substantial amounts of these social posts become inaccessible over time due to deletion, etc. @xcite . in our benchmark, we attempt to re-collect text contents of 25 datasets by using their ids but can only retrieve 58% samples on average (see to summarize, the contributions of this paper are as follows: (1) we collect, uniformize, and responsibly release massively multilingual sm datasets in a new benchamark; (2) our spar-row benchmark is essentially an archive of sm datasets that alleviates the serious issue of data decay; ## sparrow benchmark in this section, we describe clusters of tasks in our benchmark as well as our preprocessing and unification. sparrow consists of 13 types of tasks in six main categories. it contains 169 datasets from diverse online platforms and covers a wide period of time . we group different tasks in our benchmark by what we perceive to be an affinity between these tasks. for example, we group tasks of hate speech, offensive language, and dangerous language detection as anti-social language detection. meanwhile, we keep particular tasks (such as sentiment analysis and emotion recognition) distinct due to the popularity of these tasks and since there are multiple datasets representing each of them. ## public leaderboard to facilitate future work, we design a public leaderboard for scoring models on sparrow. our leaderboard is interactive and offers rich metadata about the various datasets in our benchmark. it also encourages users to submit information about their models (e.g., number of parameters, time to convergence, pretraining datasets). we also distribute a new modular toolkit for finetuning or evaluating models on sparrow. ## conclusion in order to understand the abilities of chatgpt and other instruction tuned llms on capturing sociopragmatic meaning, we introduced a massively multilingual evaluation benchmark, dubbed sparrow. the benchmark involves 169 datasets covering 64 languages from 12 language families and 16 scripts. evaluating chatgpt on spar-row, we find it struggles with different languages. we also reveal that task-specific models finetuned on sm (much smaller than chatgpt) consistently outperform larger models by a significant margin even on english.