S uper T weet E val: A Challenging, Unified and Heterogeneous Benchmark for Social Media NLP Research (2023)

Abstract: Despite its relevance, the maturity of NLP for social media pales in comparison with general-purpose models, metrics and benchmarks. This fragmented landscape makes it hard for the community to know, for instance, given a task, which is the best performing model and how it compares with others. To alleviate this issue, we introduce a unified benchmark for NLP evaluation in social media, SuperTweetEval, which includes a heterogeneous set of tasks and datasets combined, adapted and constructed from scratch. We benchmarked the performance of a wide range of models on SuperTweetEval and our results suggest that, despite the recent advances in language modelling, social media remains challenging.

## introduction there is overwhelming evidence that generalpurpose nlp systems suffer from a significant drop in performance when exposed to tasks in specialised domains. this has been shown in disparate domains such as the legal, medical and financial, to name a few. specifically, for these domains, specialised language models (lms) such as legal-bert, scibert, pubmedbert, or bloomberggpt @xcite @xcite have shown to exhibit lower perplexity and higher downstream performance across the board. the need for specialised lms and corresponding evaluation benchmarks is exacerbated in social media, where instead of (or rather, in addition to) a specialised domain, an nlp system has to deal with idiosyncrasies such as emoji @xcite @xcite , poor capitalisation @xcite , vulgarisms and colloquialisms @xcite , fast language change @xcite , and dynamic and platform-specific communicative structures @xcite . therefore, unsurprisingly, a strand of twitterspecific nlp research has produced what we would consider now de-facto models and datasets. on one hand, specialized lms, either pre-trained on multilingual twitter text alone @xcite @xcite , or including social engagements @xcite ; and on the other, joint twitter-specific models and datasets such as tweeteval @xcite . however, one area where social media nlp research seems to be lacking behind is in matching with appropriate resources the current shift towards in-context learning (icl) that large lms (llms) enable @xcite . benchmarks such as tweeteval, while helpful, are constrained to tweet classification tasks, crucially neglecting sequence tagging, generation and question answering (qa), for instance, which not only are more diverse, but better test beds for llms and comparing fine-tuning vs icl @xcite . the contributions of this paper can be summarized as follows. first, we introduce the super-tweeteval benchmark 1 . supertweeteval fills an important gap in the current nlp landscape by unifying diverse social media tasks and datasets beyond tweet classification (e.g., ner, question answering or tweet similarity) into one single benchmark, which we argue will contribute to faster and more replicable experiments on twitter. our second contribution is a suite of experiments that serve two purposes. first, to establish baselines using fine-tuned and icl approaches, and second, for analysing and producing insights from our experimental results, namely that overall better performance is obtained with fine-tuned masked language models when compared to equally sized text generation architectures, and that zero and few-shot approaches generally struggle as they are not eas-ily to adapt for certain social media tasks. in sum, our results show that supertweeteval is still challenging for general-purpose language models, regardless of their size and domain, and that specialised models are instead more competitive. ## related work the advent of general-purpose nlp systems (namely lms) has led to a proliferation of unified benchmarks agglutinating diverse nlp tasks. the general language understanding evaluation benchmark @xcite was one of the first efforts to provide a large-scale benchmark composed of diverse tasks such as natural language inference or textual similarity. glue was composed of relatively simple and homogeneous tasks, and saw automatic systems quickly reach human performance in some cases @xcite . because of this, superglue @xcite was developed with the same spirit but included a wider range of tasks and settings. since then, other general-purpose benchmarks for language models, especially those of the new generation, have emerged, such as mmmu @xcite and big-bench @xcite . in terms of social media research, there are many tasks that require modelling textual content. tweet-eval @xcite was the first unified benchmark that agglutinated different tasks into the same benchmark. however, tweeteval is limited to tweet classification, including emotion recognition @xcite , emoji prediction @xcite , irony detection @xcite , hate speech detection @xcite , offensive language identification @xcite , sentiment analysis @xcite , and stance detection @xcite . similar to the evolution of glue into superglue, the aim of this paper and supertweeteval is to construct a benchmark that is robust, large, and especially consisting of diverse tasks and settings for social media nlp research, and in particular, twitter. ## supertweeteval: the unified benchmark we convert all datasets presented in the previous section to the same json format, unifying them with the same notation, preprocessing and style. ## experimental setting for the evaluation, we rely on the datasets and splits presented in section 4.3. in particular, we evaluate all models on the test splits. each dataset uses a different evaluation metric, as introduced in section 4.2. ## results fine-tuning the results from the fine-tuning setting ( finally, regarding the models' architectures tested, our results indicate that the roberta models (and specifically timelm large ) display a better performance across most tasks when compared to flant5 and opt counterparts. zero & few shot when considering the results of our zero/few shot experiments ( the difference in performance, particularly in the regression and ordinal classification tasks of tweetintimacy and tweetsentiment, is significant. the best performing model, flant5 xxl , achieves, in a few-shot setting, scores of 29.96 and 25.72 in tweetintimacy and tweetsenti-ment respectively, which is more than a 50% drop in performance compared to the scores achieved by the best performing fine-tuned model (68.95 and 54.65). 12 ## analysis aiming to acquire a better understanding of capabilities for the model, and also the challenges that the tasks present, we organise the tasks in smaller subbenchmarks or clusters that are aimed at testing a particular feature, and investigate their performance. the clusters defined are created based on the nature of each task as well as the features that are present in them. temporal 13 . for this cluster, all the datasets that feature a temporal aspect are grouped together. in particular, we include those datasets 12 the goal of this paper is not to have the strongest models, but rather to evaluate models out-of-the-box. as such, there are tasks such as tweetemoji100 that are not easily solved by the selected zero/few shot models. 13 due to the lack of zero/few-shot results for tweetner7, we added the subset temporal* that does not include ner. ## conclusion in this paper, we have presented a new social media nlp benchmark, supertweeteval. it goes beyond simple tweet classification, including gen-erative, sequence prediction, and regression tasks, in addition to challenging classification tasks. this benchmark is aimed at unifying twitter-based evaluation protocols and provides a realistic assessment of models in this difficult and important domain. our evaluation highlighted the challenging nature of the benchmark and the social media domain. in particular, the results show how recent llms struggle with the specialised nature of this domain, with smaller but fine-tuned and more specialised models being more competitive overall. in addition to its evaluation purposes, it can also be used as the basis for multi-task learning, as well as for making use of already-trained models.