T weet E val: Unified Benchmark and Comparative Evaluation for Tweet Classification (2020)

Abstract: The experimental landscape in natural language processing for social media is too fragmented. Each year, new shared tasks and datasets are proposed, ranging from classics like sentiment analysis to irony detection or emoji prediction. Therefore, it is unclear what the current state of the art is, as there is no standardized evaluation protocol, neither a strong set of baselines trained on such domain-specific data. In this paper, we propose a new evaluation framework (TweetEval) consisting of seven heterogeneous Twitter-specific classification tasks. We also provide a strong set of baselines as starting point, and compare different language modeling pre-training strategies. Our initial experiments show the effectiveness of starting off with existing pre-trained generic language models, and continue training them on Twitter corpora.

## introduction modern nlp systems are typically ill-equipped when applied to noisy user-generated text. the high-paced, conversational and idiosyncratic nature of social media, paired with platform-specific restrictions (e.g., twitter's character limit), requires tackling additional challenges, for example, pos tagging @xcite , lexical normalization @xcite , or named entity recognition @xcite . in other more generic contexts, these challenges can be considered solved or are simply non-existent. moreover, other apparently simple tasks such as sentiment analysis have proven to be hard on twitter data @xcite , among others, due to limited amount of contextual cues available in short texts @xcite . in addition to these and other inherent difficulties, advances in nlp for user-generated data are hindered by its highly fragmented landscape and the lack of a unified evaluation framework. in the current era of pretraining and language models (lms), this is particularly relevant, as these models exhibit a versatility that currently cannot be gauged comparably across twitter datasets and tasks. this is not the case, however, in more ordinary textual genres and domains. for instance, well known benchmarks like senteval @xcite , glue @xcite and superglue @xcite include standard nlp tasks such as language inference, paraphrase detection or sentiment analysis, among others. it is undisputable that these benchmarks have contributed to the fast development of language understanding techniques, and lms in particular, as they have enabled comprehensive evaluations across several tasks in fair and reproducible conditions. we thus take inspiration from the above to develop tweeteval, a benchmark for tweet classification in english. tweeteval is a standardized test bed for seven tweet classification tasks. these are: sentiment analysis, emotion recognition, offensive language detection, hate speech detection, stance prediction, emoji prediction, and irony detection. we develop a unified framework, unified criteria for train/validation/test splits, and evaluate strong baselines inspired by current sota in these tasks. we also evaluate transformer-based models, trained entirely and partially on twitter data, with which we aim to establish a competitive high bar for subsequent contributions. the contributions of this paper are therefore as follows: (1) we compile, curate and release a suite of tasks under the umbrella of a new benchmark: tweeteval foot_0 , a unified framework comprising several tweet classification tasks; and (2) we evaluate state-of-theart lms in this new framework, and shed light on the effect of training with different corpora. ## language models for tweet classification transformer-based lms such as gpt @xcite , bert @xcite or xl-net @xcite have taken the nlp field by storm, outperforming previous linear models and neural network methods based on lstms or cnns in many tasks, including sentence and text classification @xcite . the functioning of these language models for tweet classification is conceptually simple. first, they are trained on a large unlabeled corpus. then, they are fine-tuned to the task for where an appropriate training set exists. for social media text, however, one may question whether existing pretrained models trained on standard corpora are optimal. we thus compare three different strategies 4 evaluation ## conclusion we have presented tweeteval, a unified benchmark for tweet classification consisting of seven heterogeneous tasks that are core to social media nlp research. along with the benchmark, we have included strong baselines as reference, and ran an analysis of lms with different training strategies. our results suggest that using a pre-trained lm may be sufficient, but can improve if topped with extra-training on in-domain data. for this initial benchmark and in the interest of reproducibility and accessibility, we focused on a fixed setting (i.e. classification). however, we acknowledge that other important tasks may need to be evaluated differently. thus, for future work we would like to include more tasks in the context of social media nlp research. potential improvements include, for example, accounting for the original multi-label nature of emotion classification, or covering more than only 20 emoji in emoji prediction. there are also other scenarios to be addressed as well, like sequence tagging @xcite , multimodality @xcite , and codeswitching tasks @xcite . this is similar to the evolution of glue @xcite into superglue @xcite , with both benchmarks contributing to the development of the field in different ways. it is also important to highlight that these datasets do not represent their underlying tasks as a whole but only a subsample, and therefore contain biases -automatic models trained on them might not be able to generalize to other specific settings @xcite . finally, this benchmark could foster research in multitask learning. the fact that several similar tasks co-exist (e.g. sentiment analysis and emotion detection, or hate speech detection and offensive language identification) can lead to interesting analyses where the similarity of these tasks is exploited.