Revisiting Supervised Contrastive Learning for Microblog Classification (2024)

Abstract: Microblog content (e.g., Tweets) is noisy due to its informal use of language and its lack of contextual information within each post. To tackle these challenges, state-of-the-art microblog classification models rely on pre-training language models (LMs). However, pre-training dedicated LMs is resource-intensive and not suitable for small labs. Supervised contrastive learning (SCL) has shown its effectiveness with small, available resources. In this work, we examine the effectiveness of fine-tuning transformer-based language models, regularized with a SCL loss for English microblog classification. Despite its simplicity, the evaluation on two English microblog classification benchmarks (TweetEval and Tweet Topic Classification) shows an improvement over baseline models. The result shows that, across all subtasks, our proposed method has a performance gain of up to 11.9 percentage points. All our models are open source.

## introduction microblog classification is a text classification task on microblog content (e.g., tweets). stateof-the-art microblog classification models rely on pre-training domain-specific transformer-based language models (lms), such as bertweet @xcite , xlm-t @xcite and timelms @xcite . in comparison, large language models (llms) such as chatgpt and gpt-4 fall short of this task @xcite . however, pre-training lms requires large computational resources, which is not feasible for small labs. an affordable alternative is to fine-tune a base pre-trained lm, such as roberta @xcite . in this work, we focus on the fine-tuning approach. typically, microblog content is noisy. first, the informal use of language introduces a large volume of incorrect grammar or typos. second, social figure 1 : an example of how supervised contrastive learning utilizes label information to form better representation on a hyper-sphere. the orange circle with the red edge represents an ambiguous sentence whose representation can be improved with scl. media posts are mostly short in length. due to the character limit, microblog content often lacks contextual information @xcite , which inherently increases the difficulty for the model to learn a good representation of the data. we hence investigate the use of supervised contrastive learning (scl) @xcite for microblog classification. we suggest that scl helps improve the learnt representation of models and performance on microblog classification tasks. this is because scl utilizes label information to enhance the intra-class concentration of features @xcite . figure 1 depicts a common phenomenon in microblog classification, where the model fails to represent an ambiguous sentence (circle with the red edge) in the embedding space. models trained with a scl loss explicitly pull the ambiguous sentence closer to the region where semantically similar sentences are located. therefore features of the same label are more concentrated in the embedding space. the orange arrow represents the "pulling" effect of scl's learning objective. overall, we emphasize the importance of finetuning small models in the landscape of the already scaled up computing resources (e.g., training llms and pre-training lms). the scope is thus to adopt scl for fine-tuning, which can achieve comparative or better downstream performance and it is more sample-efficient and more effective than prior approaches in the microblog domain. our contributions are: ## related work we provide two lines of literature that are related to our work: microblog classification and contrastive learning in nlp. ## method to examine the effectiveness of scl for microblog classification, we train a transformer-based sequence classifier in a supervised learning setting. the learning objective is to minimize a linear combination of a scl loss and a ce loss. ## conclusion with the observation that user-generated microblog content contains a large volume of noise that is inherent in the dataset, we develop a generic yet simple microblog classification fine-tuning framework with a scl-based regularizer in the training objective. our framework improves the baseline variant that is fine-tuned with only a cross-entropy loss by large margins across all tasks on the tweeteval and tweet topic classification benchmarks. on tweet topic classification, our model also surpassed the state-of-the-art models which are pre-trained on microblog-related corpora. the ablation study in appendix a.2 in shows the importance of utilizing label information for the scl regularizer. by qualitatively evaluating the model's prediction, we have identified two types of commonly made errors in appendix a.3.