BLP -2023 Task 2: Sentiment Analysis (2023)

Abstract: We present an overview of the BLP Sentiment Shared Task, organized as part of the inaugural BLP 2023 workshop, co-located with EMNLP 2023. The task is defined as the detection of sentiment in a given piece of social media text. This task attracted interest from 71 participants, among whom 29 and 30 teams submitted systems during the development and evaluation phases, respectively. In total, participants submitted 597 runs. However, only 15 teams submitted system description papers. The range of approaches in the submitted systems spans from classical machine learning models, fine-tuning pre-trained models, to leveraging Large Language Model (LLMs) in zero- and few-shot settings. In this paper, we provide a detailed account of the task setup, including dataset development and evaluation setup. Additionally, we provide a succinct overview of the systems submitted by the participants. All datasets and evaluation scripts from the shared task have been made publicly available for the research community, to foster further research in this domain.

## introduction sentiment analysis has emerged as a significant sub-field in natural language processing (nlp), with a wide array of applications encompassing social media monitoring, brand reputation management, market research, customer feedback analysis, among others. the advancement of sentiment analysis systems has been driven by substantial research efforts, addressing its indispensable utility across diverse fields such as business, finance, politics, education, and services @xcite . traditionally, analysis has been conducted across various types of content and domains including news articles, blog posts, customer reviews, and social media posts, and extended over different modali-ties like textual and multimodal analyses @xcite . at its core, the task of sentiment analysis is defined as the extraction and identification of polarities (e.g., positive, neutral, and negative) expressed within texts. however, its scope has broadened to encompass the identification of: (i) the target (i.e., an entity) or aspect of the entity on which sentiment is expressed, (ii) the opinion holder, and (iii) the time at which it is expressed @xcite . such advancements have primarily been made for high-resource languages. research on fundamental sentiment analysis remains an ongoing exploration, especially for many low-resource languages, primarily due to the scarcity of datasets and consolidated community effort. although there has been a recent surge in interest @xcite @xcite , the field continues to pose significant challenges. similar to other lowresource languages, the challenges for sentiment analysis in bangla have been reported in recent studies @xcite @xcite . @xcite emphasized the primary challenges associated with bangla sentiment analysis, specifically issues of duplicate instances in the data, inadequate reporting of annotation agreement, and generalization. these challenges were also highlighted in @xcite , further emphasizing the need to address them for effective sentiment analysis in bangla. to advance research in bangla sentiment analysis, we emphasized community engagement and organized a shared task at blp 2023. similar efforts have primarily been conducted for other languages as part of the semeval workshop. the analysis of sentiment in tweets serves as an example of such efforts, particularly focusing on arabic and english @xcite . an earlier attempt at such an endeavor for bangla is reported in @xcite , which mainly focused on tweets. our initiative significantly different from theirs in terms of datasets (e.g., data from multiple social media platforms and diverse domains) and evaluation setup. a total of 71 teams registered for the task, out of which 30 made an official submission on the test set, and 15 of the participating teams submitted a system description paper. the remainder of the paper is structured as follows: section 2 provides an overview of the relevant literature. section 3 discusses the task and dataset. section 4 describes the organization of the task and the evaluation measures. an overview of the participating systems is provided in section 5. lastly, section 6 concludes the paper. ## related work the current state-of-the-art research for bangla sentiment classification mainly dominated focuses on two key aspects: the development or datasets and model development. notable recent work in this direction include @xcite @xcite @xcite . @xcite curated the largest dataset from book reviews, with annotations based on the review ratings. although the dataset encompasses a large number of reviews, the class distribution poses a challenge for the negative and neutral classes. a well-balanced dataset has been explored in @xcite , comprising ∼15k manually annotated comments spanning 13 different domains. this dataset is also used as a part of this shared task. from a modeling perspective, the existing literature addresses the problem using both classical machine learning and deep learning algorithms. these include naive bayes, support vector machine, decision tree, maximum entropy, and random forest @xcite @xcite . moreover, recent studies have extensively employed deep learning models for bangla sentiment classification @xcite @xcite @xcite @xcite . common deep learning approaches incorporate lstms, cnns, attention mechanisms, and multichannel convolutional lstms. in the studies by @xcite ; @xcite , comprehensive comparisons across various datasets were conducted, illustrating that the deep learning-based pretrained language model xlm-roberta excels in performance. comparisons between classical and deep learning-based approaches have also been explored @xcite @xcite . given the significant capabilities that large language models (llms) have demonstrated across diverse applications and scenarios, @xcite explored various llms such as flan-t5 (large and xl) @xcite , bloomz (1.7b, 3b, 7.1b, 176b-8bit) @xcite , and gpt-4 (openai, 2023), comparing the results with fine-tuned models. the resulting performance demonstrate that fine-tuned models continue to outperform zero-and few-shot prompting. however, the performance of llms showcases a promising direction towards the development of systems with limited datasets for new domains. though there is a surge of research interest and progress, utilizing such systems in real applications remains a challenge in terms of performance and generalization capability. this shared task aimed to advance research through community effort and focus on a standard evaluation setup. as a starting point, we aimed to classify sentiment into three sentiment polarities: positive, neutral, and negative. this approach can be further extended in future studies. ## conclusion and future work we presented an overview of the shared task 2 (sentiment analysis) at the blp workshop 2023. task 2 aimed to classify the sentiment in textual content. notable systems employed an ensemble of pretrained language models, with the languagespecific banglabert being the most popular. also, some interesting approaches including p-tuning, few-shot learning, llms, and different loss functions have been explored for tackling the problem. in general, numerous models, including different kinds of transformers, have been used in the current submissions for the task. in future work, we plan to extend the task in various ways, such as aspect-based sentiment analysis and incorporating multiple modalities.