E mo N o B a: A Dataset for Analyzing Fine-Grained Emotions on Noisy B angla Texts (2022)

Abstract: For low-resourced Bangla language, works on detecting emotions on textual data suffer from size and cross-domain adaptability. In our paper, we propose a manually annotated dataset of 22,698 Bangla public comments from social media sites covering 12 different domains such as Personal, Politics, and Health, labeled for 6 fine-grained emotion categories of the Junto Emotion Wheel. We invest efforts in the data preparation to 1) preserve the linguistic richness and 2) challenge any classification model. Our experiments to develop a benchmark classification system show that random baselines perform better than neural networks and pre-trained language models as hand-crafted features provide superior performance.

## introduction identifying emotions has helped find solutions to numerous problems for english text, namely retrieving emotion from suicide notes @xcite , detecting insulting sentences in conversations @xcite , and providing palliative care to cancer patients @xcite . a major reason behind such success is the amount of research and development invested towards fine-grained multilabel emotion tasks such as semeval affective texts @xcite , semeval affects of tweets @xcite and goemotion @xcite . bangla is the sixth most spoken language globally 2 and is the native language of bangladesh. with the country now graduating to a middleincome country with technologies now set to reach the rural areas @xcite , it is a timely need to understand the response to enhance the overall impact on social welfare and businesses. few datasets have been made public for detecting emotion in a low-resourced bangla language @xcite . however, all such works are (1) small in size, including only a few thousand instances, and (2) incapable of cross-domain generalization, with coarse classification into ekman or plutchik emotions. in this paper, we aim to create a multi-label emotion dataset of noisy textual data collected from social media on various topics. we use the junto emotion wheel @xcite (figure 1 ) as it reflects 21 st century human psychology. during the data collection and annotation process, we invest efforts to improve the quality of the dataset by setting several predefined objectives. we also curate the test set such that it challenges any classification tasks. our contributions can be summarized as follows: objective given a predefined set of emotions -junto-6 basic emotions, the goal is to identify all emotions conveyed in a piece of text. annotation we use five annotators for each instance. emotion(s) voted by atleast three annotators were considered the final labels. instances that could not be finalized this way were sent to authors for the final tag. we will refer to the former instances as geninst and the latter as excinst. we also kept the system fully anonymous for the authenticity of the annotations foot_2 . furthermore, we evaluated the annotators with an accuracy metric. we will denote such accuracy as annoaccu. we have two different variations of equations for determining annoaccu as we have two categories of instances: for excinst: we set the following criterion when choosing annotators. annotators must be 1) well educated to understand the instances despite grammatical and spelling errors, and 2) active social media users to understand the context. before selecting an emotion, we instructed them first to identify their child emotions from the junto emotion wheel for better coherence. as such, 80 undergraduate students annotated 5 to 5,000 instances each, with 74 of them attaining annoaccu of 60% or more. ## methodology in this section, we present the methods we used to develop a benchmark model for emonoba. ## further analysis dominant features love joy surprise েবস্ট েবস্ট (best best) মু গ্ধ (amazed) খু ব সু ন্দর লাকেছ (looks very nice) েকেনা? (why?) অসাধারণ (extraordinary) আপিন েবস্ট (you are best) . . . খু ব সু ন্দর লাকেছ (looks very nice) তু িম েসরা (you are best) আর িক (what more) anger sadness fear বােলর (slang) িবচার নাই েদেশ (there is no justice in the country) ভয় থাকেতা । (fear remained) েবিশ হেয় েগেছ (too much) বােজ ভােব উপস্থাপন (poorly introduce) আল্লাহ েহফাজত কর (god protect us) না িক (no what) কান্না (cry) ফাঁ িস (execution) তু িম খু ব খারাপ (you are really bad) েশা-অফ (show-off) েবেড় েগেলা। (increased.) error analysis to investigate the test errors, we present the distribution of emotion per topic and the models' performance in figures 4a and 4b . notice that the model additionally predicts sadness in joy and love instances in personal topic. the reason is negative words, such as "েশা-অফ" (showoff), are the strongest words of sad emotion ( ## conclusion in this paper, we present emonoba, a dataset for fine-grained emotion detection on bangla text collected from comment sections of social media platforms on 12 different domains. we found that hand-crafted features performed comprehensively better than neural models. as the future work, we will exploit the findings identified in this work while incorporating contextual understanding.