Overview of the SIGHAN 2024 shared task for C hinese dimensional aspect-based sentiment analysis (2024)

Abstract: This paper describes the SIGHAN-2024 shared task for Chinese dimensional aspect-based sentiment analysis (ABSA), including task description, data preparation, performance metrics, and evaluation results. Compared to representing affective states as several discrete classes (i.e., sentiment polarity), the dimensional approach represents affective states as continuous numerical values (called sentiment intensity) in the valence-arousal space, providing more fine-grained affective states. Therefore, we organized a dimensional ABSA (shorted dimABSA) shared task, comprising three subtasks: 1) intensity prediction, 2) triplet extraction, and 3) quadruple extraction, receiving a total of 214 submissions from 61 registered participants during evaluation phase. A total of eleven teams provided selected submissions for each subtask and seven teams submitted technical reports for the subtasks. This shared task demonstrates current NLP techniques for dealing with Chinese dimensional ABSA. All data sets with gold standards and evaluation scripts used in this shared task are publicly available for future research.

## introduction aspect-based sentiment analysis (absa) @xcite @xcite is a critical nlp research topic that aims to identify the aspects of a given entity and analyze the sentiment polarity associated with each aspect. in recent years, considerable research has been devoted to absa, which can be categorized into different tasks based on the number of sentiment elements to be extracted. for example, the aspect sentiment triplet extraction (aste) task @xcite @xcite @xcite @xcite extracts three elements in a triplet, including aspect/target term, opinion term and sentiment polarity (e.g., positive, neutral, and negative). furthermore, the aspect sentiment quadruple prediction (asqp) task @xcite @xcite @xcite extracts the same three elements plus an additional aspect category to construct a quadruple. however, compared to representing affective states as several discrete classes (i.e., sentiment polarity), the dimensional approach that represents affective states as continuous numerical values (called sentiment intensity) in multiple dimensions such as valence-arousal (va) space @xcite , providing more fine-grained emotional information @xcite @xcite . therefore, we organized a chinese dimensional absa shared task (dimabsa) in the 10 th sighan workshop on chinese language processing (sighan 2024), providing finegrained sentiment intensity prediction for each extracted aspect of a restaurant review. we have three subtasks: 1) intensity prediction, 2) triplet extraction, and 3) quadruple extraction. participants are free to participate in any or all subtasks. given a sentence with/without aspects, participating systems should be able to extract the sentiment elements with the corresponding valence-arousal rating values. the rest of this article is organized as follows. section 2 provides a description of the chinese dimensional absa shared task. section 3 introduces the evaluation data construction. section 4 describes the performance metrics. section 5 compares evaluation results from the various participating teams. finally, we conclude this paper with findings and offer future research directions in section 6. ## task organization this task aims to evaluate the capability of an automatic system for chinese dimensional absa. the four sentiment elements are defined as follows: ## data preparation we first crawled restaurant reviews from google reviews and an online bulletin board system ptt. then, we removed all html tags and multimedia material and split the remaining texts into several sentences. finally, we randomly selected partial sentences to retain content diversity for manual annotation. the annotation process was conducted in two phases. we first annotated the aspect/category/opinion elements and then v#a element. in the first phase, three graduate students majoring in computer science were trained to annotate the sentences for aspect/category/opinion. one task organizer led a discussion to clarify annotation differences and seek consensus among the annotators. a majority vote mechanism was finally used to resolve any disagreements among the annotators. in the second phase, each sentence along with the annotated aspect/category/opinion was presented to five annotators majoring in chinese language for v#a rating. similarly, one task organizer also led a group discussion during annotation. once the annotation process was finished, a cleanup procedure was performed to remove outlier values which did not fall within 1.5 standard deviations (sd) of the mean. these outliers were then excluded from calculating the average v#a for each instance. we provided two versions of all datasets with identical content, but one in traditional chinese characters and the other in simplified chinese characters. the participating teams could choose their preferred version for the task evaluation. the submitted results were evaluated with the corresponding version of the gold standard and ranked together as the official results. this shared task is presented as an open test, and participating systems can use other publicly available data, but such data must be specified in the final system description paper. for example, we also provide the chinese emobank ( lee we further analyzed the aspect types in the test set, including #unique and #repeat which respectively denote the number of aspects which occurred only one time or more than one time. for subtask 1, a total of 2,658 aspects belong to the unique type, without the null and repeat cases. for subtasks 2 and 3, 1821 aspects (51.1% out of total 3,566) occurred more than one time across all testing sentences. in addition, a very small portion (near 1.5%) of aspects belonged to the null cases. similarly, we also analyzed the opinion terms, the repeat cases occupied about 8.5% (=303/3566). these findings revealed: 1) the aspect has a centered distribution, reflecting that users' opinion targets may be similar, and 2) the opinion has a diverse distribution, indicating that different affective words or phrases are used to express a user's feelings. figure 2 presents the aspect category distributions. the distributions are imbalanced for both the training and test sets for subtask 3. this finding is the same as that for the semeval-2016 restaurant dataset @xcite . the most frequently occurring category was "食 物 # 品 質"(food#quality), followed by "食物#份量與款 式 "(food#style&options) and "飲 料#品 質" (drinks#quality). in the training set, these 3 categories accounted for 87.4% of the total, with the remaining 9 categories accounting for 12.6%. in the test set for subtask 3, these 3 categories accounted for 89.5% of the total, with the other 9 categories accounting for the remaining 10.5%. ## performance metrics for subtask 1, the sentiment intensity prediction performance is evaluated by examining the difference between machine-predicted ratings and human-annotated ratings using two metrics: mean absolute error (mae) and pearson correlation coefficient (pcc), defined as the following equations. ## conclusions and future work this paper provides an overview of the sighan-2024 dimabsa task for chinese dimensional aspect-based sentiment analysis, including task descriptions, data preparation, performance metrics and evaluation results. we received a total of 214 submissions from 61 registered participants during the evaluation phase. among eleven participating teams, seven presented their task technical reports. regardless of actual performance, all submissions contribute to the development of an effective dimensional absa solution, and each task technical paper for this shared task also provides useful insights for further research. we hope the data sets collected and annotated for this shared task can facilitate and expedite future development of chinese dimensional absa. therefore, the gold standard test set and evaluation scripts are made publicly available in github repositories at: https://github.com/nycunlp/sighan2024-dimabsa future directions will focus on the development of chinese dimensional absa models. we plan to build new language resources to develop techniques for the future enrichment of this research topic, especially for reviews in the other domains.