D ia ASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis (2023)

Abstract: The rapid development of aspect-based sentiment analysis (ABSA) within recent decades shows great potential for real-world society. The current ABSA works, however, are mostly limited to the scenario of a single text piece, leaving the study in dialogue contexts unexplored. To bridge the gap between fine-grained sentiment analysis and conversational opinion mining, in this work, we introduce a novel task of conversational aspect-based sentiment quadruple analysis, namely DiaASQ, aiming to detect the quadruple of target-aspect-opinion-sentiment in a dialogue. We manually construct a large-scale high-quality DiaASQ dataset in both Chinese and English languages. We deliberately develop a neural model to benchmark the task, which advances in effectively performing end-to-end quadruple prediction, and manages to incorporate rich dialogue-specific and discourse feature representations for better cross-utterance quadruple extraction. We hope the new benchmark will spur more advancements in the sentiment analysis community.

## introduction it is meaningful to empower machines to understand human opinion and sentiment, which motivates the study of sentiment analysis @xcite @xcite . absa is an important branch of sentiment analysis aiming to detect the sentiment trends towards the fine-grained aspects of targets, which has received consistent research attention within last few years @xcite @xcite @xcite . the initial absa revolves around the study of aspect terms and sentiment polarities @xcite @xcite 2019). later, the extraction of opinion terms is considered, resulting in a triplet analysis (i.e., aspectopinion-sentiment) of absa @xcite . the latest trend of absa has been upgraded into the quadruple form by adding the category element into the triplet absa @xcite . the quadruple absa promisingly completes the absa definition and helps the comprehensive understanding of the opinion picture. yet we notice that all the current absa research is confined to the scenario of a single piece of text (i.e., sentence or document). for example, currently the most popular absa benchmark, se-meval @xcite @xcite , comes with only sentence-level annotations. this may limit the application of absa. essentially, in the real-world environment absa has a broader application under dialogue contexts. for example, people are more likely to discuss certain products, services, or politics on social media (e.g., twitter, facebook, weibo) in the form of multi-turn and multi-party conversations. also, it is practically meaningful to develop sentiment-support dialog systems to facilitate the clinical diagnosis, and treatment @xcite . unfortunately, no effort has been dedicated to the research of a holistic dialog-level absa. in this paper, we consider filling the gap of dialogue-level absa. we follow the line of recent quadruple absa and present a task of conversational aspect-based sentiment quadruple analysis, namely diaasq. diaasq sets the goal to detect the fine-grained sentiment quadruple of targetaspect-opinion-sentiment given a conversation text, i.e., an opinion of sentiment polarity has been expressed toward the target with respect to the aspect. as exemplified in fig. 1 , multiple users (speakers) on social media discuss different angles of a product (i.e., 'xiaomi' brand cellphone) in dialogue threads of multiple turns. the task aims to extract three quadruples over the dialog: ('xiaomi 11', 'wifi module', 'bad design', 'negative'), ('xiaomi 11', 'battery life', 'not well', 'negative') and ('xiaomi 6', 'screen quality', 'very nice', 'positive'). to benchmark the task, we manually annotate a large-scale diaasq dataset. we collect millions of conversational corpus of source comments and discussions closely related to electronic products from chinese social media. we hire well-trained workers to explicitly label the diaasq data (i.e., the elements of quadruples, targets, aspects, opinions, and sentiments) based on the crowd-sourcing technique, which ensures a high quality of annotations. finally, we yield the dataset with 1,000 dialogue snippets in total with 7,452 utterances. to facilitate the multilinguality of the benchmark, we further translate and project the annotations into english. data statistics show that each dialog involves around 5 speakers, and 22.2% of the quadruples are in the cross-utterance format. compared with previous single-text-based absa, diaasq challenges in two main aspects. first, diaasq includes four subtasks. directly applying the existing best-performing graph-based absa model to enumerate all possible target, aspect, and opinion terms could cause a combinatorial explosion. second, the elements of a quadru- to solve these challenges, we present an end-toend diaasq framework. specifically, based on the grid-filling method @xcite , we redesign the tagging scheme to fulfill the four subtasks in one shot effectively. moreover, during the dialogue text encoding, we additionally model the dialogue-specific representations for utterance interaction and meanwhile encode the relative distance as cross-utterance features. experiments on the diaasq data indicate that our model shows significant superiority than several strong baselines. to sum up, this work contributes in threefold: ## data construction we construct a new dataset to facilitate the diaasq task. the raw corpus is collected from the largest chinese social media, weibo foot_1 . we crawl nine million posts and comments from the tweets history of 100 verified digital bloggers. each conversation is derived from a root post, and multiple users (i.e., multiple speakers) are attended to reply to a predecessor post. the multi-thread and multi-turn dialogue forms a tree structure, as illustrated in fig. 2 . we preprocess the raw dialogues to make the contexts integrated. first, we filter the topicrelated conversations by a manually created keyword dictionary in the mobile phone field, which includes hundreds of hot words, like phone band names, aspects words to describe a mobile phone, etc. then, we normalize the tweet language expressions (e.g., abusive language, hate speech) by human examination or consulting lexicons; we prune away those meaningless replying branches that deviate too much from the main topic. we also limit the maximum number of utterances to ten for better controllable modeling. after a strict cleaning procedure, we obtain the final 1,000 dialogues. during the annotation stage, all the conversation texts are labeled with a team of crowd-workers who are pre-trained under the semeval absa @xcite annotation guideline foot_2 . also, the linguistic and computer science experts inspect the labeling schema. after annotating, annotators are required to cross-examine the labels. also, some automatic rules are applied to verify the labeling consistency. finally, cohen's kappa score of quadruples is 0.86, which indicates our annotation corpus has reached a high-level agreement. data insights. we randomly split the conversation snippets into train/valid/test sets, in the ra- figure 3 : the ratio of cross-utterance quadruples. we define the max utterance-level distance between every two items in one quadruple as the number of crossutterance. for example, the first quadruple in fig. 1 crosses two utterances. diaasq naturally includes four subtasks. different popular end-to-end absa systems can be utilized to solve our diaasq, such as the graphbased @xcite , seqto-seq @xcite and grid-tagging models @xcite ). yet enumerating all possible terms with graph-based methods will cost computational efficiency, while seqto-seq methods suffer from exposure bias. the gridtagging method advances in higher efficiency, i.e., o(n 2 ) complexity, where n denotes the sequence length. however, the labeling scheme in @xcite only supports term-pair extraction (i.e., aspect and opinion terms), which fails to directly solve our diaasq that requires term-triple extraction (i.e., target, aspect and opinion terms). here we inherit the success of the gridtagging method for an end-to-end solution and redesign the labeling scheme to fit our needs. to reach the goal, we re-decompose the task into three joint jobs: detections of the entity boundary, entity pair, and sentiment polarity. we renew the labeling scheme of grid-tagging in support of these jobs, which is shown in fig. 4 . ## diaasq model we present a diaasq model to accomplish the task based on the above grid-tagging label scheme. fig. 5 shows the overall architecture. ## conclusion this work introduces a new task of conversational aspect-based sentiment quadruple analysis, namely diaasq, which aims to detect the sentiment quadruple of target-aspect-opinion-sentiment structure in the conversation texts. diaasq bridges the gap between conversational opinion mining and fine-grained sentiment analysis. we manually construct a large-scale, high-quality dataset with chinese and english versions for the task, with 1,000 dialogue snippets, including 7,452 utterances. we then benchmark the diaasq task with an end-toend neural model, which effectively models the dialogue utterance interactions. experiments demonstrate the advantages of our method in effectively learning the dialogue-specific features for better cross-utterance sentiment quadruple extraction.