Peeking inside the black box: A Commonsense-aware Generative Framework for Explainable Complaint Detection (2023)

Abstract: Complaining is an illocutionary act in which the speaker communicates his/her dissatisfaction with a set of circumstances and holds the hearer (the complainee) answerable, directly or indirectly. Considering breakthroughs in machine learning approaches, the complaint detection task has piqued the interest of the natural language processing (NLP) community. Most of the earlier studies failed to justify their findings, necessitating the adoption of interpretable models that can explain the model’s output in real time. We introduce an explainable complaint dataset, X-CI, the first benchmark dataset for explainable complaint detection. Each instance in the X-CI dataset is annotated with five labels: complaint label, emotion label, polarity label, complaint severity level, and rationale (explainability), i.e., the causal span explaining the reason for the complaint/non-complaint label. We address the task of explainable complaint detection and propose a commonsense-aware unified generative framework by reframing the multitask problem as a text-to-text generation task. Our framework can predict the complaint cause, severity level, emotion, and polarity of the text in addition to detecting whether it is a complaint or not. We further establish the advantages of our proposed model on various evaluation metrics over the state-of-the-art models and other baselines when applied to the X-CI dataset in both full and few-shot settings.

## introduction complaining is an expression of negative emotions communicated due to a discrepancy between reality and expectations @xcite . in pragmatics theory, @xcite proposed four primary complaint severity levels: (a) no explicit reproach, (b) disapproval, (c) accusation, and (d) blame. recent studies on complaint detection have mainly dealt with automatically identifying binary complaints and the associated severity levels from social-media data @xcite . however, these studies primarily focused on improving complaint detection performance with the help of various models without providing any perspective or evaluation of the outcome's explainability. since the introduction of explainable artificial intelligence (ai) @xcite , providing interpretation for any ai algorithm's decision has become essential. therefore, rather than enhancing performance by increasing computational burden, there is a push to construct trustworthy and transparent interpretable models. as sentient beings, we use our commonsense to establish connections between what is explicitly said and inferred. as a result, we believe that adding external knowledge or commonsense knowledge @xcite to complaint detection systems can help them better grasp the user's circumstances and concerns, resulting in more efficient models. in figure 1 , the user shares information about a registered case with customer service. based on the given information, we can say the user is waiting for a callback (xneed) and wants to receive a callback for the registered case (xwant). to the best of our knowledge, all the previous attempts to incorporate commonsense reasoning have only been done for conversational agents and summarization tasks. previous research @xcite has shown that closely related tasks benefit each other when learned concurrently. however, this strategy entails several problems, such as negative transfer (where multiple tasks, rather than benefiting the learning process, begin to hinder the training process) @xcite and optimization scheme (assigning weights to different tasks during training) @xcite . to address the aforementioned multitask learning issues, and motivated by generative language models' ability to solve downstream tasks in a generative manner in both full and few-shot (low resource and data constrained) scenarios @xcite , we propose using text-to-text generation to accomplish the tasks of complaint identification, severity level classification, and cause extraction. research objectives: following are the research objectives of the current study: 1) we aim to understand the effect of using causal information on the complaint detection, and severity classification tasks in the proposed framework. the rest of the paper is organized as follows. section 2 discusses some of the previous studies on this subject. following that, in section 3, we explain the dataset development in detail. in section 4, we summarize our proposed methodology for the unified generative method-based experiments. we analyze the experiments, results, and their outcomes in section 5. finally, in section 6, we conclude our study and identify the scope of future work. ## related studies in computational linguistics, previous works on complaint detection only pivoted on identifying complaints using feature-based machine learning models @xcite , transformer networkbased models @xcite . recently multitask complaint analysis models have been developed that leveraged polarity and affect information for enhancing the complaint mining task @xcite @xcite . in affective computing, identifying the causal span of expressed emotions is crucial for understanding human emotions @xcite . motivated by previous research, we aim to investigate the reasons behind viewpoints, particularly complaints, expressed on social media providing cognitive awareness of the user's circumstances and sentiments is essential to designing effective systems for downstream tasks such as chatbots @xcite . hence, we believe that allowing complaint detection models to use commonsense information and draw conclusions from what the user has openly shared is particularly useful for better understanding the user's circumstances, resulting in more effective and socially aware customer support systems. recent breakthroughs in deep learning and pretrained language models have substantially affected the development in the field of neural text generation @xcite . models such as gpt-2 @xcite and gpt-3 @xcite are decoder-only transform-ers that can generate understandable, and consistent text because of being pre-trained on a vast amount of text data. bart @xcite and t5 @xcite are encoder-decoder transformers that have shown rapid evolution and success in many nlp applications, such as summarization and translation. after an in-depth literature review, it can be concluded that there is no work on explainable complaint detection. in this paper, we attempt to bridge this research gap. ## x-ci dataset development for this work, we utilize the complaints dataset foot_1 published in (preotiuc-pietro et al., 2019), which includes 3,449 tweet instances in english. we selected this dataset because it is openly available and comprises annotated complaints from the socialnetworking site twitter, a popular choice for data analysis. @xcite augmented complaints dataset with five severity levels (no explicit reproach, disapproval, accusation, blame, and noncomplaints). recently, @xcite enriched the complaints dataset with the sentiment (negative, neutral, positive) and emotion (anger, disgust, fear, happiness, sadness, surprise and other) classes; the 'other' emotion class depicts tweets that do not fall under the scope of ekman's six basic emotions @xcite . we utilize this extended dataset annotated with severity levels, polarity, and emotion classes for our current work. ## proposed methodology this section defines our problem before delving into the details of the proposed framework. the overall framework is shown in figure 2 . problem definition: an explainable complaint detection model should predict the review's cause, severity level, emotion, and polarity class in addition to detecting whether it is complaint or not. formally, given an input instance x @xmath0, x 1 , .., x i , .., x n } where n is the length of input instance, we intend to learn five closely related tasks: (i) complaint identification (c), (ii) polarity classification (p), (iii) emotion recognition (e), (iv) severity level classification (s), (v) cause extraction (ce), where @xmath1, c is the set of complaint classes, @xmath2, p is the set of polarity classes, @xmath3, e is the set of emotion classes, @xmath4, s is the set of severity levels and ce(x i ) ∈ (x i ) that is relevant to the complaint label, c. ## conclusion in this paper, the explainability factor has been considered while we try to tackle the complaint detection problem. as explainable ai systems increase trustworthiness and confidence when used in real-time, and complaint detection systems benefit different enterprises for robust customer support, generating rationale behind actions performed is a must. the current work makes two contributions: (a) developing the first explainable complaint detection dataset, which includes annotations of the rationale/causal span employed in decision-making (b) a commonsense-aware unified generative framework has been developed to perform five tasks (cd, sc, ce, pr, and er) simultaneously. in order to take advantage of the knowledge of sizable pretrained sequence-to-sequence models, this work demonstrates how a multitasking problem could be phrased as a text-to-text generation task. our proposed model outperforms all the baselines and the sota for the three main tasks based on extensive evaluation. in future, we will focus on expanding explainable complaint identification in the multimodal environment that considers image and text modalities.