Modeling the Severity of Complaints in Social Media (2021)

Abstract: The speech act of complaining is used by humans to communicate a negative mismatch between reality and expectations as a reaction to an unfavorable situation. Linguistic theory of pragmatics categorizes complaints into various severity levels based on the face-threat that the complainer is willing to undertake. This is particularly useful for understanding the intent of complainers and how humans develop suitable apology strategies. In this paper, we study the severity level of complaints for the first time in computational linguistics. To facilitate this, we enrich a publicly available data set of complaints with four severity categories and train different transformer-based networks combined with linguistic information achieving 55.7 macro F1. We also jointly model binary complaint classification and complaint severity in a multi-task setting achieving new state-of-the-art results on binary complaint detection reaching up to 88.2 macro F1. Finally, we present a qualitative analysis of the behavior of our models in predicting complaint severity levels.

## introduction complaining is a speech act that usually conveys negative emotions triggered by a discrepancy between reality and expectations towards an entity or event @xcite . complaints play an important role in human communication for expressing dissatisfaction. based on complainers' personalities and specific situations, expression of complaints vary from person to person @xcite . in pragmatics, complaints have been classified into various levels of severity according to their emotional intensity, the amount of face-threat that the complainer is willing to undertake and their purpose @xcite @xcite . complaining purposes might include the expression of dissatisfaction, to find solutions (e.g. ask for reparations) or both. furthermore, a complaint can be categorized as implicit (i.e. without mentioning who is responsible) or explicit (i.e. accusing someone for doing something). recent work on modeling complaints in natural language processing (nlp) has focused on distinguishing complaints from non-complaints in social media @xcite , however there is no previous study into more fine-grained complaint categories. in this paper, we present a systematic study on analyzing complaint categories with computational methods for the first time in computational linguistics. our main contributions are as follows: label example no explicit reproach are you following me? i seem unable to send you a dm. disapproval so far , the mac graphics drivers have been another disappointing update (for both my quadro 4000 & gtx -285), accusation can u stop adding the uk keyboard layout to my italian keyboard at every update? ktnxby blame thanks @xmath1, when the ati card arrives. moreover, the difference between very direct and somewhat direct is that the former highlights the responsibility of the complaint receiver while the latter does not. ## task & data we define complaint severity prediction as a multiclass classification task. given a text snippet t , defined as a sequence of tokens @xmath2, ..., t n }, the aim is to classify t as one of the four predefined severity labels. we use an existing complaints data set developed by @xcite , which consists of 1,235 complaints (35.8%) and 2,214 noncomplaints (64.2%) in english. we opted using this data set because it is publicly available with annotated complaints collected from twitter in 9 general domains (i.e. food, apparel, retail, cars, service, software, transport, electronics and other). ## predictive models since severity complaint prediction is a new task, we first evaluate the majority class as well as three strong baselines: (1) logistic regression with bagof-words; (2) a bidirectional recurrent neural network trained from scratch; and (3) finetuning a pretrained transformer-based model. furthermore, we combine linguistic information (i.e. emotion and topic information) to a transformer-based model similar to the method proposed by @xcite in the context of binary complaint classification. ## results overall, m-roberta models with linguistic features achieve best results. m-roberta emo outperforms all other models and reaches macro f1 up to 55.7. this confirms out hypothesis that injecting extra emotion information helps improve the performance of complaint severity level prediction. this is also in line with @xcite who states that the expression of complaints is relevant to different emotional states. the results of m-roberta t op and m-roberta emo+t op are comparable with 55.2 and 55.5 macro f1 respectively. roberta performs competitively but worse than the m-roberta models. we also notice that bigru-att does not perform well in our task (43.5 macro f1), which may result from the fact that it has not been pretrained. figure 1 presents the confusion matrix of our best model (i.e. m-roberta emo ). the confusion matrix is normalized over the actual values (rows). the 'no explicit reproach' category has the highest percentage (77.2%) of correctly classified data points by the model, followed by label 'disapproval' with 59.0%. these are also the two most frequent classes in the data set. on the other hand, results on 'accusation' are the lowest (32.9%) which is confused with adjacent categories ('disapproval' and 'blame'). furthermore, the differences between mis-classifications and correct classification are relatively large for 'blame'. we speculate that this is because of the unique linguistic characteristic of the 'blame' category which gives emphasis on someone's responsibility. finally, a category is more likely, in general, to be mis-classified to its adjacent severity categories. for example, when predicting 'disapproval', the number of model mis-classifications as 'no explicit reproach' and 'accusation' is larger than 'blame'. this hints that tweets belonging to neighboring levels share more semantic, syntactic and stylistic similarities. we also compare the performance of our best model (i.e. m-roberta emo ) with human agreement for each class (figure 2 ). in general, the results of the model (shown in figure 1 ) correlate to human agreement. in other words, the model and humans agree in the categories they confuse. for instance, it is easy for both of them to confuse 'accusation' with 'disapproval' (32.9% vs. 31.1% for the model and 43.6% vs. 31.% for humans). however, we observe that annotators are better at distinguishing high severity complaints from 'no explicit reproach', where 21.2% 'disapproval' and 12.4% 'accusation' are wrongly classified as 'no explicit reproach' by the model while the corresponding values are 18.5% and 8.9% by humans respectively. we argue that this is because annotators are able to identify subtle language (more details will be discussed in section 7). also, we notice that the model achieves better performance when predicting 'blame', indicating a better capability on capturing the main characteristics of this class compared to humans. ## error analysis we perform an error analysis to shed light on the limitations of our best performing model (m-roberta emo ) on complaint severity level classification. firstly, we observe that most errors happen when the differences of tweets belonging to 'accusation' are blurred with 'disapproval' and 'blame'. the following two tweets are typical examples for 'accusation' being mis-classified as 'disapproval' and 'blame' respectively: this is because some tweets belonging to 'accusation' also contain negation (e.g. 'not at all') or negative terms (e.g. 'disappointed'), which appear frequently in 'disapproval'. also, consistent with the definition by @xcite (directly or indirectly accuses someone for causing the problem), tweets belonging to 'accusation' may involve doing something and contain terms like '<@xmath3, which is similar to complaints labeled as 'blame' such as: secondly, the model struggles with complaints expressed in more subtle ways. in the following two examples, tweets belonging to 'disapproval' and 'accusation' are mis-classified as 'no explicit reproach' respectively: think someone @xmath4. please @xmath5. such complaints do not contain terms that are typical of any specific complaint severity category (e.g. negation and negative terms in 'disapproval', person pronouns and terms describing undesirable results in 'blame') thus predicting them correctly needs more contextual understanding. finally, compared to other categories, the model is more likely to confuse tweets belonging to 'no explicit reproach' and 'disapproval'. this happens because some tweets express weak dissatisfaction, which is difficult to identify. the following tweet is mis-classified as 'no explicit reproach': @xmath6: there really needs to be an easier method to report names that are @xmath7 the model might need to learn more contextual information about such tweets instead of capturing certain relevant terms. also, these two labels contain more similar terms such as 'dm', 'please help', 'can't work' and interrogative tone. examples of a 'no explicit reproach' and 'disapproval' are the following (where similarities are in bold): hey guys, i love this product featured @xmath8. @xmath9 ## multi-task learning for binary complaint prediction we further experiment with multi-task learning (mtl) @xcite for using severity categories to improve binary complaint prediction (i.e. complaint or non-complaint). mtl enables two or more tasks to be learned jointly by sharing information and parameters of a model. we explore whether or not the severity level of a complaint helps in complaint identification. we use the same data set as @xcite , where each tweet is annotated as a complaint or not and our severity level annotations. foot_6