Complaint Identification in Social Media with Transformer Networks (2020)

Abstract: Complaining is a speech act extensively used by humans to communicate a negative inconsistency between reality and expectations. Previous work on automatically identifying complaints in social media has focused on using feature-based and task-specific neural network models. Adapting state-of-the-art pre-trained neural language models and their combinations with other linguistic information from topics or sentiment for complaint prediction has yet to be explored. In this paper, we evaluate a battery of neural models underpinned by transformer networks which we subsequently combine with linguistic information. Experiments on a publicly available data set of complaints demonstrate that our models outperform previous state-of-the-art methods by a large margin achieving a macro F1 up to 87.

## introduction complaining is a basic speech act, usually triggered by a discrepancy between reality and expectations towards an entity or event @xcite @xcite . social media has become a popular platform for expressing complaints online @xcite where customers can directly address companies regarding issues with services and products. complaint detection aims to identify a breach of expectations in a given text snippet. however, the use of implicit and ironic expressions and accompaniment of other speech acts such as suggestions, criticism, warnings and threats @xcite make it a challenging task. identifying and classifying complaints automatically is important for: (a) improving customer service chatbots (coussement and @xcite @xcite linguists to analyze complaint characteristics on large scale @xcite ; and (c) psychologists to understand the behavior of humans that express complaints @xcite . previous work has focused on binary classification between complaints and non-complaints in various domains @xcite @xcite . furthermore, some studies have performed more fine-grained complaint classification. for instance, complaints directed to public authorities have been categorized based on their topics @xcite or the responsible departments @xcite @xcite . other categorizations are based on possible hazards and risks (bhat and culotta, 2017) as well as escalation likelihood @xcite . most of these previous studies have used supervised machine learning models with features extracted from text (e.g. bag-of-words, topics, features extracted from psycho-linguistic dictionaries) or task-specific neural models trained from scratch. adapting state-of-the-art pre-trained neural language models based on transformer networks @xcite such as bert @xcite and xlnet @xcite ) has yet to be explored. in this paper, we focus on the binary classification of twitter posts into complaints or not (2019). we adapt and evaluate a battery of pre-trained transformers which we subsequently combine with external linguistic information from topics and emotions. ## complaint prediction task and data given a text snippet (i.e. tweet), we aim to classify it as a complaint or not. for that purpose, we use the data set foot_0 by @xcite which contains tweets written in english that were manually annotated as complaints or not. it includes 1,232 complaints (62.4%) and 739 non-complaints (37.6%) over 9 domains (e.g. food, technology, etc.). data statistics are shown in ## transformer-based models transformer architectures trained on language modeling have been recently adapted to downstream tasks demonstrating state-of-the-art performance @xcite @xcite . in this paper, we adapt and subsequently combine transformers with external linguistic information for complaint prediction. bert, albert and roberta bidirectional encoder representations from transformers (bert) @xcite learns language representations by jointly conditioning on both left and right contexts using transformers. it is trained on masked language modeling where some of the tokens are randomly masked with the aim to predict them using only the context. we further experiment with albert @xcite and roberta @xcite . albert uses two parameter-reduction methods to address memory limitations and long training time of bert: (a) factorized embedding parameterization; (b) cross-layer parameter sharing. roberta is an extension of bert trained on more data with larger batch size using dynamic masking (i.e. changeable masked tokens of each sequence during training epochs). we adapt bert, albert and roberta by adding a linear layer with a sigmoid activation and then fine-tune it on the complaint classification data. xlnet xlnet @xcite uses a similar architecture to bert to learn bidirectional contextual information. instead of masked tokens used in bert, xlnet maximizes the expected log-likelihood of all possible factorization orders. we adapt and fine-tune the xlnet model for complaint prediction similar to bert. m-bert to combine our model with external linguistic information, we adapt the multimodal bert (m-bert) @xcite model structure that has been introduced for multimodal modeling (text, image, speech). instead of cross-modal interactions, we inject extra linguistic information as alternative views of the data into the pre-trained bert model. we use (a) emotion, a 9 dimensional vector obtained by quantifying six basic emotions of @xcite for each tweet using a predictive model by @xcite ; (b) topics, a 200 dimensional vector representing word frequencies in word clusters designed to identify semantic themes in tweets by @xcite . to inject external linguistic information to m-bert, 3 we first project the linguistic information into vectors with similar size to the bert cls embeddings. then we concatenate word representations obtained from bert and the linguistic information (emotion, topics or emotion+topics) to generate combined embeddings. during concatenation, an attention gating mechanism called multimodal shifting gate @xcite is applied to control the importance of each representation. finally, the combined embeddings are fed to bert for fine-tuning. the rest of the architecture is the same as bert. ## experiments baselines we compare the transformer-based models with two previous approaches for complaint identification by @xcite and a transfer learning method: (1) logistic regression with bag-of-words trained using the original and distantly supervised complaint data (lr-bow + dist. supervision); (2) a long-short term memory (lstm) network @xcite that takes as input a tweet, maps its words to embeddings and subsequently passes them through the lstm to obtain a contextualized representation which is finally fed to the output layer; (3) adapting the pre-trained universal language model fine-tuning (ulmfit) model @xcite for complaint prediction. ulmfit uses a awd-lstm @xcite encoder for language modeling. hyper-parameters we use bert, albert and roberta base uncased models; fine-tuning them with learning rate @xmath0, @xmath1, 1e-5, 2e-5, 1e-6}. we use the base cased pre-trained xlnet tuning the learning rate over the same range as for bert models. for ulmfit, we use awd-lstm trained on wikitext-103. we simplify the default fine-tuning by only unfreezing the last 1 layer, the last 2 layers and all layers with learning rates l 1 = 1e-4 2.6 4 , l 2 = 1e-4 2.6 3 and l 3 = 1e-3 respectively. for m-bert, the size of feature embeddings (emotion, topics and emotion+topics) is @xmath2, @xmath3, 400, 768} with dropout @xmath4.1, @xmath5.1, .5} using the same parameters as bert. the maximum sequence length is set to 49 covering 95% of tweets in the training set. evaluation following @xcite , we use a nested 10-fold cross-validation approach to conduct our experiments for complaint prediction. in the outer 10 loops, 9 folds are used for training and one for testing; while in the inner loops, a 3-fold cross-validation method is applied where 2 folds are used for training and one for validation. during training, an early stopping method is applied based on the validation loss. we measure predictive performance using the mean accuracy, precision, recall and macro f1 over 10 folds (we also report the standard deviations). predictive results distant supervision is beneficial only to ulmfit and m-bert while bert and other transformer models perform worse, which are consistent with results of @xcite for sentiment analysis. model acc p r f1 preotiuc-pietro et al. (2019) lr-bow + dist. supervision 81.2 --79.0 lstm 80.2 --77.0 transfer learning baseline ulmfit 82.4 ± .04 81.1 ± .04 81.8 ± .04 81.2 ± .05 ulmfit + dist. supervision 83.3 ± .05 82.5 ± .05 81.8 ± .04 81.9 ± .05 transformers bert 88.0 ± .03 87.1 ± .03 87.3 ± .03 87.0 ± .03 albert 85.9 ± .03 84.8 ± .03 84.6 ± .03 84.6 ± .03 roberta 87.6 ± .03 86.6 ± .03 86.9 ± .03 86.6 ± .03 xlnet 83.9 ± .04 83.2 ± .04 82.3 ± .03 82.4 ± .05 m-bert -emotion also, results of m-bert models are comparable to bert, among which m-bert -topics is slightly better with 86.4 macro f1. we notice that injecting external linguistic information in bert's structure for fine-tuning does not help in our case without substantially hurting performance. we speculate that modifying bert embeddings by injecting extra linguistic information is not complementary to bert's text representations. ## conclusion we evaluated a battery of transformer networks on the twitter complaint identification task and obtained 87 macro f1, which outperforms the previous state-of-the-art results of @xcite . we further presented a thorough analysis of the limitations of our models in predicting complaints. in future work, we intend to explore more in how we can combine other sources of linguistic information with transformers as well as information from other modalities such as images.