Semi-Supervised Iterative Approach for Domain-Specific Complaint Detection in Social Media (2020)

Abstract: In this paper, we present a semi-supervised bootstrapping approach to detect product or service related complaints in social media. Our approach begins with a small collection of annotated samples which are used to identify a preliminary set of linguistic indicators pertinent to complaints. These indicators are then used to expand the dataset. The expanded dataset is again used to extract more indicators. This process is applied for several iterations until we can no longer find any new indicators. We evaluated this approach on a Twitter corpus specifically to detect complaints about transportation services. We started with an annotated set of 326 samples of transportation complaints, and after four iterations of the approach, we collected 2,840 indicators and over 3,700 tweets. We annotated a random sample of 700 tweets from the final dataset and observed that nearly half the samples were actual transportation complaints. Lastly, we also studied how different features based on semantics, orthographic properties, and sentiment contribute towards the prediction of complaints.

## introduction social media has lately become one of the primary venues where users express their opinions about various products and services. these opinions are extremely useful in understanding the user's perceptions and sentiment about these services. they are also useful in identifying potential defects @xcite and thus critical to the execution of downstream customer service responses. therefore, automatic detection of user complaints on social media could prove beneficial to both the clients and the service providers. to build such detection systems, we could employ supervised approaches that would typically require a large corpus of labeled training samples. however, labeling social media posts that capture complaints about a particular service is challenging because of their low prevalence and also the vast amounts of inevitable noise @xcite . additionally, social media platforms are also likely to be plagued with redundancy, where the posts are rephrased or structurally morphed before being re-posted @xcite . prior work in event detection @xcite ) has demonstrated that simple linguistic indicators (phrases or n-grams) can be useful in the accurate discovery of events in social media. though user complaints are not the same as events, more of a speech act @xcite , we posit that similar indicators can be used in complaint detection. to pursue this hypothesis, we propose a semi-supervised iterative approach to identify social media posts that complain about a specific service. in our approach, we first begin with a small, manually curated dataset containing samples of social media posts complaining about a service. we then identify linguistic indicators (phrases or n-grams) that serve as strong evidence of this phenomenon. these indicators are then used to extract more posts from the unannotated corpus. this newly obtained data is then used to create a new set of indicators. this process is repeated until it reaches a certain convergence point. since the set of indicators is growing after each iteration, they are re-evaluated continuously in terms of their relevance. this process is similar to the mutual bootstrapping approach for information extraction proposed in @xcite . we employ this approach to the problem of complaint detection for transportation services on twitter. transportation and its related logistic services are critical aspects of every economy as they account for nearly 40% of the value of international trade @xcite . as with most businesses @xcite , transportation also often relies on social media to ascertain feedback and initiate appropriate responses @xcite . in our experimental work, we started with an annotated set of 326 samples of transportation complaints, and after four iterations of the approach, we collected 2,840 indicators and over 3,700 tweets. we annotated a random sample of 700 tweets from the final dataset and observed that over 47% of the samples were actual transportation complaints. we also characterize the performance of basic classification algorithms on this dataset. in doing so, we also study how different linguistic features contribute to the performance of a supervised model in this domain. the main contributions of this paper are as follows: ## related work complaints are often considered dialogue acts used to express a mismatch between the expectation and reality @xcite . the problem of complaint detection is of great interest to the marketing and research teams of various service providers. previous works on complaint identification have applied text mining with lda and sentiment analysis on user-generated content @xcite . prior works have also focused on leveraging data streamed from social media platforms for outage and complaint detection as they are publicly available @xcite . @xcite inspected customer support dialogue for support. different complaint expressions have been explored by analyzing variations across cultures @xcite , sociodemographic traits @xcite and temporal representations @xcite . however, mentioned works on user-generated content have focused on static data repositories only. these have not been robust to linguistic variations @xcite and morphological changes @xcite . our pipeline builds on linguistic identifiers to expand on lexical cues in order to identify complaint relevant posts. researches have proposed many semisupervised architectures for identification of events pertaining to societal and civil unrest @xcite , using speech modality @xcite @xcite and hidden markov models @xcite . these have been documented to give better performance as compared against their counterparts @xcite with minimal intervention @xcite . for our analysis, the semi-supervised approach has been preferred as opposed to supervised ones because: (a) usage of supervised approach relies on carefully choosing the training set making it cumbersome and less attractive for practical use @xcite and (b) imbalance between the subjective and objective classes lead to poor performance @xcite . ## methods and data our proposed approach begins with a large corpus of transport-related tweets and a small set of annotated complaints. we use this labeled data to create a set of seed indicators that drive the rest of our iterative complaint detection process. ## modeling we conducted a series of experiments to understand if we can automatically build simple machine learning models to detect complaints. these experiments also helped us evaluate the quality of the final dataset. additionally, this experimental work also studies how different types of linguistic features contribute to the detection of social media complaints. for these experiments, we used the annotated sample of 700 posts as a test dataset. we built our training dataset by selecting another 2,000 posts from the original corpus c, and anno- ## conclusion and future work in this paper, we presented a semi-supervised iterative approach for the detection of complaints in social media platforms. the process begins with a small sample of annotated examples, and then iteratively builds more linguistic identifiers to expand the dataset. we evaluated this approach on the domain of transportation on twitter, starting with a sample of 326 annotated tweets. after four iterations, we were able to construct a corpus with over 3,700 tweets. annotation of random samples established that nearly half the tweets were actual complaints. we evaluated the predictive power based on semantic, orthographic, and sentiment features. we observed that complaint is a complex speech act, which is related to many other linguistic properties. automatic detection of complaints is not only useful to service providers as feedback; it could also prove helpful in improving service providers' operations and in downstream applications such as developing chat-bots. additionally, it could also be of interest to linguists in understanding how humans express grievances and criticism. this proposed methodology could be applied to many other products or services to detect complaints. this would only additionally require some lexicons and a small annotated dataset. we also expect it would be fairly straightforward to adapt this technique to many other types of speech acts. further investigation is necessary to understand how this method compares against supervised or completely unsupervised techniques.