A Deep Learning System for Sentiment Analysis of Service Calls (2020)

Abstract: Sentiment analysis is crucial for the advancement of artificial intelligence (AI). Sentiment understanding can help AI to replicate human language and discourse. Studying the formation and response of sentiment state from well-trained Customer Service Representatives (CSRs) can help make the interaction between humans and AI more intelligent. In this paper, a sentiment analysis pipeline is first carried out with respect to real-world multi-party conversations - that is, service calls. Based on the acoustic and linguistic features extracted from the source information, a novel aggregated method for voice sentiment recognition framework is built. Each party’s sentiment pattern during the communication is investigated along with the interaction sentiment pattern between all parties.

## introduction the natural reference for ai systems is human behavior. in human social life, emotional intelligence is important for successful and effective communication. humans have the natural ability to comprehend and react to the emotions of their communication partners through vocal and facial expressions @xcite . a long-standing goal of ai has been to create affective agents that can recognize, interpret and express emotions. early-stage research in affective computing and sentiment analysis has mainly focused on understanding affect towards entities such as movie, product, service, candidacy, organization, action and so on in monologues, which involves only one person's opinion. however, with the advent of human-robot interaction (hri) such as voice assistants and customer service chatbots, researchers have started to build empathetic dialogue systems to improve the overall hri experience by adapting to customers' sentiment. sentiment study of human-human interactions (hhi) can help machines identify and react to human non-verbal communication which makes the hri experience more natural. the call center is a rich resource of communication data. a large number of calls are recorded daily in order to assess the quality of interactions between csrs and customers. learning the sentiment expressions from well-trained csrs during communication can help ai understand not only what the user says, but also how he/she says it so that the interaction feels more human. in this paper, we target and use real-world data -service calls, which poses additional challenges with respect to the artificial datasets that have been typically used in the past in multimodal sentiment researches @xcite , such as variability and noises. the basic 'sentiment' can be described on a scale of approval or disapproval, good or bad, positive or negative, and termed polarity @xcite . in the service industry, the key task is to enhance the quality of services by identifying issues that may be caused by systems of rules, or service qualities. these issues are usually expressed by a caller's anger or disappointment on a call. in addition, service chatbots are widely used to answer customer calls. if customers get angry during hri, the system should be able to transfer the customers to a live agent. in this study, we mainly focuses on identifying 'negative' sentiment, especially 'angry' customers. given the non-homogeneous nature of full call recordings, which typically include a mixture of negative, and nonnegative statements, sentiment analysis is addressed at the sentence level. call segments are explored in both acoustic and linguistic modalities. the temporal sentiment patterns between customers and csrs appearing in calls are described. the paper is organized as follows: section 2 covers a brief literature review on sentiment recognition from different modalities; section 3 proposes a pipeline which features our novelties in training data creation using real-world multi-party conversations, including a description of the data acquisition, speaker diarization, transcription, and semisupervised learning annotation; the methodologies for acoustic and linguistic sentiment analysis are presented in section 4; section 5 illustrates the methodologies adopted for fusing different modalities; section 6 presents experimental results including the evaluation measures and temporal sentiment patterns; finally, section 7 concludes the paper and outlines future work. ## related work in this section, we provide a brief overview of related work about text-based and audio-based sentiment analysis. ## dataset and pipeline the data resources used for our experiments are described in section 3.1. data preparation including speech transcription and speaker diarization is discussed in section 3.2. the sentiment annotation guideline is introduced in section 3.3. section 3.4 presents a semi-supervised learning annotation pipeline that chains data preparation, model training, model deploying and data monitor. ## bimodal sentiment analysis to model information for sentiment analysis from calls, we first obtain the streams corresponding to each modality via the methods described in section 3.2, followed by the extraction of a representative set of features for each modality. these features are then used as cues to build classifiers of binary sentiment. ## fusion there are two main fusion techniques: feature-level fusion and decision-level fusion. in our experiments, we employ decision-level fusion. decisionlevel fusion has many advantages @xcite . one benefit of the decision-level fusion is we can use classifiers for text and audio features separately. the unimodal classifier can use data from another communication channel of the same type to improve its accuracy, e.g. text data from the chat windows is borrowed to improve the c t accuracy in our study. separating modalities permit us to use any learner suitable for the particular problem at hand. in practice, the two unimodal classifiers can be applied separately, e.g. to analyze text data from chat windows d @xmath0, apply c t only to get sentiment labels d u t , then add d u t (m ) to d lt . another benefit of the decisionlevel fusion is its processing speed since fewer features are used for each classifier and separate classifiers can be run in parallel. decision-level fusion usually adds probabilities or summarized predictions from each unimodal classifier with weights or takes the majority voting among the predicted class labels by unimodal classifiers. in this paper, various fusion methods are evaluated, including two novel approaches that use linguistic ensemble results as the baseline, while then checking acoustic results to modify classification decisions. in fus1, if the audio ensemble classifies negative and one or more text models classifies negative, we then reclassify the result to negative. in fus2, if the audio ensemble classifies a sample as negative, we then reclassify the result to negative directly without checking the linguistic modality. the fus1 and fus2 approaches are proposed, because for borderline linguistic utterances, acoustic features are more important than linguistic features to understand the spoken intention of the speaker. ## experiment results the test dataset consists of 21 calls with 1,890 utterances, which are manually annotated for negative (848) and nonnegative (1,042). ## discussion and future work a new dataset bscd consisting of real-world conversation, the service calls, is introduced. human communication is a dynamic process, and our eventual goal is to develop a bimodal sentiment analysis engine with the ability to learn the temporal interaction sentiment patterns among conversation parties. in the process of fusion, we have approached the study of audio sentiment analysis from an angle that is somewhat different from most people's. future research will concentrate on evaluations using larger data sets, exploring more acoustic feature relevance analysis, and striving to improve the decision-level fusion process. a call is constituent of a group of utterances that have contextual dependencies among them. however, in our semi-supervised learning annotation pipeline, about half of the segments in calls are discarded. therefore the interdependent modeling is out of the scope of this paper and we include it as future work.