Entity-level Sentiment Analysis in Contact Center Telephone Conversations (2022)

Abstract: Entity-level sentiment analysis predicts the sentiment about entities mentioned in a given text. It is very useful in a business context to understand user emotions towards certain entities, such as products or companies. In this paper, we demonstrate how we developed an entity-level sentiment analysis system that analyzes English telephone conversation transcripts in contact centers to provide business insight. We present two approaches, one entirely based on the transformer-based DistilBERT model, and another that uses a neural network supplemented with some heuristic rules.

## introduction businesses that provide contact center as a service (ccaas) often leverage artificial intelligence (ai) technologies to transcribe telephone conversations and generate aggregated insights reports for contact centers across various industry verticals. customers occasionally make evaluative comments about specific products or companies during a customer support call to a contact center. these comments provide valuable competitive insights to the business, e.g. positive comments may provide information useful to a marketing department as it formulates an advertising campaign while negative comments may provide valuable insights that can be used to improve a product or a service. in such scenarios, building a system that can identify user sentiments towards entities like product or companies could be useful. though aspect-based sentiment analysis (absa) @xcite ) that aims to extract the user sentiment expressed towards a specific aspect associated with a given target could be a possible solution to solve such problems, it should be noted that absa has a key limitation in this regard. for example, in tasks like competitor analysis where the objective is to understand the overall user sentiment within a certain period of time on specific products or general company services, absa techniques could not be useful as they provide more fine-grained opinions towards predefined aspects (e.g. features of a product, ease of use of a software, or aspects of restaurant experience) instead of providing a generic user sentiment towards a specific entity @xcite . while building the entity-level sentiment analysis (elsa) system for real-world contact center use-cases, we observe several key challenges. first of all, to the best of our knowledge, there are no existing public datasets available for the entity-level sentiment analysis task. meanwhile, this task becomes more challenging when the requirement is to use a dataset constructing from telephone conversations since constructing a dataset from speech transcripts generated from telephone conversations is non-trivial as telephone transcripts are generated by automatic speech recognition (asr) systems that have their own unique characteristics. for instance, an asr system may have mistranscription errors as well as linguistic disfluencies (e.g. filler words) @xcite that usually occur in a conversational speech dataset @xcite . the factors mentioned above make the implementation of an entity-level sentiment analysis model very challenging to detect user opinions towards entities that appear in contact center calls. in this paper, we address the existing limitations behind developing an entity-sentiment model for commercial scenarios in the domain of business telephone conversation data in contact centers. since there is no suitable publicly available dataset for the entity-level sentiment analysis task, we briefly describe how we sampled and annotated the data for this task. we then propose two approaches that leverage neural models for this task (i) one is based on the distilbert @xcite model in which we modify its architecture such that the model can also be utilized to extract the opinion term(s) while detecting the sentiment po-larity towards a named entity; (ii) while the other approach uses a convolutional neural network (dos santos and gatti, 2014) supplemented with some pre-defined heuristic rules. we compare the effectiveness of both approaches through extensive experiments and discuss our findings to provide valuable insights for future developments of elsa models for real world commercial scenarios. ## related work since the entity-level sentiment analysis task is closely related to aspect-level sentiment analysis, in this section, we first briefly review the aspect-level sentiment analysis task followed by the entity-level sentiment analysis task in order to clarify the distinction between these two tasks while discussing our rationale behind developing an entity-level sentiment analysis model for contact centers. ## task description let us assume that we have an utterance @xmath0, w 2 , ..., w n containing n words. the goal of the elsa task is to identify m opinion words o @xmath1, ow 2 , ..., ow m , (where @xmath2, and classify the sentiment of the identified opinion words towards the target entity e in the given utterance. in ## dataset construction as noted earlier, there is no publicly available dataset for the elsa task. we therefore had to create and annotate our own dataset. the first major issue that we observed while constructing a dataset for elsa is that the entity-level sentiment events in our telephone transcripts are very infrequent. hence, random data sampling techniques might yield an imbalanced dataset where most utterances would not have any positive or negative sentiments towards an entity. we therefore used two pre-existing models -a named entity recognition (ner) model based on distilbert @xcite that was trained to identify organization and product type entities and a convolutional neural network (cnn) @xcite @xcite sentiment analysis model -to sample 13000 utterances that contained at least one named entity and one positive or negative sentiment predicted by these models. to balance the dataset, we sampled an additional 10000 utterances containing at least one entity and having no polarized sentiments (i.e., only neutral sentiment). the resulting 23000 i work at google and i love it a lot. she's very impressed how mac works so well. he has hard time finding a good yogurt from walmart. it's quite difficult to navigate the mobile app of instacart. ## our proposed models for performance evaluation, we propose two approaches: (i) distilbert-based model, and (ii) cnn-based model with heuristics rules. below, we present our proposed approaches. ## experiment in this section, we present the training parameters, evaluation metrics, and the experimental results. in our experiments, we use the following three models: ## commercial application we have deployed the distilbert + sst model in our production system to generate entity sentiment data for contact centers as it has better accuracy than the cnn + heuristics model. due to the small model size and efficient inference of distil-bert, each model instance is assigned 1 cpu and 1gb memory. once there are enough entity-level sentiment predictions, there is a dedicated pipeline to aggregate entity sentiment in different granularity for each customer. there are many use cases of the aggregated insights of entity sentiment. contact center managers can use this information to improve contact center efficiency by investigating why customers are not happy with certain products (e.g., itelephone 13 pro max) and develop desired responses when the customer is complaining about it, so that the agents can handle the difficult situation more efficiently. the collected negative feedback can be used to inform the product team how to improve the products. the insights can also be used to conduct comparisons between several products or companies and help with competitor analysis. ## conclusion in this paper, we described the creation of a taskspecific dataset and a new model that extracts opinion words while performing entity sentiment polarity detection. the resulting distilbert-based model is currently deployed as a commercial application for entity-level sentiment analysis for english contact center conversations. in the future, we will investigate how to extend our proposed methods to other applications @xcite of the entity recognition task @xcite in telephone transcripts and explore how to improve model performance on utterances that contain more than one entity.