AVA - TMP : A Human-in-the-Loop Multi-layer Dynamic Topic Modeling Pipeline (2022)

Abstract: A phone call is still one of the primary preferred channels for seniors to express their needs, ask questions, and inform potential problems to their health insurance plans. Alignment Healthis a next-generation, consumer-centric organization that is providing a variety of Medicare Advantage Products for seniors. We combine our proprietary technology platform, AVA, and our high-touch clinical model to provide seniors with care as it should be: high quality, low cost, and accompanied by a vastly improved consumer experience. Our members have the ability to connect with our member services and concierge teams 24/7 for a wide variety of ever-changing reasons through different channels, such as phone, email, and messages. We strive to provide an excellent member experience and ensure our members are getting the help and information they need at every touch —ideally, even before they reach us. This requires ongoing monitoring of reasons for contacting us, ensuring agents are equipped with the right tools and information to serve members, and coming up with proactive strategies to eliminate the need for the call when possible. We developed an NLP-based dynamic call reason tagging and reporting pipeline with an optimized human-in-the-loop approach to enable accurate call reason reporting and monitoring with the ability to see high-level trends as well as drill down into more granular sub-reasons. Our system produces 96.4% precision and 30%-50% better recall in tagging calls with proper reasons. We have also consistently achieved a 60+ Net Promoter Score (NPS) score, which illustrates high consumer satisfaction.

## introduction as a consumer-centered healthcare company, we provide our members with experienced member services and concierge teams through our contact center that are available around the clock for a wide variety of inquiries or potential problems. it is crucial for us to monitor the typical causes of why members contact us so that we proactively address the problems or help our members need through their preferred channel even before they reach out to us. to this end, during each call, member service agents take detailed notes on what is discussed during the call, the reason for the call, and any actions taken. they also tag each call with one or more call reason categories from a pre-defined list that was initially built by member experience supervisors to capture call reasons in a more structured manner. there are several drawbacks and limitations to this approach: in this paper, we present a novel nlp-based multi-layer dynamic topic modeling pipeline, ava-tmp, with an effective human-in-the-loop approach that leverages subject matter experts. it enables highly accurate and timely call reason logging, monitoring, and reporting, as well as increased first-time resolution rates. our pipeline produces a high-quality call reasons list with subreason drill-downs and automatically identifies newly emerging high-quality topics eliminating the need for labor-intensive evaluation of highvolume call notes by the customer service supervisors. our pipeline also increases the efficiency of customer service agents by automatically suggesting a ranked list of relevant topics to tag as agents take call notes. the presented pipeline is not limited to inbound calls, and it is applicable to omnichannel communications such as emails, messages, and chats. in the remainder of the paper, we first present the related work in section 2. next, we describe our approach and evaluate its performance using several real-world datasets in section 3. then, we evaluate the performance of our algorithm. finally, we conclude in section 4. ## related work it has been a great stride in natural language processing (nlp) advancement in recent years for learning, understanding, and producing human language content in more efficient and scalable ways @xcite , such as text summarization @xcite , name entity recognition @xcite , sentimental analysis @xcite ), text classification (bhavani and kumar, 2021), and topic modeling @xcite . topic modeling is used to automatically identify the themes, i.e., topics, in unstructured text datasets @xcite , especially when there is a large volume of document collections and not enough time. these machinegenerated topics are then used for reporting and tracking purposes. the traditional topic modeling pipelines @xcite @xcite do not allow the users to refine, clean, or personalize the model-generated topics hence lack an understanding of the end-user needs @xcite . the absence of an interactive and human-centered approach to refine topics and adjust data processing results in bias, noise, and unexpectedly poor real-world model performance. it is crucial to have an efficient and effective level of human contribution with ai to ensure satisfactory model performance. previous works have utilized topic modeling techniques with some level of human intervention to address customer needs. for example, a study by agudelo and manuel used latent dirichlet allocation (lda) @xcite based topic modeling approach to identify topics of inbound call transcripts for creating a better interactive voice response (ivr) routing option so that customers are routed to the right agents (agudelo and manuel, 2019). however, they only provide the most important words to the end users and not the topic. the users then need to manually identify the topic associated with the words presented by the model which can be inconsistent among different users and adds an additional burden to their workflow. this process also results in assigning only one topic to a call which might not be a true representation of customer needs as one call might have multiple reasons and resolutions. another study by chen and wang utilized the lda topic modeling technique to extract topics from chat transcripts between librarians and students in order to identify needs, provide help and allocate resources accordingly @xcite . however, this work lacks defining a qualitative performance measurement. also, they assessed the quality of the topics intuitively with visualizations. besides topic modeling methods from machine learning, there are also qualitative analytic methods, such as grounded theory methodologies @xcite , for identifying topics in text datasets. previous work by @xcite focused on the comparison between grounded theory and topic modeling @xcite . the authors found that the results of the two methods exhibited a degree of alignment in which many of the patterns found in the grounded theory were also represented in the topic modeling results. however, grounded theory is time-consuming and resource-intensive. therefore, it doesn't scale well with large datasets. different from these previous works, we implemented a novel nlp-based system with humanin-the-loop stages to accurately tag high-level call reasons (one or multiple) along with sub-category drill-downs in a scalable and timely manner for each call. we also measured the real-world performance of our approach and compared it with the baseline (manual labeling). ## methodology our comprehensive end-to-end topic modeling pipeline, ava-tmp, is illustrated in figure 1 . with ava-tmp, we enhance the traditional nlp topic modeling pipeline with human-in-the-loop stages to significantly improve reporting accuracy, create operational efficiencies and increase member satisfaction. standard high-level topic modeling steps include problem definition, data collection, data preparation (tokenization, lemmatization, and stop-word removal), modeling, post-processing & model eval- agents' call comments nlp-based call reason categories refined call reason categories based on nlp model output pre-defined call reason categories for each topic member experience supervisor initial input member experience agent agent documents every call with detailed unstructured notes as well as tags every call with relevant call reason categories from the crm system member experience supervisor call reason trend reporting dashboards topic modeling text pre-processing data modeling topic post-processing refresh call reason categories for each topic adjustment needed uation, and model deployment. we altered and improved the standard topic modeling pipeline in three ways: in the following subsections, we describe the major end-to-end steps from initial data collection to the final model deployment, and how they are being used. ## conclusion in this paper, we demonstrate a novel topic modeling pipeline, ava-tmp, that consists of multilayer dynamic hierarchical topic modeling with the human-in-the-loop approach for call analytics. our results illustrate having human-in-the-loop in an advanced nlp pipeline can optimize model performance, reduce manual tasks in the current workflows, and improve overall business outcomes while helping achieve high member satisfaction. with this framework, we achieved significantly better performance using real-world datasets for evaluation. alignment health has also consistently achieved an overall nps, which is a widely used metric to measure customer experience, of 60+ as a medicare advantage plan, compared to the industry average of 30 for healthcare insurance @xcite . ava-tmp enables us to have an improved call reporting and tracking system. we can identify seasonal trends for demands, inquiries, or emerging issues and develop proactive and systemic plans for improvement. it also creates operational efficiencies by eliminating the need for various laborintensive tasks such as i) having an up-to-date call reason list, ii) manual call reason tagging, and iii) manual ad-hoc reports. even though we illustrated the ava-tmp pipeline on call notes, it is applicable to omnichannel communications such as calls, emails, messages, and chats. it can be also used on call transcripts in addition to the call notes for improved outcomes.