Graph Integrated Language Transformers for Next Action Prediction in Complex Phone Calls (2024)

Abstract: Current Conversational AI systems employ different machine learning pipelines, as well as external knowledge sources and business logic to predict the next action. Maintaining various components in dialogue managers’ pipeline adds complexity in expansion and updates, increases processing time, and causes additive noise through the pipeline that can lead to incorrect next action prediction. This paper investigates graph integration into language transformers to improve understanding the relationships between humans’ utterances, previous, and next actions without the dependency on external sources or components. Experimental analyses on real calls indicate that the proposed Graph Integrated Language Transformer models can achieve higher performance compared to other production level conversational AI systems in driving interactive calls with human users in real-world settings.

## introduction building and maintaining complex production quality conversational systems has been an ongoing challenge in industry. one approach to solve complex conversational tasks such as outbound call automation, is to use a dialogue manager @xcite to encode business logic. conversational systems which use dialogue managers have multiple components which consist of natural language understanding (nlu) @xcite , dialogue state tracking @xcite , next action prediction @xcite , and response generation @xcite . figure 1 describes the process of call automation systems with the aforementioned components. handling next action prediction is one of the critical tasks dialogue managers take care of, as it affects the response generation directly @xcite . next action prediction is the process of analyzing human utterance, current and previous state figure 1 : a schematic visualization of dialogue managers' components which utilize an nlu pipeline of models to extract intents and fill slots from human utterances, and predict the next action based on the current and previous state. finally, the system generates an utterance to respond to the human users (e.g., using llms or predefined templates). recently, there has been significant progress in the field of generative ai and large language models (llms) for end-to-end conversational systems which alleviate the need for manually engineered dialogue managers @xcite . however, they sometimes have issues with hallucinations and can underperform in domain specific, targeted conversations such as those that require knowledge graph retrieval @xcite . in most industry settings, templates are used with action prediction to generate the response. by predicting an action, we are determining which response template(s) to return to the user @xcite @xcite . action prediction using response templates instead of figure 2 : an example of visual navigation assistant as a dialogue manager. at each time-step t, the dialogue manager extracts the entities such as slots and intents from human utterance u t (i.e., green rectangles on the left side) and predicts the next action a t (i.e., red rectangles). using the predicted action (i.e., blue rectangles) the dialogue manager generates a system response (i.e., green rectangles on the right side). to solve the next action prediction problem, different nlp methods from traditional symbolic ai techniques such as knowledge graph models @xcite , to more modern transformer based techniques (e.g., @xcite have been introduced; however, two main challenges still persist. 1) a majority of prior work depends on slot-filling (sf) and intent-classification (ic) techniques to extract dependencies and relies on external sources (i.e., knowledge or rule based approaches) to find the relationship between the extracted information and actions @xcite . instability in detecting sf and ic causes incorrect next action prediction. 2) many conversational systems handle grounding poorly @xcite @xcite ; this is when users' responses differ from expected inputs (e.g., referring to a previous point in the conversation, moving backwards to change a previous response, or no action-related slots being detected). for example, in figure 2 , the human user sets a new ground by mentioning the elevator instead of the their location. this information may be slightly different than what a next action prediction model expects and can respond to. lack of grounding in a conversation and more specifically in a model may result in misunderstanding @xcite and can damage the conversation. this paper introduces an approach to predict the next action without any dependency on information extraction (i.e., sf and ic) or external resources foot_0 such as ontology (e.g., @xcite or knowledge-base (e.g., vizcarra and jokinen, 2022) approaches. the proposed models, graph integrated language transformers, learns co-occurrences of actions and human utterances through a graph component (i.e., graph neural network or a graph embedding layer) and combines it with language transformers to add language understanding in production settings. the model is trained on conversations that followed a standard operating procedure (sop) foot_1 without the need for explicit encoding. the proposed model can be trained on any similar dataset that has an inherent action-to-action relationships. the list below summarizes the contribution of this paper. to examine the performance and robustness of the proposed models in real-world settings with noisy input, the evaluation is done in a production setting and goes beyond classification metrics; the evaluation includes industry critical factors such as human experience using the conversational system and considers real-time constraints such as latency of output generation. ## related work next action prediction approaches can be categorized in three chief groups. first, structured-based approaches that consider sequential relationships between previous actions, other actions, and their requirements. these approaches assume that the current state (i.e., the previous action) is known @xcite . on the one hand, local structure-based approaches such as question & answer systems @xcite consider local adjacency of the actions, utterance features, and next potential actions. on the other hand, global structured-based approaches define problem space using dialogue-grammars or finitestate networks @xcite . however, none of structured-based approaches provide the ability to train a model and they require expert to design them @xcite . the second group of next action prediction approaches are principle-based. these techniques choose next actions based on the filled information rather than sequential order between actions, thus behaving both locally and globally @xcite . slot-filling (sf) and intent-classification (ic) based techniques (i.e., joined or separate components) are common principle based approaches @xcite . recently, neural models including rnns and language transformers which act solely on input are receiving more attention for sf-ic based techniques @xcite @xcite . these methods are mainly using dialogue history alongside additional information such as schema of the task (e.g., "hotel booking" or "scheduling a doctor's appointment") e.g., using embedding layers with or without attention layers fused with a language transformer (e.g., @xcite @xcite . however, most of these language transformer based techniques were only evaluated on datasets with low number of actions, 10 or less @xcite , or perform poorly on larger number of actions (i.e., 30 actions) for one top output selection @xcite . ## methodology this section discusses the problem definition of the next action prediction task (i.e., section 3.1), and introduces the proposed models (i.e., section 3.2). ## experimental setup and results this section describes the process of collecting data for training the models, comparing the trained models regarding classification metrics (i.e., f 1), and evaluating the proposed models as well as the dm system foot_2 , explained in detail in appendix a, using a human-centered approach. ## conclusion this paper proposes graph integrated language transformers technique to improve next action prediction performance to resolve the dependency on slot-filling and intent-classification techniques and grounding issue @xcite . the analyses indicate that keeping the action history with order of the actions using a graph embedding layer and combining with language transformers generates higher quality of outputs in comparison to more complex models that include connection details of actions (i.e, gnns including the connection details through edges). the proposed model(s) improve the next action prediction regarding f 1 as well as product-level metrics and human-centered evaluation. they can improve the robustness regarding next action prediction (e.g., less unexpected results or being stuck in a loop) in comparison to other techniques and handle complex tasks better in comparison to the dm system in long noisy phone calls. additionally, the proposed models can reach to a high performance level with as low as 60k dialogue turns. we hope future research can employ a similar method combined with generative ai models to extract the information from human utterances as well as generating custom responses to automate calls without dependency on other components.