Structural Constraints and Natural Language Inference for End-to-End Flowchart Grounded Dialog Response Generation (2022)

Abstract: Flowchart grounded dialog systems converse with users by following a given flowchart and a corpus of FAQs. The existing state-of-the-art approach (Raghu et al, 2021) for learning such a dialog system, named FLONET, has two main limitations. (1) It uses a Retrieval Augmented Generation (RAG) framework which represents a flowchart as a bag of nodes. By doing so, it loses the connectivity structure between nodes that can aid in better response generation. (2) Typically dialogs progress with the agent asking polar (Y/N) questions, but users often respond indirectly without the explicit use of polar words. In such cases, it fails to understand the correct polarity of the answer. To overcome these issues, we propose Structure-Aware FLONET (SA-FLONET) which infuses structural constraints derived from the connectivity structure of flowcharts into the RAG framework. It uses natural language inference to better predict the polarity of indirect Y/N answers. We find that SA-FLONET outperforms FLONET, with a success rate improvement of 68% and 123% in flowchart grounded response generation and zero-shot flowchart grounded response generation tasks respectively.

## introduction task-oriented dialog systems converse with users to achieve a specific task (e.g., restaurant recommendation) using information from an associated knowledge source (e.g., a kb of restaurants). end-to-end approaches @xcite @xcite et al. the best performing end-to-end approach for learning flowchart grounded dialogs @xcite has two limitations. it follows a retrieval augmented generation (rag) framework, which first retrieves a document (e.g., flowchart node) based on the dialog history and then generates the response using the retrieved document. the first issue arises due to the representation of the flowchart as a bag of nodes by the rag retriever, which fails to capture the node connectivity structure in the flowchart. due to this shortcoming, the model incorrectly grounds consecutive utterances in a dialog on non-adjacent flowchart nodes, whereas they are expected to be grounded on adjacent nodes. for example, in figure 1 (a) the system utterance s2 is grounded on a node that is not adjacent to the node corresponding to the previous system utterance. secondly, agents typically ask polar (y/n) questions to traverse the flowchart. for example, the agent may ask 'does the car stop when you apply the brakes?". users often respond indirectly to these questions without any explicit use of polar words. for example, the user can indirectly respond to the agent question as "the car keeps going when i activate the brakes". as shown in figure 1 (b), the existing approach fails to map the indirect user response to the correct polarity. to tackle these two limitations, we propose structure aware -flonet (sa-flonet) that also follows the rag framework. our proposed approach augments the rag retriever with memory to enable the retriever to store past decisions. the memory when used along with the flowchart structure ensures the dialog mostly follows a path in the flowchart. this helps consecutive utterances be grounded on adjacent nodes in the flowchart. sa-flonet poses understanding user utterances as a natural language inference (nli) task to leverage complex language reasoning ability from existing nli models. we evaluate the performance sa-flonet on the flodial dataset. com-starter cranks? battery read over 12v? starter spins? i have had my lincoln mkc for the past three years and it will not start. this has caused me to use vacation days because i can not get to work. does the starter crank when you turn the key to start the engine? it turns over as usual. i don't think there is any issue with that. is the battery reading more than 12v? no no yes breaks stop car? pedal to floor? i've had my cadillac escalade esv for 3 years. i've noticed that the brakes aren't working. please help me fix this. once you activate the brakes, does the car come to a stop? the car keeps going when i activate the brakes. is your emergency parking brake doing fine? pared to the state-of-the-art approach, sa-flonet improves the task specific metric (i.e., success rate) by 68% and 123% on flowchart grounded response generation (fgrg) and zero-shot flowchart grounded response generation (zs-fgrg). ## related work in this work, we propose a novel neural architecture for end-to-end flowchart grounded response generation and its zero-shot variant. our main contributions are (1) augmenting rag @xcite with memory for infusing structural constraints of flowcharts and (2) using nli @xcite for predicting polarity of indirect y/n answers. we now briefly discuss existing literature that uses rag for dialog applications and then review works related to nli. ## preliminaries in this section, we describe the problem of end-toend learning of flowchart grounded task oriented dialogs. we then briefly describe the previous work (flonet) @xcite ) over which we build our proposed approach. ## experimental results our experiments answer three research questions: 1. how does the performance of sa-flonet compare with existing approaches? (section 6.1) 2. what is the performance gain from each novel contribution in sa-flonet? (section 6.2) 3. what is the importance of content based scorer in sa-flonet? (section 6.3) ## discussion we illustrate the benefit of the structure aware scorer (sas) using a qualitative example shown in figure 3 . the figure visualizes how sas ensures the retriever follows a path in the flowchart rather than making random jumps across the flowchart. the figure shows the input dialog context along with the gold response, the part of the flowchart necessary for explaining the benefit of sas, and components of sa-flonet responsible for ranking the documents. we represent each component using the probabilities of states/documents computed based on the input dialog context. in the given dialog context, the agent utterance a 1 is grounded on the flowchart node z 2 . the belief tracker should ideally assign a probability close to 1 to the state s 2 corresponding to the node z 2 . however, it has only a weak belief of around 0.16 on this state. once the user responds with u 2 , we expect it to be mapped to the n o child of z 2 and and hence the retriever should assign the highest probability to the node z 3 . we use a simplified representation of the nli model with just the document distribution conditioned on s 2 and u 2 , as we need just the true belief (s 2 ) to explain the benefit of sas. the nli module assigns a high probability (0.97) to z 3 indicating that it confidently understands that the user is saying no given s 2 . the beliefs and nli scores are combined according to equation 2 to compute the structure aware scores. the content based scorer fails to assign a high probability on the correct document z 3 and incorrectly assigns the highest probability to node z 5 . when the two scores are combined to compute the overall retriever score, we see that node z 3 receives the highest score and the response is grounded correctly. the example clearly shows how the two scorers have to work together to identify the correct document. moreover, it shows the ability of the nli module to correct the system even when both the content based scorer and the belief tracker do not provide strong signals. ## conclusion we propose sa-flonet for learning flowchart grounded dialogs in an end-to-end manner. sa-flonet augments the rag retriever with memory and incorporates flowchart structural constraints into the retrieval process. it uses nli to better understand indirect user response to polar agent questions. sa-flonet achieves the state-of-theart results on both the seen flowchart and the unseen flowchart split of the flodial dataset. it outperforms existing approaches by 15 or more points on success rate. human evaluations show that sa-flonet responses are more relevant than the previous state-of-the-art.