Discern: Discourse-Aware Entailment Reasoning Network for Conversational Machine Reading (2020)

Abstract: Document interpretation and dialog understanding are the two major challenges for conversational machine reading. In this work, we propose “Discern”, a discourse-aware entailment reasoning network to strengthen the connection and enhance the understanding of both document and dialog. Specifically, we split the document into clause-like elementary discourse units (EDU) using a pre-trained discourse segmentation model, and we train our model in a weakly-supervised manner to predict whether each EDU is entailed by the user feedback in a conversation. Based on the learned EDU and entailment representations, we either reply to the user our final decision “yes/no/irrelevant” of the initial question, or generate a follow-up question to inquiry more information. Our experiments on the ShARC benchmark (blind, held-out test set) show that Discern achieves state-of-the-art results of 78.3% macro-averaged accuracy on decision making and 64.0 BLEU1 on follow-up question generation. Code and models are released at https://github.com/Yifan-Gao/Discern .

## introduction conversational machine reading (cmr) is challenging because the rule text may not contain the literal answer, but provide a procedure to derive it through interactions @xcite . in this case, the machine needs to read the rule text, interpret the user scenario, clarify the unknown user's background by asking questions, and derive the final answer. taking figure 1 as an example, to answer the user whether he is suitable for the loan program, the machine needs to interpret the rule text to know what are the requirements, understand he meets "american small business" from the user scenario, ask follow-up clarification questions about "for-profit business" and "not get financing rule text: 7(a) loans are the most basic and most used type loan of the small business administration's (sba) business loan programs. it's name comes from section 7(a) of the small business act, which authorizes the agency to provide business loans to american small businesses. the loan program is designed to assist for-profit businesses that are not able to get other financing from other resources. user scenario: i am a 34 year old man from the united states who owns their own business. we are an american small business. user question: is the 7(a) loan program for me? follow-up q1: are you a for-profit business? follow-up a1: yes. follow-up q2: are you able to get financing from other resources? follow-up a2: no. final answer: yes. (you can apply the loan.) figure 1 : an example dialog from the sharc @xcite dataset. the machine answers the user question by reading the rule text, interpreting the user scenario, and keeping asking follow-up questions to clarify the user's background until it concludes a final answer. requirements in the rule text are bold. from other resources", and finally it concludes the answer "yes" to the user's initial question. existing approaches @xcite @xcite decompose this problem into two sub-tasks. given the rule text, user question, user scenario, and dialog history (if any), the first sub-task is to make a decision among "yes", "no", "inquire" and "irrelevant". the "yes/no" directly answers the user question and "irrelevant" means the user question is unanswerable by the rule text. if the user-provided information (user scenario, previous dialogs) are not enough to determine his fulfillment or eligibility, an "inquire" decision is made and the second sub-task is activated. the second sub-task is to capture the underspecified condition from the rule text and generate a follow-up question to clarify it. @xcite adopt bert @xcite to reason out the decision, and propose an entailment-driven extracting and editing framework to extract a span from the rule text and edit it into the follow-up question. the current state-of-the-art model emt @xcite uses a recurrent entity network @xcite with explicit memory to track the fulfillment of rules at each dialog turn for decision making and question generation. in this problem, document interpretation requires identification of conditions and determination of logical structures because rules can appear in the format of bullet points, in-line conditions, conjunctions, disjunctions, etc. hence, correctly interpreting rules is the first step towards decision making. another challenge is dialog understanding. the model needs to evaluate the user's fulfillment over the conditions, and jointly consider the fulfillment states and the logical structure of rules for decision making. for example, disjunctions and conjunctions of conditions have completely different requirements over the user's fulfillment states. however, existing methods have not considered condition-level understanding and reasoning. in this work, we propose discern: discourse-aware entailment reasoning network . to better understand the logical structure of a rule text and to extract conditions from it, we first segment the rule text into clause-like elementary discourse units (edus) using a pre-trained discourse segmentation model @xcite . each edu is treated as a condition of the rule text, and our model estimates its entailment confidence scores over three states: entailment, contradiction or neutral by reading the user scenario description and existing dialog. then we map the scores to an entailment vector for each condition, and reason out the decision based on the entailment vectors and the logical structure of rules. compared to previous methods that do little entailment reasoning @xcite or use it as multi-task learning @xcite , discern is the first method to explicitly build the dependency between entailment states and decisions at each dialog turn. discern achieves new state-of-the-art results on the blind, held out test set of sharc @xcite . in particular, discern outperforms the previous best model emt @xcite by 3.8% in micro-averaged decision accuracy and 3.5% in macro-averaged decision accuracy. specifically, discern performs well on simple in-line conditions and conjunctions of rules while still needing improvements on understanding disjunctions. finally, we conduct comprehensive analyses to unveil the limitation of discern and current challenges for the sharc benchmark. we find one of the biggest bottlenecks is the user scenario interpretation, in which various types of reasoning are required. ## discern model discern answers the user question through a three-step process shown in figure 2: 1. first, discern segments the rule text into individual conditions using discourse segmentation. ## related work entailment reasoning in reading comprehension. understanding entailments (or implications) of text is essential in dialog and question answering systems. ropes @xcite requires reading descriptions of causes and effects and applying them to situated questions, while sharc @xcite , the focus of discern, requires to understand rules and apply them to questions asked by users in a conversational manner. most existing methods simply use bert to classify the answer without considering the structures of rule texts @xcite @xcite . @xcite propose explicit memory tracker (emt), which firstly addresses entailment-oriented reasoning. at each dialog turn, emt recurrently tracks whether conditions listed in the rule text have already been satisfied to make a decision. in this paper, we also explicitly model entailment reasoning for decision making, but there are three key differences between our discern and emt: (1) we apply discourse segmentation to parse the rule text, which is extremely helpful because there are many in-line conditions in rules; (2) our stacked inter-sentence transformer layers extract better features for entailment prediction, which could be seen as a generalization of their recurrent explicit memory tracker. (3) different from their utilization of entailment prediction which is treated as multi-task learning for decision making, we directly build the dependency between entailment prediction states and the predicted decisions. ## conclusion in this paper, we present discern, a system that does discourse-aware entailment reasoning for conversational machine reading. discern explicitly builds the connection between entailment states of conditions and the final decisions. results on the sharc benchmark shows that discern outperforms existing methods by a large margin. we also conduct comprehensive analyses to unveil the limitations of discern and challenges for sharc. in future, we plan to explore how to incorporate discourse parsing into the current decision making model for end-to-end learning. one possibility would be to frame them as multi-task learning with a common (shared) encoder.