Towards End-to-End Open Conversational Machine Reading (2023)

Abstract: In open-retrieval conversational machine reading (OR-CMR) task, machines are required to do multi-turn question answering given dialogue history and a textual knowledge base. Existing works generally utilize two independent modules to approach this problem’s two successive sub-tasks: first with a hard-label decision making and second with a question generation aided by various entailment reasoning methods. Such usual cascaded modeling is vulnerable to error propagation and prevents the two sub-tasks from being consistently optimized. In this work, we instead model OR-CMR as a unified text-to-text task in a fully end-to-end style. Experiments on the ShARC and OR-ShARC dataset show the effectiveness of our proposed end-to-end framework on both sub-tasks by a large margin, achieving new state-of-the-art results. Further ablation studies support that our framework can generalize to different backbone models.

## introduction in a multi-turn dialogue comprehension scenario, machines are expected to answer high-level questions through interactions with human beings until enough information is gathered to derive a satisfying answer @xcite @xcite @xcite . as a specific and challenging dialogue comprehension task, conversational machine reading (cmr) @xcite requires machines to understand the given user's initial setting and dialogue history before the machine itself is able to give a final answer or inquire for more clarifications according to rule texts (see figure 1 ). in terms of acquisition of rule texts which are the main reference for tackling the cmr, there is closed-book setting where the rule texts are all given and there is correspondingly open-retrieval setting where the rule texts need to be retrieved from a knowledge base @xcite (see figure 1 ). in terms of problem objectives, current approaches in general divide the targets into two categories, one as decision making and one as question generation @xcite @xcite . for decision making sub-task, the machine is required to give decisions to directly answer the user question which concludes the dialogue or generate clarifying questions which continues the dialogue. for question generation sub-task, the machine is required to generate the clarifying questions that are essential to the later final decision making. following this line of approaching the cmr task, a variety of works have been proposed mainly based on modeling the matching of elementary conditions @xcite @xcite @xcite @xcite in either a sequential encoding or graph-based manner. however, by tackling the cmr task with two divided sub-tasks, the corresponding division of the optimization on decision making sub-task and the optimization on the question generation sub-task may result in problems including error propagation, thus hindering further performance advance. @xcite has shown that transferring some knowledge between the training of two subtasks is beneficial for better performance. however, reducing the gap between two sub-tasks to achieve an end-to-end optimization cmr task still needs further and more comprehensive attempts. in this work, we propose a completely unified end-to-end framework for conversational machine reading tasks (unicmr 1 ) to tackle the division of optimization challenge by formulating the cmr/or-cmr task into a single text-to-text task. our contributions are summarized as follows: ## problem formulation as shown by figure 1 , in traditional cmr task, the machine will be given: user scenario s, user initial question q, a gold rule text r, and dialogue history d := {(q 1 , a 1 ), (q 2 , a 2 ), . . . , (q n , a n )} which consists of n follow-up question-answer pairs. the machine is required to do the two subtasks: beyond cmr, open-retrieval conversational machine reading (or-cmr) @xcite further mimics the more challenging second scenario, which is the focus of this work. as shown by figure 1 , the difference between the cmr and or-cmr lies in the rule text part r. in cmr, the machine is provided with a gold rule text in a closed-book style. while in or-cmr, the machine needs to retrieve rule texts from a knowledge base in an open-retrieval style alternatively. the machine is given a knowledge base b containing rule texts. therefore, under the or-cmr setting, the machine needs to first retrieve m rule texts r 1 , r 2 , . . . , r m to complete the input for the same downstream decision making and question generation sub-tasks. ## framework our model is composed of two main modules: a retriever and a text-to-text encoder-decoder model. the retriever is applied to retrieve rule texts r 1 , r 2 , . . . , r m from a given knowledge base b. the text-to-text encoder-decoder model will take in the preprocessed textual input and generate the textual answer directly as a whole. subsequent extraction methods will be applied for decision making and question generation sub-tasks to obtain the predictions for each sub-task respectively. ## analysis 6.1 number of retrieved rule texts the model performance under different choices of the number of retrieved rule texts is shown in in terms of decision making, our model is quite stable in seen test dataset when the number of rule texts varies. that means our model well captures the useful and trash conditions in rule texts and fulfillment states in dialogue history in the training stage. besides, the unusual boost of question generation performance in the unseen test set might suggest that using more than the necessary number of rule texts possibly pushes the model to gain more power of generalization in the training stage. ## conclusion in this paper, we study open-retrieval setting of the conversation machine reading task and promote a novel framework to first unify the optimizations of the two sub-tasks to achieve optimization synergy. with a retriever module and a parameter-efficient text-to-text encoder-decoder module, we have achieved new sota results in both the cmr and the or-cmr benchmarks. further experiments shows that our unified training form with an endto-end optimization method largely contributes to the advanced performance in decision making and reduces the error propagation to boost question generation performance. it's also shown that our framework well generalize to other backbone models. further qualitative analysis also verifies our framework's effectiveness.