Cascaded Span Extraction and Response Generation for Document-Grounded Dialog (2021)

Abstract: This paper summarizes our entries to both subtasks of the first DialDoc shared task which focuses on the agent response prediction task in goal-oriented document-grounded dialogs. The task is split into two subtasks: predicting a span in a document that grounds an agent turn and generating an agent response based on a dialog and grounding document. In the first subtask, we restrict the set of valid spans to the ones defined in the dataset, use a biaffine classifier to model spans, and finally use an ensemble of different models. For the second sub-task, we use a cascaded model which grounds the response prediction on the predicted span instead of the full document. With these approaches, we obtain significant improvements in both subtasks compared to the baseline.

## introduction unstructured documents contain a vast amount of knowledge that can be useful information for responding to users in goal-oriented dialog systems. the shared task at the first dialdoc workshop focuses on grounding and generating agent responses in such systems. therefore, two subtasks are proposed: given a dialog extract the relevant information for the next agent turn from a document and generate a natural language agent response based on dialog context and grounding document. in this paper, we present our submissions to both subtasks. in the first subtask, we focus on modeling spans directly using a biaffine classifier and restricting the model's output to valid spans. we notice that replacing bert with alternative language models results in significant improvements. for the second subtask, we notice that providing a generation model with an entire, possibly long, grounding document often leads to models struggling to generate factually correct output. hence, we split the task into two subsequent stages, where first a ground-ing span is selected according to our method for the first subtask which is then provided for generation. with these approaches, we report strong improvements over the baseline in both subtasks. additionally, we experimented with marginalizing over all spans in order to be able to account for the uncertainty of the span selection model during generation. ## related work recently, multiple datasets and challenges concerning conversational question answering have been proposed. for example, @xcite introduced sharc, a dataset containing ca. 32k utterances which include follow-up questions on user requests which can not be answered directly based on the given dialog and grounding. similarly, the coqa dataset @xcite provides 127k questions with answers and grounding obtained from human conversations. closer related to the dialdoc shared task, the task in the first track of dstc 9 @xcite was to generate agent responses based on relevant knowledge in task-oriented dialog. however, the considered knowledge has the form of faq documents, where snippets are much shorter than those considered in this work. pre-trained trained language models such as bart @xcite or roberta @xcite have recently become a successful tool for different kinds of natural language understanding tasks, such as question answering (qa), where they obtain state-of-the-art results @xcite . naturally, they have recently also found their way into task-oriented dialog systems @xcite , where they are either used as end-to-end systems @xcite or as components for a specific subtask @xcite . ## task description the task of dialog systems is to generate an appropriate systems response u t +1 to a user turn u t and preceding dialog context u t -1 1 := u 1 , ..., u t -1 . in a document-grounded setting, u t +1 is based on knowledge from a set of relevant documents d ⊆ d, where d denotes all knowledge documents. @xcite identify three tasks relevant to such systems, namely 1) user utterance understanding; 2) agent response prediction; 3) relevant document identification. the shared task deals with the second task and assumes the result of the third task to be known. they further split this task into agent response grounding prediction and agent response generation. more specifically, one subtask focuses on identifying the grounding of u t +1 and the second subtask on generating u t +1 . in both subtasks exactly one document @xmath0. each document consists of multiple sections, whereby each section consists of a title and the content. in the doc2dial dataset, the latter is split into multiple subspans. in the following, we refer to these given subspans as phrases in order to avoid confusing them with arbitrary spans in the document. ## data the shared task uses the doc2dial dataset @xcite which contains 4,793 annotated dialogs based on a total of 487 documents. all documents were obtained from public government service websites and stem from the four domains social security administration (ssa), department of motor vehicles (dmv), united states department of veterans affairs (va), and federal student aid (studentaid). in the shared task, each document is associated with exactly one domain and is annotated with sections and phrases. the latter is described by a start and end index within the document and associated with a specific section that has a title and text. each dialog is based on one document and contains a set of turns. turns are taken either by a user or an agent and described by a dialog act and a list of grounding reference phrases in the document. the training set of the shared task contains 3,474 dialogs with in total 44,149 turns. in addition to the training set, the shared task organizers provide a validation set with 661 dialogs and a testdev set with 198 dialogs which include around 30% of the dialogs from the final test set. the final test set includes an additional domain of unseen documents and comprises a total of 787 dialogs. documents are rather long, have a median length of 817.5, and an average length of 991 tokens (using the bart subword vocabulary). thus, in many cases, truncation of the input is required. ## experiments we base our implementation foot_0 on the provided baseline code of the shared task foot_1 . furthermore, we use the workflow manager sisyphus @xcite to organize our experiments. for the first subtask, we use the base and large variants of roberta @xcite and elec-tra @xcite instead of bert large uncased. in the second subtask, we use bart base instead of the large variant, which was used in the baseline code, since even after reducing the batch size to one, we were not able to run the baseline with a maximum sequence length of 1024 on our nvidia gtx 1080 ti and rtx 2080 ti gpus due to memory constraints. all models are fine-tuned with an initial learning rate of 3e-5. base variants are trained for 10 epochs and large variants for 5 epochs. we include agent follow-up turns in our training data, i.e. such turns u t made by agents, where the preceding turn u t-1 was already taken by the agent. similar to other agent turns, i.e. where the preceding turn was taken by the user, these turns are annotated with their grounding span and can be used as additional samples in both tasks. in the baseline implementation, these are excluded from training and evaluation. to maintain comparability, we do not include them in the validation or test data. for evaluation, we use the same evaluation metrics as proposed in the baseline. in the first subtask, exact match (em), i.e. the percentage of exact matches between the predicted and reference span (after lowercasing and removing punctuation, articles, and whitespace) and the token-level f1 score is used. the second subtask is evaluated using sacrebleu @xcite . ## conclusion in this paper, we have described our submissions to both subtasks of the first dialdoc shared task. in the first subtask, we have experimented with restricting the set of spans that can be predicted to valid phrases, which yields constant improvements in terms of em. furthermore, we have employed a model to directly hypothesize entire spans and shown the benefits of combining multiple models using bayesian model averaging. in the second subtask, we have shown how cascading span selection and response generation improves results when compared to providing an entire document in generation. we have compared marginalizing over spans to just using a single span for generation, with which we obtain our best results in the shared task.