Combining Open Domain Question Answering with a Task-Oriented Dialog System (2021)

Abstract: We apply the modular dialog system framework to combine open-domain question answering with a task-oriented dialog system. This meta dialog system can answer questions from Wikipedia and at the same time act as a personal assistant. The aim of this system is to combine the strength of an open-domain question answering system with the conversational power of task-oriented dialog systems. After explaining the technical details of the system, we combined a new dataset out of standard datasets to evaluate the system. We further introduce an evaluation method for this system. Using this method, we compare the performance of the non-modular system with the performance of the modular system and show that the modular dialog system framework is very suitable for this combination of conversational agents and that the performance of each agent decreases only marginally through the modular setting.

## creating a combined dataset for question answering and intent recognition to our knowledge, no dataset exists to evaluate a tods and a qa system at the same time. therefore, we combined the squad and clinc150 datasets to form a single dataset for the evaluation of the combined system. beyond the original labels from clinc150 (intents) and squad (answers), each sample has an additional label for the module selection which we call the true module. one must note that due to the nature of the mds, we need to train each module on its own. drqa is already pre-trained on the squad training dataset and we did not retrain it. we kept 50% of the clinc150 samples to train the frankenbot and call this dataset train f . we then train the module selection on the train m s part of the dataset. for parameter selection, we reserve valid m s samples. finally, we use the dataset test f ull for the evaluation of the full system. 1 shows statistics about this dataset. we used 11,250 samples to train the frankenbot tods (train f ). we do not need a training set for drqa in our dataset and use the development subset of squad only for train f , valid m s, and test f ull . we reserved 10,534 samples to train and validate the module selection. with a 75-25 split, it results in 7,900 samples for training (train m s ) and 2,634 samples for testing (valid m s ). for the evaluation of the full system, we reserved 10,606 samples (test f ull ). we aimed for an equal amount of samples from clinc150 and squad in the train ms , test ms , and test full sections of the dataset and therefore randomly subsampled the clinc150 dataset. the squad dataset contains multiple questions for each context, e.g. it contains 810 questions related to a short paragraph about a super bowl game. if we split the squad dataset randomly, the module selection might overfit on such statistical cues, learning that the word super bowl is a hint that this utterance is aimed at the odqa system. therefore, we did not assign the squad questions to the datasets at random, but split it along those contexts so that each context appears in either train m s or test f ull , but not in both. while the input questions of squad use correct casing, the user utterances of clinc150 use a mix of correct casing and lower casing. furthermore, clinc150 uses punctuation marks only sometimes while squad always uses punctuation marks, mostly question marks. this makes our dataset unrealistic because it leads to distinguishing criteria which will not occur in real world data. we removed these differences to make the utterances more uniform and therefore more realistic by lowercasing all user utterances and removing all punctuation marks. many questions from squad can be answered only when the context of the question is known. for example the question "who approved of this plan?" is only answerable with the context paragraph at hand. drqa retrieves the context from wikipedia and therefore cannot answer this question. to get a more realistic impression of the performance of the drqa module, we manually annotated 100 questions from test f ull that can be answered in the odqa scenario. we published the dataset under the creative commons attribution share alike license cc-by-sa 4.0 under this link 3 . ## modular dialog system figure 1 shows the architecture of the mds that we used in our experiments. it contains two modules: the odqa system drqa and the tods frankenbot. the module selection component decides for each user utterance which module will answer the 3 link will be available in camera ready version utterance. we formulate this as a text classification task with the candidate modules as target classes, i.e., the module selection predicts each incoming user utterance as either drqa or frankenbot. we used bert with a sequence classification head @xcite for this classification task. it is trained on dataset train ms and evaluated on test ms . ## results and discussion ## conclusion we used the mds framework to combine tods and odqa using the example of drqa and frankenbot. using this framework, one can extend the capabilities of odqa with the conversational capabilities of a tods. further, we introduced an evaluation method that a) evaluates the performance of module selection and b) compares the performance of the underlying odqa and tods systems in the modular and in the non-modular setting. the evaluation showed that drqa and frankenbot work very well together as a mds. the mds introduces only a minimal additional error. we believe that the mds framework is especially suitable for practical applications because one can extend an existing tods with a odqa system or vice versa easier using mds compared to a framework that performs tods and odqa together in a joint model. although we did not prove it, we expect that when we exchange one of the modules, e.g. the frankenbot system with rasa foot_3 , google dialogflow or ibm watson assistant foot_4 , the performance of the mds will change only in that module. we present this evaluation framework for our specific use case, but we expect it to generalize to other settings as well such as using more than two modules. it can also work with other performance measures than f1 scores, although one needs to think about how to calculate a joint score out of different scores and how to deal with errors of the module selection in the modular scenario.