Exploring the Influence of Dialog Input Format for Unsupervised Clinical Questionnaire Filling (2022)

Abstract: In the medical field, we have seen the emergence of health-bots that interact with patients to gather data and track their state. One of the downstream application is automatic questionnaire filling, where the content of the dialog is used to automatically fill a pre-defined medical questionnaire. Previous work has shown that answering questions from the dialog context can successfully be cast as a Natural Language Inference (NLI) task and therefore benefit from current pre-trained NLI models. However, NLI models have mostly been trained on text rather than dialogs, which may have an influence on their performance. In this paper, we study the influence of content transformation and content selection on the questionnaire filling task. Our results demonstrate that dialog pre-processing can significantly improve the performance of zero-shot questionnaire filling models which take health-bots dialogs as input.

## introduction work on question answering (qa) and machine reading comprehension (mrc) mostly focuses on wh-questions of arbitrary types @xcite whose answer can be found in text. the answer can be extractive where a short span of the text is identified as the answer @xcite or it can be abstractive where a free-form answer is generated from the question and some support document @xcite . here, we focus instead on a qa setting where questions are restricted to polar (yes/no) and agreement likert scale (als) questions and where answers are contained in a dialog rather than a paragraph text. as illustrated in figure 1 , this setting is useful for automatic questionnaire filling (aqf) in the medical field. given a dialog between a patient and a health bot, the goal of automatic questionnaire filling is to answer a set of predefined questions from a medical questionnaire (here the pain dialog bot: what is the most difficult for you about your sleep ? patient: i have back pain that prevents me from sleeping. bot: i'm sorry to hear that. how long have you had back pain? patient: since i've been working out, i've had constant back pain at night. bot: do you think pain can last for long? patient: i think it will stop once i stop playing sports. bot: should we let time fix the pain? patient: my doctor thinks that i need to get used to doing sports and that the pain will disappear after a while. ## related work we briefly situate our work with respect to three tasks which have similarities with automatic questionnaire filling namely, machine reading comprehension, question answering and aspect-based sentiment analysis (absa). ## automatic questionnaire filling (aqf) task. given a dialog d and a questionnaire q, the automatic questionnaire filling task consists in providing an answer a i for each question q @xmath0. we address the task in a zero-shot setting (no training data). for evaluation, we provide a test set consisting of 100 dialogs and their associated questions and answers. questionnaire. we consider two types of questions: closed questions (cq) and agreement likert scale (als) questions. cq have three possible answers (yes, no or not applicable, i.e. the dialog does not address the question) and als has five (totally disagree, rather disagree, agree, totally agree, na). as illustrated in figure 1 , questions are reformulated as declarative statements with multiple choice answers. with the emergence of health-bots, aqf can help transform human-bot dialogs into structured data which can be used by physicians to track patients condition. in particular, it can be used to fill in questionnaires such as the pain beliefs and perceptions inventory (pbpi, @xcite ) questionnaire which includes 16 questions and is standardly used in the context of clinical studies. collecting dialogs that include information for all of these questions is a difficult task however. to facilitate data collection for the creation of the test set, we therefore decrease the number of questions by selecting five questions out of sixteen. because the questions in the pbpi are often very similar, and knowing the answer to one of them allows deriving the answer to others, we chose questions that are semantically distinct from one another. the list of all pbpi questions is given in appendix c and the five selected questions are indicated in bold. test data. to evaluate our approach, we create a test set of 100 dialogs and their associated question/answer pairs. the creation of the test data involves first, collecting human-bot dialogs and second, extracting answers to the pbpi questions from the collected dialogs. collecting dialogs. we collect the dialogs using the amazon mechanical turk platform and asking turkers to interact with the combot health bot @xcite while behaving as if they had chronic pain issues. to avoid turkers introducing the pbpi questions verbatim in the dialog, they were given a list of topics to be mentioned rather than the questions themselves (see details in appendix d). in this way, we ensure that the collected dialogs address the questions to be answered while encouraging their diversified paraphrasing during the conversation. turkers received bonuses each time they mention a topic. turkers were also given the ability to modify the bot utterance in order to redirect the conversation more easily: they could reject the current candidate in which case, the turn with the next highest confidences score would be displayed by the bot. more information about turkers payments is provided in the ethic section (sec. a). details of the instructions given to the turkers and a screenshot of the annotation interface are given in the appendix. ## approach following toudeshki et al. (2021), we model question answering as an nli task where the premise is derived from the dialog, the hypothesis from the question and the answer from the nli result. given a question and a dialog, our model, illustrated in figure 2 , answers the question in three steps as follows. deriving an nli premise from the dialog. the nli premise is derived from the input dialog using first, content transformation and second, content selection. as detailed in section 5, we experiment with different ways of transforming and selecting content. ## nli-oriented dialog pre-processing we consider different ways of transforming and selecting dialog content. we also study the impact of the nli model used, comparing deberta, the model used in @xcite , with roberta @xcite , the model used in our approach. the deberta model (he et al., 2020) foot_1 extends the bert architecture with two innovative techniques: disentangled attention mechanism and an enhanced mask decoder. we compare aqf models with and without pre-processing and based on roberta vs. deberta, and find that whereas, when no pre-processing is applied, a deberta model generally outperforms a roberta-based model, the reverse is true when pre-processing is applied. this shows that while the improved deberta-based, nli model helps bridge the gap between dialog and text, explicit pre-processing still yields better results. ## results we evaluate our approach using macro and weighted f1 score. figure 4 shows the results for all combinations of our content transformation and selection methods 6 . improvement over the baseline. comparing our best model (ct answer , cs nli ) with the nopreprocessing ct null , cs null baseline, we see (figure 4 ) that pre-processing can multiply the two turns bot: do you feel anxiety or stress during nights awakenings ? patient: i feel stressed during night awakenings although i am not feeling guilty about being in pain. generated summary patient feels stressed during night awakenings although he's not in pain. ## conclusion in this paper, we studied how dialog pre-processing can impact the task of filling medical questionnaires based on patient-bot interactions. our experimental results show that converting pairs of adjacent turns to declarative sentences and selecting input units based on their entailment relation with the question can significantly enhance performance. figure 9: interface 13