Zero-Shot Clinical Questionnaire Filling From Human-Machine Interactions (2021)

Abstract: In clinical studies, chatbots mimicking doctor-patient interactions are used for collecting information about the patient’s health state. Later, this information needs to be processed and structured for the doctor. One way to organize it is by automatically filling the questionnaires from the human-bot conversation. It would help the doctor to spot the possible issues. Since there is no such dataset available for this task and its collection is costly and sensitive, we explore the capacities of state-of-the-art zero-shot models for question answering, textual inference, and text classification. We provide a detailed analysis of the results and propose further directions for clinical questionnaire filling.

## introduction chatbots in healthcare can be used to collect information about the user with different purposes: treatment adherence, monitoring, patient support program, patient education, behavior change, diagnosis @xcite . considering monitoring, patient-doctor conversations have mainly been used for the automation of medical records' creation through the extraction of clinical entities such as symptoms, medications, and their properties @xcite , generating reports @xcite and summaries @xcite . surprisingly, the task of filling clinical questionnaires received less attention. patients fill standard questionnaires during each medical visit, which frequency is usually several weeks. performing this task in an automated way based on serendipitous talk (through a chatbot) opens the opportunity to get updated information more regularly and then monitor the patient more closely and in a seamless way. @xcite were the first to introduce a questionnaire filling task as a classification problem, with the targets in the form of symptom phrases. though, there is plenty of questionnaires that consist of full meaningful assertions or questions. the difference between questionnaire filling and slot filling is in the complexity of the questions that require machine reading comprehension (mrc) and question answering (qa). the goal of a typical mrc task is to process a (set of) text passage(s) and then to answer questions about the passage(s). though usually, multi-choice answer options are semantically different, in the case of questionnaires, the answers are often on the same scale (agree-disagree, often-rare). more concretely, we make the following contributions: ## related work four formats are commonly used for posing questions and answering them: extractive (ex), abstractive (ab), multiple-choice (mc), and yes/no (yn). unifiedqa @xcite ) is a single pre-trained qa model, that performs well across 20 qa datasets spanning 4 diverse formats. @xcite proposed a sentence transformation model which converts question-answer pairs into their declarative forms that allows to solve the task with nli models. on the other hand, @xcite shows how zeroshot-tc models may be successfully used as nli models, being partially inspired by the way @xcite transformed relation classification task into nli problem. questions in multiple-choice race @xcite require holistic reasoning about the text, involving people's mental states and attitudes and the way they express them. @xcite described reasoning categories present in race and showed that if given passage is a dialogue, nli model performs better than qa model. @xcite presented a new medical information extraction task which uses questionnaires to convert medical text data into structured data. their work is based on neural network classifier which makes selection among given options for closed-end type questions to fill out one complete questionnaire using only one model. to our best knowledge, it is a first work to address the problem of filling in clinical questionnaires based on dialogue history. in section 3 we are going to discuss the most common question types in questionnaires, including likert scale which wasn't addressed before in clinical natural language processing. also in section 6 we show state-of-the-art models' ability to solve this task. the task of filling questionnaires from user-bot dialogue history is a very specific sub-field of mrc in nlp and as a result, reduces the chance of availability of such data for training/fine-tuning current models or later for evaluation, specially in medical domain. both dialogues and answered questionnaires based on them are required that makes it difficult to collect such data on a large scale. this low resource setting is conducive to zero-shot approaches. question type questionnaire open question (oq) morin (morin, 1993) qcd (daut, 1983) mos-ss (sherbourne and stewart, 1991) closed question (cq) qpc (c. thomas-antérion, 2004) epices (bihan et al., 2005) agreement likertscale (als) tsk (kori, 1990) pbpi (williams and thorn, 1989) tas-20 (bagby et al., 1994) lot-r (scheier et al., 1994) ieq-cf (sullivan, 2008) jck (karasek, 1985) frequency likertscale (fls) mos-ss (sherbourne and stewart, 1991) mbi (maslach et al., 1997) pcl-s (weathers et al., 1993) had (zigmond and snaith, 1983) sf-12 (ware jr et al., 1996) visual analogue scale (vas) qcd (daut, 1983) dallas @xcite leeds @xcite fabq-w @xcite for each question type present in medical questionnaire, we provide examples of questionnaires and question in ## models in this section, we present the models that can be used in zero-shot setting with selected question types. ## evaluation since for each question type we had different output, we used different evaluation scores. for evaluating the performance of unifiedqa on open-question types, we have used two common evaluation metrics: rouge @xcite and @xcite . rouge counts how many n-grams in the generated response matches the n-grams in the reference answer. since we used abstractive mode of unifiedqa, generated outputs might have different n-grams from reference ones. therefore, using bertscore might show a more realistic evaluation perspective since it computes the semantic similarity between generated and referenced answers based on embeddings. we evaluate closed questions (cq), agreement likert scale (als), frequency likert scale (fls) and visual analogue scale (vas) with macro and weighted f1 score. the results are shown in the the experiments were conducted with a laptop having intel® core™ i7-10610u cpu @ 1.80ghz * 8 and nvidia quadro p520. ## results and discussion detecting unanswerable questions considering all open questions, we can see from impact of number of choices the measuring agreement is the most challenging task from importance of text input from our experiments, we can derive that the models are sensitive to the input text format and noise. also, the different models are sensitive to the different data preprocessing technics. they may include using speaker tags, punctuation cleaning, selecting a subset of input text on some criteria. such experiments should show what kind of preprocessing may boost performance without unnecessary data collection and training. these results may be a contribution to green and sustainable nlp. ## conclusion in this paper, we introduced the approaches for clinical questionnaire filling in a zero-shot setting for the five question types most used in clinical studies. also, we describe the data collection process for their evaluation. with the results, we show that this task is not easy to solve. there is a type of questions which are more wellknown in multi-hop reading comprehension. for such type of questions, there is a need to properly integrate multiple pieces of evidence to answer them. @xcite investigates graph convolutional network (gcn) and graph recurrent network to perform evidence integration. as future work, we plan to take advantage of graph convolutional networks (gcns) to improve the textual entailment for questionnaire filling. it would be possible due to enriching the model with knowledge graphs both in the open domain and close domain (medical). another direction to explore is text transformations. for example, we plan to transform statementquestion into cloze form by masking the most important word and providing it as one option of the answers, use common-sense knowledge and graph structure for better reasoning.