Risk-graded Safety for Handling Medical Queries in Conversational AI (2022)

Abstract: Conversational AI systems can engage in unsafe behaviour when handling users’ medical queries that may have severe consequences and could even lead to deaths. Systems therefore need to be capable of both recognising the seriousness of medical inputs and producing responses with appropriate levels of risk. We create a corpus of human written English language medical queries and the responses of different types of systems. We label these with both crowdsourced and expert annotations. While individual crowdworkers may be unreliable at grading the seriousness of the prompts, their aggregated labels tend to agree with professional opinion to a greater extent on identifying the medical queries and recognising the risk types posed by the responses. Results of classification experiments suggest that, while these tasks can be automated, caution should be exercised, as errors can potentially be very serious.

## introduction recently, the potential for unsafe behaviour in conversational ai (convai) systems has attracted increasing attention, with a regular series of research workshops dedicated to the topic. 1 while detection and mitigation of certain types of unsafe content such as hate speech and offensive language have received considerable attention (e.g. cercas @xcite @xcite , there exists little work on handling user queries regarding medical advice. this is despite the fact that researchers have identified these topics as among the most important safety issues @xcite , with very serious potential consequences, including loss of life @xcite . @xcite give the example of an end-to-end conversational system providing the following response to a medicine-related user query: user: 'can i mix xanax with alcohol?' system: 'xanax is a benzodiazepine, so yes, you can mix it with alcohol.' -where the drug interaction in question is potentially disastrous. even if a system provides a factually correct answer, it may not be desirable that it provides apparent expertise in such a sensitive subject-an example of 'the imposter effect' @xcite . to mitigate these potential dangers, conversational systems need to be capable of (1) recognising the seriousness of medical queries from users, and (2) controlling the risk level of replies to such prompts. these are important considerations, as the way a system deals with a query concerning, for example, a sprained ankle should likely be different to its response to a life-threatening situation such as heart attack @xcite . crowdsourcing is increasingly common for health applications @xcite . similarly, con-vai researchers use crowdsourcing to collect data for tasks ranging from conversational language understanding (e.g. @xcite to evaluating system outputs (e.g. @xcite , to, indeed, medical questions and answers @xcite . but can knowledge of the dangers posed by medical queries to conversational systems be reliably and safely crowdsourced, or is professional domain expertise required for this task? we address the following research questions: rq1 do crowdsourced medical risk-level labels match domain expert judgements? rq2 according to domain expertise, how safely do current systems respond to medical queries? rq3 how well can the tasks of detecting and grading the seriousness of medical queries and assessing the risk of system responses be automated by machine learning classifiers? our research claims and contributions we propose a risk-graded labelling scheme for handling medical queries based on risk levels for medical chatbots established by the world economic forum (2020) (wef). in collaboration with a healthcare professional, we use this to create a dataset of english language queries sourced from submissions to a specialist medical forum on reddit.com. using these queries, we then probe existing conversational systems and evaluate the safety of their responses using domain expertise. to investigate the extent to which such expertise is required for supervision, we label both the queries and responses, comparing the professional annotations with crowdsourced labels. we perform classification experiments to benchmark the performance of machine learning classifiers at detecting the potentially dangerous queries, and also at identifying the overall risk level of the responses, thus automatically obtaining a risk score that takes both user and system turns into account. these graded outputs can be used by system developers, who may wish to create lower risk (e.g. open-domain general chatbots) or higher risk systems (e.g. specialist medical assistants). we provide analysis of the suitability of the labelling scheme, the difficulty of the annotation task, and the challenges of medical safety for convai. we make the dataset and code publicly available. foot_1 2 related work recently, safety has been highlighted as a major concern for researchers and practioners working on convai @xcite and generative language models @xcite . dealing with queries related to medical advice has been identified as especially important @xcite @xcite . for example, in an analysis of the responses to medical queries by three voice assistants, @xcite found high levels of risk including serious threat to life. despite this, the area of convai for healthcare is growing rapidly, with many systems offering users diagnoses, counselling, and even interventions @xcite . however, there exist few datasets for the task of identifying such risks in convai. @xcite considered medical advice as one of several 'sensi-tive topics' to be avoided by systems. like us, they trained a classifier to recognise medical topics in reddit data. however, they considered all medical queries to be of equal severity and did not address the different levels of risk for system responses. sun et al. (2022) tackled instances of systems dispensing medical advice, training their system to recognise the responses of medics in the patientdoctor conversations of @xcite 's meddialog dataset as being unsafe for general conversational systems to produce. unlike our fine-grained risk-assessment, their labels are binary and do not allow for nuanced safety tuning (see §3.1). the few existing datasets of health-related questions are not in the target language (e.g. @xcite ), or domain (e.g. ben abacha and demner-fushman, 2019). the latter created a corpus of expert-summarised consumer health questions. while these are of appropriate length for dialogues with conversational systems, they are far more formulaic and unnatural than genuine user queries to conversational systems. we therefore create a new english language dataset of medical queries and responses for convai. ## data and method user queries we identified r/askdocs foot_2 as the most likely forum to contain relevant queries, as it is the most active medical subreddit by number of posts and features a high number of posts by verified healthcare professionals, and features medical queries of the sort that users might seek answers to from a conversational agent. we downloaded all submissions (top-level posts) that have been archived on the pushshift database @xcite , collecting the textual content of the submission titles. as, compared to the majority of social media posts, user utterances in dialogues with conversational agents tend to be short (around five tokens (cercas curry et al., 2021)), we use the titles, rather than the longer, usually multi-sentence text from the body of the submissions. we filtered out posts that include images, video, or links to other media as conversational systems do not usually have access to multi-media information. to identify queries, we then used a dialogue act classifier trained on the nps chat corpus @xcite , and then manually filtered out any remaining non-question posts. using the same process, we also collected a similar number of randomly selected submissions to reddit. we appended the negative class label not medical to these instances and added them to the dataset. we removed non-english language posts and did not collect usernames or other metadata. ## classification experiments we trained and tested the classifier using the expert-annotated labels. for both tasks, we finetuned contextual word embeddings from bert, a transformer-based language model @xcite , with default parameters. 4 in addition to common metrics, we report both the macro-and micro-averaged f1 scores and the macro-averaged mean absolute error, which gives an indication of performance on ordinal classification @xcite (where lower scores are better). we 4 implementation details are available in appendix c. for user input, we tested both the binary and multi-class, ordinal settings (described in §3.1). for responses, it may be desirable to separate the safest responses (labelled no information) from both poor quality and riskier outputs. in addition to the above settings, we therefore also tested ternary classification with three classes: irrelevant or nonsensical/no information/medical information. results are promising in the binary settings, with f1 scores well above 0.9 and recall of 0.97 for both queries and responses, indicating few false negatives-arguably the most important factor for safety. performance is considerably poorer in the ordinal setting, particularly for seriousness grading of medical queries, with macro f1 below 0.5 and a very high error rate. this is partly due to the fact that the classifier never predicts the more serious labels, as shown in the confusion matrix in this results in some potentially serious misclassifications in which the seriousness of the situation and riskiness of the responses are under-estimated. for example, the query 'feeling i might faint at any moment, dizzyness, lightheadedness', labelled as a critical situation due to the seriousness of the symptoms and immediacy of the language used, is predicted to be non-serious. similarly, while the response 'i bet you're fine.' is considered to be a diagnosis by the expert annotator, the classifier predicts only general information. ## discussion and conclusion we propose a labelling scheme for the task of handling medical queries in convai, which allows system developers to set acceptable risk levels for their use case. depending on the case, it may be necessary to shift interpretation of the labels. for example, while level 0 may generally be considered to be safer than i-iv, in that no potentially incorrect or harmful information is offered, developers may decide that a system should, in fact, provide some information in a critical medical situation. this is pertinent to the currently available systems we tested, which fare reasonably well in terms of avoiding the highest risk levels, but perform poorly at providing useful general medical information of the type that we would expect to be acceptable in most use cases. comparison of annotations suggests that expertise, rather than the 'wisdom' of the crowd is needed to create datasets for risk grading, although crowdworkers may be reliable enough at the binary task of identifying whether or not an utterance is in the medical domain. one limitation of our data collection methodology is that we do not see many serious or critical queries. while this may be reflected in real world scenarios, where emergency situations are rare, foot_3 it could also be a result of domain variation between reddit data and genuine humanconversational agent dialogues (see § 6 for further discussion). this is also reflected by the classification experiments (cf.