Supporting Complaints Investigation for Nursing and Midwifery Regulatory Agencies (2021)

Abstract: Health professional regulators aim to protect the health and well-being of patients and the public by setting standards for scrutinising and overseeing the training and conduct of health and care professionals. A major task of such regulators is the investigation of complaints against practitioners. However, processing a complaint often lasts several months and is particularly costly. Hence, we worked with international regulators from different countries (the UK, US and Australia), to develop the first decision support tool that aims to help such regulators process complaints more efficiently. Our system uses state-of-the-art machine learning and natural language processing techniques to process complaints and predict their risk level. Our tool also provides additional useful information including explanations, to help the regulatory staff interpret the prediction results, and similar past cases as well as non-compliance to regulations, to support the decision making.

## introduction nurses and midwives play important roles in the healthcare system as they provide highly skilled and often complex care in both hospitals and communities. to protect and prioritise the safety of the public from harmful practices, most countries have specific health professional regulators to set rules, monitor and shape the practice of nurses and midwives. when concerns over a nurse or midwife's practice are raised, a formal complaint can be submitted to the regulator, and investigations will be performed to decide further actions (e.g., warnings to the nurse/midwife in question, or even suspension of their practice). as the investigation results have significant impact on the practitioners' career and reputation, processing complaints is highly time-consuming and costly (see @xcite , p49), hence, the need for effective tools to support investigations is crucial. in this paper, we present a decision support system to improve the efficiency of complaints investigation for nursing and midwifery regulators, by employing state-of-the-art machine learning and natural language processing (nlp) techniques with a human-in-the-loop. we worked closely with the uk nursing and midwifery council (nmc foot_0 ), the us texas board of nursing (tbon foot_1 ), and the australian health practitioner regulation agency (ah-pra foot_2 ), to understand their requirements for the system and collect data for training the machine learning models. fig. 1 illustrates the major components and workflow of our proposed system. as new cases arrive, the system processes the corresponding complaints for each case and provides the following results: (i) risk level prediction: each case is labelled as either high or low risk, along with a confidence score, which allows regulators to prioritise the new complaints. (ii) explanations of the risk prediction results, by highlighting the most salient words in the complaint texts that led to the prediction. (iii) similar previous cases, so that users can refer to relevant past cases to make decisions on the current case. (iv) entries in the regulation code that a new complaint is most related to, that can help the regulators quickly link the allegations in the complaints to relevant requirements in the regulation code. a major challenge in developing the system is data sparsity. due to the sensitive nature of the healthcare data and the strict data-sharing policies of the regulators, we had access to a small amount of data (initially 1.2k complaints, later 5.7k complaints) to develop and test our system. to mitigate this problem, we use ensemble methods based risk level prediction (ensemble) lime linking to regulations linking to similar cases new case previous cases regulations predicted risk & probability confidence score explanation confidence prediction similar cases relevant regulations system output on both classical and neural models, including an adapted version of bert @xcite . in addition, to ensure that the predictions made by our system were gender unbiased, we pre-processed the text appropriately and experimented with several bias mitigation techniques. experimental results show that the risk predictions made by the system achieved an accuracy of 0. to the best of our knowledge, this is the first nlp system that supports complaints investigation for nursing and midwifery regulators. ## our system initially, we used 1,241 real cases from one regulator to develop and test our system. each case i consists of multiple fields, falling into three categories: the complaint text t i , in which sensitive information is replaced with its corresponding entity type, e.g., all names are replaced with @xcite ; meta information of the case (c i1 , ..., c ik ), e.g., status of the case, and who submitted the complaint; and the investigation results, including the risk level y i of the case (high or low), and some additional assessment results (a i1 , ..., a im ), e.g., whether serious harm was caused to the patient or not. we understand from our collaborating regulatory agencies that the most essential functionality they need is to be able to predict the risk level of the case, as it allows them to prioritise the high-risk cases and better manage the workload. hence, we formulated the problem as a binary classification task, which takes a complaint t i and its meta-information we developed an ensemble model to predict the risk level ( §3.1) and provided some additional information to further support the decision-making process of the regulator and help them interpret the prediction results ( §3.2). ## system implementation backend. we used flask 1.0.2, a python based web development framework, to develop the backend of the system. we used sqlite 3.34.0 to manage the database, sqlalchemy 1.2.6 for relational mapping, redis 3.5.3 for internal messaging and caching, nonconformist for conformal prediction, and wtforms 2.1 to manage forms. the system receives new complaints in real time and can make predictions either in real time or batch so as to minimise the response time. charts.js 2.5.4. functionalities like tool traversal, event handling, and animation are implemented using jquery 3.5.1. figure 2 shows a screenshot of a result page for a specific complaint using fictitious data. it depicts the complaint text on the left and the predicted risk as well as additional information on the right. the user can provide feedback for the predictions (accept or reject a prediction result, and provide reasons for the same). they can also provide feedback about the relevance of each similar case and regulation code, suggested by the system, to the selected case. ## system evaluation risk level classification results are presented in ## conclusion in this work, we have presented the first system to support complaints investigation for nursing and midwifery regulators. the system exploits state-ofthe-art text classification, summarisation, semantic similarity measurement and nli techniques, and provides different types of information to assist the regulators, including risk level assessment, similar past cases, and non-compliance to regulations. in addition, explanations (in the form of highlighted words) are provided to improve the transparency of the system, and gender debiasing operations are performed to reduce systemic gender biases. feedback received from domain experts confirmed the system's usefulness and potential. we will continue our collaboration with the nursing and midwifery regulatory bodies and collect more labelled data, e.g., relevant case pairs and noncompliance to regulations; this data will help us develop domain-specific sentence similarity measurement and nli models to further improve the performance of the system. we are considering extending the system with additional functionalities, for example, applying active learning @xcite to allow the system learn more efficiently from human feedback and thus be constantly updated online. we also plan to perform additional experiments in control groups with domain experts to test the effectiveness of the system, e.g., by comparing the average time consumed to process a case with and without the use of our system. regulatory bodies in different jurisdictions face similar problems (e.g., long processing time, high cost, and an increase in the number of cases to investigate) and have similar requirements on the functionalities of the system (risk prediction, similar past cases, non-compliance to regulations). hence, we hope this work will inspire more ai/nlp-based decision support systems across different jurisdictions, and encourage more collaborations between the nlp researchers and regulatory bodies in the legal, financial and healthcare sectors.