Detecting Urgency in Multilingual Medical SMS in K enya (2022)

Abstract: Access to mobile phones in many low- and middle-income countries has increased exponentially over the last 20 years, providing an opportunity to connect patients with healthcare interventions through mobile phones (known as mobile health). A barrier to large-scale implementation of interactive mobile health interventions is the human effort needed to manage participant messages. In this study, we explore the use of natural language processing to improve healthcare workers’ management of messages from pregnant and postpartum women in Kenya. Using multilingual, low-resource language text messages from the Mobile solutions for Women and Children’s health (Mobile WACh NEO) study, we developed models to assess urgency of incoming messages. We evaluated models using a novel approach that focuses on clinical usefulness in either triaging or prioritizing messages. Our best-performing models did not reach the threshold for clinical usefulness we set, but have the potential to improve nurse workflow and responsiveness to urgent messages.

## introduction in many low-and middle-income countries, access to healthcare is limited and unaffordable. interactive short message service (sms) communication with healthcare workers has shown great potential to promote access to care in such contexts by providing remote information and support @xcite . one such system is the mobile solutions for women and children's health (mobile wach) platform, an interactive semi-automated platform designed to connect pregnant and postpartum women to healthcare workers through sms @xcite @xcite @xcite . studies using this platform have reported significant impacts on health outcomes like breastfeeding and postpartum contraception @xcite . while promising, a major limitation of mobile health interventions is the human effort required to manage messages. nurses involved in the mobile wach platform received hundreds of messages per day @xcite . many of these messages did not require immediate responses; however, nurses couldn't distinguish between urgent and non-urgent messages without reading them. natural language processing (nlp) can potentially be used to automatically triage and prioritize incoming messages. this may significantly improve worker efficiency and improve reliability of the healthcare system in low-resource settings @xcite . in recent years, researchers have used nlp for content analysis of incoming messages in digital health interventions @xcite and for analyzing messages in mental health discussions @xcite . this paper explores the possibility of a clinically useful model to detect urgent participant messages in an interactive mobile health system in kenya. our study focuses on a dataset drawn from the mobile wach neo studies @xcite . the dataset contains real-world, informal messages in multiple low-resource languages (swahili, luo and sheng) and english. we explore methods for handling the unique challenges presented in our dataset, including additional pretraining @xcite and adding prior conversation context. we tested several approaches to classifying urgency in labeled participant messages, with a focus on classifiers that could have clinical utility in improving healthcare worker workflow. models based on mbert can achieve performance levels close to our threshold of clinical usefulness, suggesting such systems could be useful to healthcare workers in the future. we discuss our findings and next steps for integrating such nlp models into real-world systems that could significantly improve global healthcare delivery. ## task definition we aim to introduce models that classify messages based on urgency as a way of prioritizing or triaging messages for nurses. specifically, given a message, our task is to identify if the message requires immediate nurse attention (urgent) or can be looked at later @xcite . figure 1 summarizes the mobile wach system and nlp task. because of the sensitivity of all participant messages in this context, our task is not intended to replace nurses by filtering participant messages or generating responses. ## dataset our data consists of messages, from the mobile wach neo pilot (messages sent between 05-12-2017 and 20-02-2019 ) @xcite and mobile wach neo rct (messages sent between 09-09-2020 and 04-05-2022) studies @xcite . messages were exchanged between pregnant/postpartum women, nurses and the automated mobile wach system. the mobile wach neo pilot dataset consists of a total of 58,834 messages that were exchanged between 800 participants, the automated system and 2 nurses. the mobile wach neo rct had a total of 161,735 messages that were exchanged between 1,724 participants, the automated system and 12 nurses. therefore, the combined dataset consisted of 220,560 messages from 2,523 participants and 14 nurses (after cleaning). automated messages were sent to participants weekly during pregnancy until 38 weeks gestation, then 2 messages daily for the first 2 weeks after delivery, and then every 2 days for 6 weeks followup post delivery. participants could send messages to the system at any time. nurses in the study manually replied to participant messages. these nurses had the same training and qualification as nurses in the public health facilities, however, they were employed by the study and did not have routine care provision responsibilities outside of study. a total of 112,220 (50.9%) messages were sent by the mobile wach system, 65,572 (29.7%) by participants and 42,768 (19.4%) by nurses ( the dataset we present here is typical of how language is used in kenya @xcite . for instance, swahili words used by participants in nairobi may have different connotations from the same word in standard swahili or swahili used in western kenya. it is worth noting that this dataset also contains languages (sheng and luo) not commonly included in training for multilingual transformer-based models like mbert @xcite . ## classifying urgency we defined the task of predicting message urgency as a binary classification task. we tested two modeling approaches. our first approach was a penalized logistic regression (penalty: l2, maximum iterations: 570) with bigram lexical features as input. the bigram features were extracted from uni-and bi-grams from the messages using scikitlearn's count vectorizer @xcite . in our second approach, we evaluated a fine-tuned multilingual bert model (mbert) @xcite . mbert was pretrained in 104 languages including english and swahili. linear models have been used as a baseline in mobile health classification tasks @xcite , and mbert is a strong multi-lingual text classification model. ## evaluation while f1 score is a common classification evaluation metric, clinically useful systems may not require both high recall and high precision to improve healthcare worker workflow. we visualize the trade offs of models' precision and recall with precisionrecall curves. we can use these pr curves to pick potential models that could be clinically useful by defining two regions in the graph: a triage region and a prioritize region. a precision-recall curve shows the trade offs of models' precision and recall across a range of classification thresholds. this allows us to visualization of trade-offs of the results of the system between high precision and high recall. we can use models' precision-recall curves to visualization of trade-offs of high precision and high recall and pick potential models that could be clinically useful by defining two regions in the graph: one region for a triage model and the other for a prioritize model (see figure 2 ). we defined three potential model use cases and their evaluation criteria: 1) triage, 2) prioritize, or 3) combination. most machine learning models for binary classification output a real number between 0 and 1 and use 0.5 as the default threshold for classification. while in most scenarios this threshold is sufficient, in our case it is helpful to examine the model performance with a range of thresholds which might be better suited for different scenarios (e.g., triaging or prioritizing messages). below we define the evaluations for these regions ( §5.2 & 5.1) and their combination ( §5.3). ## results ## discussion & conclusion consistent with prior literature @xcite , our results showed that performing additional pretraining boosts performance. our evaluations show that our modeling approaches have the potential to support healthcare workers in a unique low-resource and multilingual setting, though more work must be done to have the models achieve clinical usefulness based on our measures. moving forward, to improve performance of these models, future studies could look into how to optimize the models when the dataset is skewed for nonurgent messages (as is the case currently). another approach would be to explore models explicitly trained on the languages in our dataset, for example, models trained on swahili datasets or codeswitched languages from east africa @xcite and (flax community). we also plan to validate our highest-performing models with healthcare workers and implement a model in a pilot context similar to the mobile wach sms system.