H indi Chatbot for Supporting Maternal and Child Health Related Queries in Rural I ndia (2023)

Abstract: In developing countries like India, doctors and healthcare professionals working in public health spend significant time answering health queries that are fact-based and repetitive. Therefore, we propose an automated way to answer maternal and child health-related queries. A database of Frequently Asked Questions (FAQs) and their corresponding answers generated by experts is curated from rural health workers and young mothers. We develop a Hindi chatbot that identifies k relevant Question and Answer (QnA) pairs from the database in response to a healthcare query (q) written in Devnagri script or Hindi-English (Hinglish) code-mixed script. The curated database covers 80% of all the queries that a user of our study is likely to ask. We experimented with (i) rule-based methods, (ii) sentence embeddings, and (iii) a paraphrasing classifier, to calculate the q-Q similarity. We observed that paraphrasing classifier gives the best result when trained first on an open-domain text and then on the healthcare domain. Our chatbot uses an ensemble of all three approaches. We observed that if a given q can be answered using the database, then our chatbot can provide at least one relevant QnA pair among its top three suggestions for up to 70% of the queries.

## introduction with inequality in healthcare access across urban and rural parts of india, pregnant and postpartum women in rural areas suffer from low access to healthcare due to limited time with healthcare professionals, language barriers in doctor-patient communication, and societal barriers. in resourceconstrained environments, digital support groups are a common platform to seek information about various maternal and child healthcare-related issues @xcite . the moderators of such support groups are overburdened with enormous queries and find it challenging to provide answers timely. moreover, group members often ask their health queries in regional languages such as hindi or hinglish foot_0 . given the doctor-to-population ratio of 4.8 doctors per 10000 people in india @xcite , the scalability of such healthcare interventions involving doctors becomes challenging @xcite . thus, it presents an opportunity to extend informational support to pregnant and postpartum women through a chatbot that can answer their written queries in their local language. chatbots are used in various domains, from railways ticket reservations to food delivery foot_1 . chatbots have taken up different roles in healthcare, such as psychotherapists, nurses, doctors, and medical consultants @xcite . chatbots have the potential to act as the first point of contact for women seeking answers for maternal and child healthcare-related queries, especially in resourceconstrained environments @xcite . in this work, we explore the potential of a chatbot to provide accurate healthcare information by retrieving the best matching faqs with their corresponding answers @xcite . we developed a chatbot that provides k most relevant faqs with their corresponding answers (qna pairs) in response to a healthcare query. the chatbot uses a curated database of qna pairs in the hindi language with answers vetted by healthcare professionals. our chatbot can process user queries written in latin script (native script for english) and devanagari script (native script for hindi). figure 1 illustrates the overall architecture of the proposed chatbot. for evaluation, we obtained a set of healthcare queries from asha workers foot_2 . in this paper, we discuss different algorithmic approaches to developing chatbots and the efficiency of these approaches in providing relevant qna pairs. the three primary approaches used in this work are (i) the rule-based method, (ii) sentence embeddings, and (ii) paraphrasing classifiers. an ensemble model of all three primary approaches was found to be performing better than other methods. we release the source code of our chatbot to encourage future research in this direction foot_3 . ## related work earlier works on developing chatbots in healthcare using ai started with user query reformulation and using knowledge from search engines @xcite . they were made for the english language, and the same techniques could not be used for hindi speakers due to the scarcity of resources. @xcite aimed to develop a faq retrieval system for the unstructured english language written as a shorthand for sms by the indian population. it relied on character-level features to calculate the sentence similarity scores. initial works on building a qna system for the hindi language were restricted to exploiting information from shallow speech features like pos tags @xcite . in constructing an automatic question-answering system for english-hindi code-switched language (also known as hinglish), the word-level translation of code-switched queries to english queries was a common practice due to a lack of resources in the hindi language @xcite . such approaches fail to gen-eralize because hindi-to-english word-level translations are highly dependent on the position of the hindi word in the sentence @xcite . previously cross-lingual word embeddings have been used to solve a healthcare qna system in lowresource african languages @xcite . it has been empirically shown that fine-tuned machine learning models using embeddings from pre-trained transformer-based encoders like bert outperform many other traditional ai models on various tasks @xcite . earlier works have shown the efficiency of bert-based models in measuring sentence similarity for faq retrieval tasks @xcite . in this paper, we compared the performance of different approaches for measuring sentence similarity between hindi sentences from the maternal healthcare domain. for a given user query (q), the most similar question (q) and its corresponding answer (a) are fetched from the asha-faq database, which is described in the next section. ## data description we collected data from four prior studies by taking permission from the authors @xcite @xcite . the data consists of hundreds of pairs of questions and answers (in audio and text modality), as asked in the real world by community health workers and pregnant and postpartum women regarding maternal and child health issues. health experts have provided the answers to these questions. the audio data was transcribed and annotated with the help of two healthcare professionals. both annotators had a bachelor degree in medicine and surgery, a master's in public health, and experience working in maternal and child health. the two annotators manually transcribed each session in the devanagari script. in this work, annotations were performed using an online transliteration tool foot_4 and audino @xcite . more than 18 hours of audio in the healthcare domain were transcribed to obtain 1150 question-answer (qna) pairs. subsequently, we received 217 maternal health question-answer pairs from yadav et al. @xcite and added them to our asha-faq database resulting in a total of 1365 unique questions and 1338 unique answers 6 . due to the covid-19 pandemic, ai model field testing was not feasible. therefore, to evaluate the models on real-time data, a total of 336 new user queries (q) were collected from asha workers with the help of a non-governmental organization (ngo) partner. we requested asha workers to provide us with queries that they frequently encounter. with the help of public health professionals (with a master's degree in public health), the authors annotated these 336 queries with relevant questions from the asha-faq database. for each query q, authors identified completely and partially matching qna pairs from the asha-faq database. both types of matching (complete and partial) have been treated as relevant in this work. it has been found that, among 336 queries, 270 user queries had at least one relevant question in the database. hence, the coverage of the asha-faq database is 80% in our experiment. the 270 questions, as mentioned above, will be treated as the hold-out test set to evaluate the performance of different faq-retrieval approaches used by our chatbot. in order to train a deep-learning model to calculate the sentence similarity score between two hindi sentences, we scraped hindi news articles from the inshorts website foot_6 . each data point (d i ) in the scraped dataset (d) consisted of news article text (t i ), its headline (h i ), a summary of the text (t s i ), and a paraphrased headline (h p i ). we collected more than 17k data points in our dataset. for a negative (or not-paraphrased) headline of h i , a random headline is chosen from the paraphrased headlines foot_7 . our inshorts dataset contains 35k hindi sen-tences, from the news domain, classified into two classes (paraphrased or not-paraphrased), with equal representation of both classes. we are releasing the scraping scripts and hyperlinks to the news articles in the repository mentioned above. to the best of our knowledge, it is the most expansive dataset available for paraphrase detection in the hindi language. since the inshorts dataset is from the open domain (news), we constructed a question paraphrase dataset in the healthcare domain. we manually paraphrased questions from the asha-faq database and treated them as positive examples of paraphrases. random sentences were taken as negative examples. the dataset thus created is called the ashaqs dataset, and it contains about 1500 healthcare-related question pairs classified into two classes (paraphrased or not-paraphrased) in a balanced manner. the performance of different faq retrieval models is compared using five information retrieval evaluation metrics, namely: mean average precision (map), mean reciprocal rank (mrr), success rate (sr), normalized discounted cumulative gain (ndcg), and precision at 3 (p@3) @xcite . success rate is the simplest to understand because it represents the percentage of user queries for which at least one relevant suggestion was given in the top-k suggestions. ## methodology our work aims to take input as a user query (q) and produces an output as top-k most relevant qna pairs from the asha-faq database. therefore, the given task is modeled as a faq retrieval problem. we tried to solve this faq retrieval problem through three primary approaches. results from best-performing approaches are taken to form an ensemble method. all three of our approaches are able to convert latin script in user input query to devanagari script. we used the indic-trans library for the transliteration @xcite . ## results it has been observed that, among the top-3 suggestions, dtp gave at least one relevant suggestion only in 27.1% of user queries in the hold-out test set. we explored the possible reasons for its failures and found out that the method could not handle the polysemous nature of words. for example, dtp considers शु गर (sugar) and डायिबटीज (diabetes) as entirely different words. however, the two words are interchangeably used in the indian subcontinent to describe a prevalent disease called diabetes mellitus. we tried to solve the polysemous word problem by maintaining buckets of such words. whenever a single word from a bucket is encountered in either q or q i , the rest of the words from the bucket are added to the sentence. expanding the query in such a manner is called query-expansion (q-e) in automatic question answering @xcite . it is shown to improve the dtp method by giving relevant suggestions in 59% of the user queries. multiple encoders were used to build the sp c model. it was observed that the bert-basemultilingual-cased (mbert) encoder by @xcite , gave better results than other pretrained multilingual encoders. moreover, finetuning sp c model with three linear layers on top of the encoder resulted in the best performance. since rogers et al. @xcite suggested that early layers of encoders contain more syntactic information, we froze the early layers of the encoder. we observed more stable results across different random seeds. we first fine-tune the resulting model on the open domain inshorts dataset and then fine-tune it further on the ashaqs dataset in the healthcare domain. we observed that it boosted the performance of sp c on the hold-out test-set in the fourth epoch, as shown in figure 2 . the sp c approach majorly dominates the inference time of the ensemble method. it was observed that, with a gpu-enabled server, the ensemble chatbot gives real-time suggestions in 4 seconds and consumes a memory of 2.3 gb on the gpu. however, the chatbot takes a few minutes to generate top-k suggestions without a gpu and consumes a memory of 6.0 gb of ram. ## limitations in our study, we tested the chatbot on the hindi database, which humans heavily annotated. thus, when the database size is enormous, the scalabil-ity of the annotation approach is a critical question. since the questions and answers could be possible in different languages, it will require considerable effort to translate them and, at the same time, preserve their context. in our study, we observed the success ratio of the developed chatbot to be 70% for hindi queries. however, it is not indicative of its performance in different natural languages. for a given user query (q), the performance of our best approach for the faq-retrieval system is highly dependent on the number of different relevant questions (q) existing in our asha-faq database for the given q. considering the large number of user queries that can be asked in the healthcare field, the small size of our asha-faq database is a significant reason behind the instances where our method fails to suggest relevant questions (q) to the user. moreover, our work does not analyze the quality of answers present in the asha-faq database. hence, a user study would be required to analyze the questions' diversity and the answers' quality in our asha-faq database. ## conclusion and future work in this paper, we presented the development of a chatbot to reduce the workload of healthcare professionals for extending informational support regarding maternal and child healthcare concerns in a resource-constrained environment. we followed a faq-based model to develop our chatbot using a healthcare database curated in hindi. our developed faq chatbot can process hindi user queries written in either the native script of hindi (devanagari) or in the native script of english (latin). we experimented with different faq-retrieval methods to extract the most relevant qna pairs from a faq database. we found that the chatbot has the potential to provide relevant qna pairs for up to 70% queries that our faq database can answer. in the future, we plan to evaluate the bot in the wild with healthcare professionals involved. we plan to evaluate our chatbot with pregnant and postpartum women in a resource-constrained environment to understand the performance of the chatbot in the wild. we also plan to incorporate a healthcare professional to answer questions beyond the chatbot's capacity. the answer obtained from the professional will be further added to the existing qna database for handling future queries, which would improve the chatbot's success rate over time.