Intent Identification and Entity Extraction for Healthcare Queries in I ndic Languages (2023)

Abstract: Scarcity of data and technological limitations for resource-poor languages in developing countries like India poses a threat to the development of sophisticated NLU systems for healthcare. To assess the current status of various state-of-the-art language models in healthcare, this paper studies the problem by initially proposing two different Healthcare datasets, Indian Healthcare Query Intent-WebMD and 1mg (IHQID-WebMD and IHQID-1mg) and one real world Indian hospital query data in English and multiple Indic languages (Hindi, Bengali, Tamil, Telugu, Marathi and Gujarati) which are annotated with the query intents as well as entities. Our aim is to detect query intents and corresponding entities. We perform extensive experiments on a set of models which in various realistic settings and explore two scenarios based on the access to English data only (less costly) and access to target language data (more expensive). We analyze context specific practical relevancy through empirical analysis. The results, expressed in terms of overall F-score show that our approach is practically useful to identify intents and entities.

## introduction healthcare is a top priority for every country. people across the world ask millions of health-related queries, hoping to get a response from a domain expert @xcite . these queries mostly deal with medical history of patients, possible drug interactions, disease related concerns, treatment protocols and so on. conversational agents for healthcare play a pivotal role by facilitating useful information dissemination @xcite . in order to understand these * authors contributed equally queries better, practical conversational systems for healthcare need to be developed. however, the primary obstacle in developing such technologies for low-resource languages is the lack of usable data @xcite @xcite . india is a country with a diverse language speaking population suffering from abject poverty and low-economic status @xcite . this linguistic diversity and complex socio-economic situation in india certainly poses significant challenges in developing automatic healthcare systems; and there is a lack of linguistic resources specific to the medical domain. for example, situations such as the patient and the doctor speaking in different languages, is not an uncommon situation in rural india. these individuals are unable to avail the existing systems and facilities which exist mainly in the english language. recent efforts in developing automatic translation systems, even from extremely low resource languages such as ' @xcite , should ideally improve this situation, but there is no extensive study on that front. in order to bridge this language barrier, massively multilingual transformer based language models (mmlm) @xcite have made impressive advancements on a wide range of downstream applications. but the real-world implications of such advancements in the indian healthcare system remain largely unexplored. in this paper, we aim to explore scarcity of the data and study the extent to which the existing language technologies can be leveraged to develop practically useful healthcare systems for the low-resource languages in developing countries. with an aim to answer our research question, we create two different multilingual healthcare datasets, namely, ihqid-webmd and ihqid-1mg. these datasets are created by crawling frequently asked questions from two healthcare websites, webmd and 1mg. these datasets comprise frequently asked questions about drugs, diseases and treatment methods in seven different languages, namely english, hindi, bengali, tamil, telugu, gujarati and marathi. the queries are manually tagged with intent labels and entity tags by domainexperts and translated by native speakers of the corresponding languages. we also collect real world indian hospital queries (annotated) in seven languages to check the empirical effectiveness of our approach. fig. 1 shows an example of a health query belonging to 'treatment' intent class manually translated into three different languages. then we evaluate the performance of state-of-the-art language models (lms), for both english and multilingual setups on our datasets, to answer the questions regarding their deployability and practicality. various experimental configurations (section 4) have been tried on these datasets where we try to figure out the ways of using best technologies through extensive experimentation in two real-world scenarios. first, we assume to have access to only english training queries (less costly) and the test queries are multilingual in nature. we observe that translatetest setup on roberta seems to be a reasonable choice of technology. second, we assume to have access to manually written multilingual training and test queries in the target languages, which is indeed quite expensive in terms of data collection effort. however, back-translation of both train and test queries proves to be a reasonable choice if we have budget of collecting data in target languages. in sum, our contributions are four folds: ## related work we pivot our study of related works into the following buckets -generalised intent and entity detection, entity and intent detection in healthcare, health care in indian languages and multi-lingual healthcare datasets. a) generalised intent and entity detection approaches: @xcite @xcite focus on detecting novel intents in the form of outlier detection. @xcite explore intent classification on legal data. people also work on different detection approaches -few shot @xcite , zero shot @xcite , clustering frameworks @xcite . @xcite @xcite all explore entity detection tasks. @xcite develop a hierarchical multi-task architecture for semantic parsing sentences for cross-domain spoken dialogue systems. most of these approaches are very domain and language specific and thus not very useful for the healthcare domain in indian languages. b) entity and intents in health care: zhou et al. (2021) solve different tasks in smart healthcare. bao et al. (2020) build a chat-bot framework using user intents. bai et al. (2022) aim at incremental medical intent detection. razzaq et al. (2017); amato et al. (2017) develop an e-health application using intent-context relationships. zhang et al. (2017) explore medical query intents. most of the works are done for english and chinese languages and there is no proper architecture for indian multilingual scenarios for intent and entity extraction.. c) health care in indian languages: some researchers focus on indian languages -hindi medical conversation system, medbot @xcite , detecting hindi and english covid-19 posts @xcite , tamil health information (santosh, 2019), bengali health-bot @xcite , telugu covid-19 health information @xcite . but none of the work aims at indian health query datasets and model analysis. @xcite highlights the gaps when using existing state-of-the-art commercial frameworks for nlu tasks in a few asian and african low-resource languages, especially when the goal is to develop conversational agents for healthcare during covid. in our work, we strengthen the claims made in their paper for generic healthcare specific datasets in indian context, and highlight the potential drawbacks of the existing lms. d) multilingual health care dataset: liu et al. (2020) develop meddg (medical dialogue dataset of common gastrointestinal diseases) in chinese. zeng et al. (2020) proposes meddialog, a chinese and english medical dataset, and explores medical dialogue generation tasks. zhang et al. (2021) build a medical intent evaluation dataset in chinese and kim et al. (2022) has constructed a korean health intent dataset. our work differs from the existing research in two ways: 1) we focus on developing a gold standard healthcare nlu dataset in indian languages, 2) cost parameter and availability oriented usage of models for intent detection and entity extraction, and 3) end-to-end evaluation of the state-of-the-art solutions for healthcare in both english and indic languages which leads to interesting implications and generates important future recommendations for the language community. ## strategies of evaluation in this section, we illustrate the strategies of evaluating the state-of-the-art lms on our dataset. our evaluation of these models for healthcare is scoped down to two fundamental nlu tasks: a) intent recognition (section 5.1) b) entity extraction (section 5.2) evaluation setup description: our evaluation of the models has been conducted while keeping in mind about the availability of human-translated monolingual and multilingual training data in two possible real-life scenarios: 1) scenario a: in this setup, we assume to have access to only english training data (less costly) and in 2) scenario b: we assume to have access to manually written training queries in all the target languages (very expensive). during inference/testing, we expect all the queries are in the corresponding target languages. scenario a: setup 1) backtranslated test (s1): [translate-test] here we develop our system by training the models on the english queries, and evaluate the intent detection and entity extraction systems in different languages by automatically backtranslating the test queries into english (e.g. similar to @xcite ). setup 2) zero-shot cross-lingual test (s2): cross lingual transfer learning is a useful methodology used for tasks involving scarce data @xcite . in this setup, the models make use of zeroshot based cross-lingual capabilities from training on the english data (scraped from webmd and 1mg) and use it for inference on test queries in indic languages. setup 3) bridge language backtranslation (s3): here a relatively low-resource language is first translated to an intermediate language and then finally to english. the motivation behind this setup lies in the fact that even though these indic languages belong to different scripts, there are linguistic and morphological similarities among them which may improve the translation to english if they are used as intermediate languages. in this paper, we have considered 'hindi' as the bridge language. this notion of such "bridge" languages has been explored previously in the context of machine translation @xcite and zero/few-shot transfer in mmlms @xcite . ## experiments and results experimental setup: our experiments are conducted on two tesla p100 gpus with 16 gb ram, 6 gbps clock cycle and gddr5 memory. all methods of entity extraction and intent detection took less than 30 gpu minutes for training. we perform a hyperparameter search and report the results of the settings which achieve the best results, and then fixed the same for all the models. the batch size is kept at 16, number of epochs is 10, optimization algorithm used is adamw and the learning rate is 1e-5 with cross-entropy as the loss function. ## discussion and error analysis we categorize the issues in mis-classification and identify two broad themes of the reasons. the primary reason is model prediction error. figure 4 shows the model prediction errors for various intents in different languages. for an example, 'how common is syphilis' is of 'disease' intent category but model wrongly predicts it as 'other' category. another reason is the misclassification due to incorrect translation of the medical entities such as the disease 'uticartia' has been transformed into 'ambat' during backtranslation as shown in figure 5 which is not detected as an entity. so, the backtranslation error leads to intent mis-classification and entity extraction error. we speculate such random absurd behaviour due to the context of the query and languages are semantically different. secondly, there are also certain issues in fluency and grammatical meaning after backtranslation. for instance, 6 https://github.com/indichealth/indic-health-demo figure 5: error in back-translation ## conclusion we focus on developing novel indian healthcare query datasets and propose frameworks to detect intents and extract entities from queries in different indian languages. through extensive experiments on our proposed datasets, we recommend the community to use backtranslation of test queries to english in two real-life scenarios as a reasonable choice when we have access to english training data. however, the same strategy can be applied to both train and test queries if we have the budget of collecting data in target languages. backtranslation of queries using an intermediate bridge language also proves to be a useful strategy in some cases.