Global Readiness of Language Technology for Healthcare: What Would It Take to Combat the Next Pandemic? (2022)

Abstract: The COVID-19 pandemic has brought out both the best and worst of language technology (LT). On one hand, conversational agents for information dissemination and basic diagnosis have seen widespread use, and arguably, had an important role in fighting against the pandemic. On the other hand, it has also become clear that such technologies are readily available for a handful of languages, and the vast majority of the global south is completely bereft of these benefits. What is the state of LT, especially conversational agents, for healthcare across the world’s languages? And, what would it take to ensure global readiness of LT before the next pandemic? In this paper, we try to answer these questions through survey of existing literature and resources, as well as through a rapid chatbot building exercise for 15 Asian and African languages with varying amount of resource-availability. The study confirms the pitiful state of LT even for languages with large speaker bases, such as Sinhala and Hausa, and identifies the gaps that could help us prioritize research and investment strategies in LT for healthcare.

## introduction the world witnessed one of the worst pandemics in early 2020, covid-19, infecting over 250 million people globally. scientists and technologists from various fields joined hands, lending support to deal with this global crisis. language technology (lt), particularly the conversational agents (aka chatbots), played a crucial role during the pandemic by facilitating correct information dissemination @xcite and early disease screening @xcite . nevertheless, today practically useful chatbots and other benefits of lt are available only in a handful of languages @xcite . despite impressive gains made by the massively multilingual transformer based language models * equal contribution (mmlm) @xcite @xcite @xcite on standard nlp benchmark tasks @xcite @xcite , the realworld implications of such advancements remain largely unexplored. @xcite has highlighted such a disparity and proposed a language hierarchy that comprises of the languages of world classified into six classes based on their resourceavailability. in this hierarchy, class 5 represents the most resource-rich languages for whom benefits of lt are readily available; and class 0 denotes the most under-resourced languages. in this paper, we ask the following two questions: (1) today, in which languages can we build practically useful lt systems, especially chatbots, that could serve as beneficial assistants during the pandemic? (2) how should we prioritize research and resource building investments so that lt is globally ready before the next pandemic? in order to answer these questions, we review the existing literature and resources on covid-19 chatbots, and classify them based on the languages they support and the solutions they provide. quite unsurprisingly, the survey reveals a strong disparity in lt solutions between resource-rich and resource-poor languages. in order to quantify this gap and measure the pandemic-readiness of various languages today, we select 15 asian and african languages (except english) with various degrees of resource-availability, and attempt to build covid-19 faq bots for them. since building an end-to-end chatbot is a substantial engineering effort, we scope the problem down to building an intent classifier for these languages, which forms the core of the natural language understanding (nlu) unit. we also experiment with entity recognition for a subset of these languages. our code and datasets have been made publicly available to foster future research. our study shows that despite using the best available commercial multilingual chatbot frameworks (e.g., google dialogflow foot_1 , microsoft bot framework (ms bot) foot_2 ), advanced machine translation (mt) systems foot_3 , and state-of-the-art massively multilingual language models (mbert @xcite and xlm-r @xcite ), there is a 20-30% drop in performance for class 0 -2 languages as compared to english. the drop is large for all the african languages (e.g., hausa and somali) and some of the asian languages (e.g., marathi and sinhala). note that our experiments were limited to languages which are supported by at least one of the chatbot frameworks, mt systems or mmlms. there are thousands of other languages which are supported by none. we have experimented with one such language (kikuyu) and observed near random performance. therefore, we hypothesize that such languages are not at all ready for the next pandemic. we extrapolate our findings at global scale and construct a global lt readiness map for pandemicresponse and healthcare. based on this map as well as error analysis of the chatbot experiments, we identify a set of research problems as well as resource-prioritization strategies which we believe are key to ensure global lt readiness before the next pandemic. more specifically, the purpose of our work is about comparing the systematic inequalities that exist across different languages while deploying chatbots for emergency situations, as well as showing that certain geographical regions are in a disadvantaged position because even the languages that are spoken by a large portion of their population are ill-supported by current lt. finally, we provide recommendations on how this gap can be bridged by suggesting investment strategies for building lt systems which is otherwise a tough ethical question. the rest of the paper is organized as follows: sec 2 presents the literature survey on lt response to covid-19, specifically focusing on chatbots built for the pandemic. sec 3 describes the chatbot building experiments, where in 3.1 we motivate the choice of languages for the experiment, in 3.2 and 3.5 we discuss the intent and entity detection experiments respectively. in sec 4 we present the global lt readiness map and in sec 5, we conclude with our recommendations. ## literature survey in the recent years, nlp for healthcare has witnessed a major uptake and an impressive volume of work has significantly pushed the research forward by developing sophisticated domain-specific language models @xcite @xcite . these models have been adopted to serve different axes of healthcare such as patient provider communication @xcite , information dissemination @xcite , and self-care management and therapy @xcite @xcite . the role of healthcare chatbots becomes crucial along all these axes because of the recent adoption of telehealth technology services @xcite . chatbots have received a considerable interest during the recent covid-19 pandemic. due to the worldwide spread and severity of the virus and subsequent global response, we believe that the study of covid-19 chatbots can provide us an accurate picture of the global-readiness of lt. we surveyed covid-19 chatbots that are mentioned in the literature and/or deployed in the real-world 5 . ## rapid chatbot building exercise how quickly can one build a pandemic response chatbot in a language based on the best publicly available systems? in order to answer this question we have to understand the pandemic-readiness of various languages. to do this, we made an attempt to build chatbots for answering frequently asked questions about covid-19 using google dialogflow, microsoft bot framework (ms bot), as well as two of the most popular massively multilingual language models (mmlm) -mbert and xlm-r. since building an end-to-end chatbot is complex, we chose to conduct rapid prototyping experiments for intent recognition in 16 languages, and entity recognition in 3 languages. ## measuring global readiness although our current work focuses on analyzing pandemic-preparedness of only 16 languages, here we try to generalize our findings to other languages by introducing a readiness score for every language which empirically measures the pre-paredness of current lt to serve its speakers in a pandemic-like emergency situation. the definition of readiness is based on the assumption that one has access to the best available lt by considering the highest intent detection accuracy of a * l for a language across different frameworks and training setups. we then define the readiness of a language l as its relative accuracy with respect to english as r @xmath0, where a * en denotes the best case accuracy on english, and a random is the accuracy of a random classifier: a @xmath1. observations: from figure 3 , one can observe that south and east african countries are extremely ill-prepared, due to the high dominance of low-resource languages. for instance, people in zambia's speak bemba, chewa and luzi, all of which are severely under-resourced. as pointed out in @xcite , these regions might also be worse-hit in a pandemic situation, and therefore, require immediate attention. for ill-prepared regions such as bolivia in south america, and guatemala in latin america, r l values are slightly better due to the abundance of spanish speakers, however there is a sizeable population speaking under-served languages such as q'eqchi and guarani. countries that fall within fully to moderately prepared categories typically have large native speaker population of one or more of the class 5 languages (english, french, chinese, arabic) and/or well-supported languages (e.g., korean, bengali, malay). it is important to note that while approximating readiness of a language, we assumed same value for all its diverse linguistic variants and dialects, which in certain cases results in overestimation of r c . high r c for north and central african countries (e.g., libya, egypt and sudan) might be due to sizeable population of a resource-rich language arabic. however, arabic has several dialects, which vary from the modern standard arabic at various linguistic levels, and consequently the performance of lt systems for such dialects also vary considerably (zbib et al., 11 infoplease languages spoken in each country of the world: https://bit.ly/3hoas9k 2012; @xcite . it holds true for spanish and portuguese spoken in latin america @xcite and french dialects of western africa. ## conclusion and recommendation from our chatbot development experiences, we uncover a set of interesting insights to arrive at the following recommendations which can improve the state of preparedness of languages to develop useful technologies during the next pandemic. our study confirms that except english, only a few european and asian languages push forward the state-of-the-art research in lt for healthcare. our preliminary investigation suggests that instead of demographic demand, it is the economic prowess of the users of a language that drives the investment towards developing sophisticated lt solutions for a given language. for instance, swahili, even though considered as the lingua franca of africa, is still under-served by commercial chatbot frameworks. similar trends were observed for hausa which has a considerably large speaker base compared to dutch (resource-rich) 12 . we believe that these findings will play a crucial role in making the community aware of the disparity that needs to be addressed before the next pandemic hits.