A Speech-enabled Fixed-phrase Translator for Healthcare Accessibility (2021)

Abstract: In this overview article we describe an application designed to enable communication between health practitioners and patients who do not share a common language, in situations where professional interpreters are not available. Built on the principle of a fixed phrase translator, the application implements different natural language processing (NLP) technologies, such as speech recognition, neural machine translation and text-to-speech to improve usability. Its design allows easy portability to new domains and integration of different types of output for multiple target audiences. Even though BabelDr is far from solving the problem of miscommunication between patients and doctors, it is a clear example of NLP in a real world application designed to help minority groups to communicate in a medical context. It also gives some insights into the relevant criteria for the development of such an application.

## motivation access to healthcare is an important component of quality of life, but it is often compromised by the language barrier which prevents effective communication. in hospitals, medical staff are increasingly confronted with patients with whom they do not share a common language. lack of clear communication can lead to increased risk for patients @xcite but also discourages vulnerable groups from seeking medical assistance. when professional interpreters are not easily available, for example in emergency situations, there is a crucial need for tools to overcome the language barrier in order to provide medical care. while many generic translation solutions are available on the web, they present numerous disadvantages, including the unreliability of machine translation @xcite , the insufficient data confidentiality of cloud services or the absence of resources for minority languages. to overcome these issues, specifically designed tools based on a limited set of pre-translated sentences have been developed. these phraselators @xcite have the advantage of portability, accuracy and reliability. although these tools have limited coverage, and do not solve all communication issues, recent studies have shown that they are generally preferred to machine translation as they are perceived as more reliable and trustworthy in these safety critical contexts @xcite . this paper aims to provide an overview of the nlp components included in the speech-enabled phraselator called babeldr. in section 2 we will give an overview of babeldr usage. we then explain the artificial training data derived from the grammar to specialise the different components in section 3. in sections 4, 5, 6, 7 and 8 we explain babeldr's components in detail, as well as the possible outputs available to users. we then present several usage studies with target groups in section 9.1, report on the performance of the whole system in section 9.2 and conclude in section 10. ## babeldr babeldr foot_0 is a joint project between the faculty of translation and interpreting of the university of geneva and geneva university hospitals (hug). @xcite . the aim of the project is to develop a speech to speech translation system for emergency settings which meets three criteria: reliability, data security and portability to low-resource target languages relevant for hug. it is designed to allow french-speaking medical practitioners to carry out triage and diagnostic interviews with patients speaking albanian, arabic, dari, farsi, spanish, swiss french sign language and tigrinya. babeldr is a web application designed to function on desktops and mobiles. built on the principle of a phraselator, it relies on a limited set of pre-translated sentences, hereafter called coresentences, collected with doctors. for improved usability and more natural interaction with the patient, it includes a speech recognition component: instead of searching for utterances in menus, medical staff can speak freely and the system will map the spoken utterances to the closest pre-translated core-sentence. this sentence is then presented for validation, in a backtranslation step, ensuring that the doctor knows exactly what is being translated for the patient. the patient can then respond by means of a pictogram-based interface. all components can be deployed on a local server with no dependency on cloud services, thus ensuring the data confidentiality that is essential for medical applications. figure 1 illustrates the usage of babeldr. ## training data and grammars due to confidentiality issues, training data for spoken french medical dialogues is scarce. for this reason, the first version of the system was built around a manually defined synchronous context free grammar (scfg, @xcite , used for grammar-based speech recognition and parsing @xcite . this grammar is now leveraged to generate artificial data used both for backtranslation (section 5) and for specialising speech recognition (section 4). the grammar maps source variation patterns, described in a formalism similar to regular expressions, to core-sentences. due to the repetitive nature of the content, the grammars make use of compositional sentences to make resources more compact. these sentences contain one or more variables, which are replaced by different values at system compile time. figure 2 gives an example of a compositional utterance rule. the current version of the grammar includes 2629 utterance rules, organised by medical domain, which expand to 10'991 core-sentences once variables are replaced by values. these core-sentences are mapped to hundreds of millions of surface sentences. figure 3 shows an example of the aligned core-sentence -variation corpus that can be generated from the grammar. ## speech-to-text to ensure both accuracy and usability, the system uses a hybrid approach for speech recognition, combining two recognisers. the first is a grammar based speech recogniser using grxmls generated from the original scfg (see section 3). while this is fast and accurate, since it directly yields a core-sentence, it is unable to handle utterances that are out of grammar coverage. it is therefore complemented by a large-vocabulary recogniser specialised with the monolingual artificial corpus described in section 3. the results of the two approaches are combined based on the confidence score provided by the grammar based recogniser: if the score is over a pre-defined threshold, this result is kept, else the system falls back on the large-vocabulary result. their performance is evaluated in terms of wer, which is 38.9% for the grxml grammar and 14.4% (see for the grxml recogniser we use the nuance asr v10 and the nuance transcription engine 4 for the large-vocabulary one. both can be accessed over the network through our custom api using http post requests. the recognition is filebased and it proves to work well for any real-time interaction. the distributed nature of our back-end platform permits easy scaling and load balancing so that multiple users can interact simultaneously with the recognisers. especially, for the grxml case, we can load and compile grammars on the fly or change the parameters of the recogniser dynamically. we can also parse any text against a specific grammar using an http request. ## backtranslation the backtranslation (introduced in section 2) is an essential step in babeldr since it maps the speech recognition result to a core-sentence that is presented to the doctor for validation. for the grxml recogniser, backtranslation is performed directly by the grammar. for the large vocabulary recogniser, as the set of core-sentences is limited (see section 3), the backtranslation task can be seen as a sentence classification task where the coresentences are the categories, or as translation task into a controlled language. as a resource, we use the bilingual corpus generated from the gram-mar as training data. @xcite introduced an approach based on tf-idf indexing and dynamic programming (dp) achieving 91.8% on accuracy (assuming perfect speech recognition and 1-best). @xcite then applied different approaches using deep learning methods, neural machine translation (nmt) and sentence classification achieving 93.2% (see ## elliptical sentences in dialogues, elliptical utterances are very common, since they ensure the principle of economy and usually avoid duplication @xcite . in babeldr, they allow doctors to question patients in a more efficient way @xcite . however, literal translation of these utterances could affect communication as illustrated in to avoid a wrong backtranslation in elliptical sentences, a context-level information (the previous accepted utterance) is added to the model. therefore, when an utterance is identified as an ellipsis, utterance translation do you have pain in your stomach? good translation: ¿le duele la cabeza? on an artificial ellipsis data set, the model achieves 98% accuracy on detecting elliptical sentences and 88% on backtranslating them to a core-sentence (see more, @xcite . ## output after validation of the backtranslation, babeldr presents the target language output to the patient in written and spoken form, which are both based on the same human translations of the core-sentences. in the following sections we first outline the translation approach and then describe how the translations are rendered for the patient, in audio (for spoken languages) or video format (for sign language). ## patient response interface the original babeldr system was limited to yes-no questions or questions where the patient could respond non-verbally, for example by pointing at a body part. this restrictive approach was problematic both for doctors, who are used to asking open questions, and for patients who had little means to actively contribute to the direction of the dialogue. to build a bidirectional version that would allow more complex responses from the patient, we considered different options. building a system that would allow patients to respond with speech presents numerous difficulties. no speech recognisers exist for many of the minority languages targeted by our system, and few or no resources such as speech corpora are available to build such systems. a text interface, as found in traditional phraselators, while easier to implement, would not be accessible to patients with low literacy. additionally, in the context of a fixed phrase translator, some user training is necessary to familiarise with system coverage, which is not possible for patients who arrive at an emergency service. for these reasons, we chose to add a simple pictograph based response interface, shown in figure 4 . each core-sentence is linked to a set of corresponding response pictographs among which the patient can select their response. evaluation of these pictographs in terms of understandability and acceptability by patients of different educational and cultural backgrounds is ongoing @xcite . a taskbased evaluation showed that all patients prefered the bidirectional version since they could explain their symptoms more efficiently. ## evaluation the system is currently in use at the hug outpatient emergency unit and a user satisfaction study is ongoing to collect patients' and doctors' feedback on system usage in real emergency settings by means of questionnaires @xcite . the study includes only patients with no understanding of french and no common language with the doctor. overall, 90% of the 30 patients included so far reported a positive level of satisfaction. the doctors reported 87%. ## conclusion healthcare translation is required to facilitate the engagement with people with diverse language, cultural, and literacy backgrounds. the development of culturally effective and patient-oriented translation tools has become increasingly urgent. although babeldr is far from solving the problem of miscommunication, it is an example of a concrete application of natural language processing to help minority groups communicate in a medical context. the developed tool, resources and evaluations are a first step toward accessible healthcare apps. this research is essential to define criteria which can be used in the development and evaluation of new medical interpreting technologies with a view to enhancing the usability among patients from refugee, migrant, or other socioeconomically disadvantaged populations.