G erman- A rabic Speech-to-Speech Translation for Psychiatric Diagnosis (2020)

Abstract: In this paper we present the natural language processing components of our German-Arabic speech-to-speech translation system which is being deployed in the context of interpretation during psychiatric, diagnostic interviews. For this purpose we have built a pipe-lined speech-to-speech translation system consisting of automatic speech recognition, text post-processing/segmentation, machine translation and speech synthesis systems. We have implemented two pipe-lines, from German to Arabic and Arabic to German, in order to be able to conduct interpreted two-way dialogues between psychiatrists and potential patients. All systems in our pipeline have been realized as all-neural end-to-end systems, using different architectures suitable for the different components. The speech recognition systems use an encoder/decoder + attention architecture, the text segmentation component and the machine translation system are based on the Transformer architecture, and for the speech synthesis systems we use Tacotron 2 for generating spectrograms and WaveGlow as vocoder. The speech translation is deployed in a server-based speech translation application that implements a turn based translation between a German speaking psychiatrist administrating the Mini-International Neuropsychiatric Interview (M.I.N.I.) and an Arabic speaking person answering the interview. As this is a very specific domain, in addition to the linguistic challenges posed by translating between Arabic and German, we also focus in this paper on the methods we implemented for adapting our speech translation system to the domain of this psychiatric interview.

## introduction in psychiatry the mini-international neuropsychiatric interview (m.i.n.i.) is a short structured diagnostic interview for psychiatric disorders @xcite . in germany it is, among others, used for diagnosing arabic speaking refugees. here, the language barrier is an obvious one, that is normally overcome with the help of human interpreters. however, human interpreters are scarce, expensive and very often not readily available when an urgent diagnosis is needed. in the project removing language barriers in treating refugees-relater we are therefore building a speech-to-speech translation (s2st) system for interpreting between a german speaking psychiatrist and an arabic speaker taking the m.i.n.i. interview. the natural language processing (nlp) technology for this scenario faces two challenges: a) the general linguistic challenges when translating between german and arabic and b) the specific domain of the interview for which only very little adaptation data is available. in this paper, we describe the arabic related nlp components with which we implemented the speech to speech translation between german and arabic for the interview. these components are part of a server-based application, where the client application is capable of running on mobile platforms; whereas the components themselves run on remote powerful computation servers. the application realizes a turnbased interpretation system between the psychiatrist and the potential patient. while end-to-end s2st translation systems are the latest trend in research, for our system we opted for pipe-lined systems, because a) pipe-lined systems still outperform end-to-end systems @xcite , and b) to the best of our knowledge no suitable corpus for training german-arabic end-to-end s2st systems exists, for any domain. all systems in our pipeline have been realized as all-neural end-to-end systems, using different architectures suitable for the different components. the speech recognition system uses an encoder/decoder + attention architecture @xcite , the text segmentation component, and the machine translation are based on the transformer architecture @xcite , and for the speech synthesis we use tacotron 2 foot_0 @xcite for generating spectrograms and waveglow as vocoder @xcite . the rest of the paper is structured as follows: section 2 describes the arabic speech recognition system in the pipeline, while section 3 describes the machine translation component and section 4 the arabic speech synthesis system. as the the topic of diacritization is very prominent for arabic nlp, we discuss the specific issues we faced in our s2st system in section 5. in section 6, we study the real-time aspect of our pipeline system which is essential for the deployment. ## automatic speech recognition for automatic speech recognition (asr), we employ an lstm-based sequence to sequence (s2s) model with encoder/decoder + attention architecture. this model yields better performance compared to the self-attention s2s model with a comparable parameter size. before the lstm layers in the encoder, we place a two-layer convolutional neural network (cnn) with 32 channels and a time stride of two to down-sample the input spectrogram by a factor of four. in the decoder, we adopt two layers of unidirectional lstms and the approach of scaled dot-product (sdp) attention to generate context vectors from the hidden states. more details about the models can be found in @xcite . in section 2.1, we list the data used for the training, testing, and domain adaptation. for the latter, we use primary methods described in section 2.2. we report some implementation details and experimental results in section 2.3. ## machine translation it is a known fact that translating between structurally or morphologically different languages is a very difficult task. two well-known examples of these hard language pairs are english-german and english-arabic. in this work, we are associating the worst of these two worlds: arabic and german. an example of the unmistakable differences between these two languages is that both of them are morphologically rich: arabic is highly inflectional ( @xcite ) and german possesses word compounding. at the syntactic level, word order is a substantial difference between the two languages. arabic is much more flexible in this respect. in this work, we face two major challenges: data scarcity caused by the understudied language pair and the specificity of the domain of application. indeed, the language pair under consideration here has been out of the focus of the international machine translation campaigns. such campaigns (such as wmt foot_2 and iwslt foot_3 ) are organized on a yearly basis, and have boosted considerably both the performance and the available resources for the language pairs they consider. the data scarcity problem becomes even more severe when we know that the resulting system is to be involved in the communication between psychiatrists and their patients. the genre of the language used therein is quite different from that used in the news. this latter makes most of the available training data. in the following, we explain the steps we followed to overcome these limitations and to produce a reasonable translation system. then, we show the developed systems in action through empirical evaluations. ## speech synthesis for speech synthesis or text-to-speech (tts), we use the state-of-the-art model, namely tacotron 2 foot_11 @xcite which is a recurrent s2s network with attention that predicts a sequence of mel spectrogram frames for a given text sequence. for generating time-domain waveform samples conditioned on the predicted mel spectrogram frames, we chose waveglow @xcite where the authors claim that it has a faster inference than wavenet @xcite used with tacotron 2 in @xcite . ## diacritization as mentioned earlier (section 4), the dicritization was introduced to make the tts understandable. we were, indeed, unable to synthesize good arabic speech without these short vowels. the synthesis, then, consists of incomprehensible mumbles as the system tends to insert the vowels arbitrarily. diactritics have a crucial role in giving the arabic text a phonetics @xcite . moreover, they allow for better comprehension by reducing the level of ambiguity inherent in the arabic transcription system. having that said, most modern written arabic resources omit these diacritics and rely on the ability of the native speakers to guess them from the context. in particular, all the parallel data used to train our translation systems has no diacritics. we employ a sequence-tagger trained using the flair-framework @xcite , which is a bilstm-crf proposed by @xcite . besides @xcite introduces contextual string embeddings. thereby, sentences are passed as sequences of characters into a character-level language model (lm) to form word-level embeddings. this is done by concatenating the output hidden state after the last character in the word from the forward lm and the output hidden state before the word's first character from the backward lm. these two language models result from utilizing the hidden states of the forward-backward recurrent neural network (see @xcite for more details about the approach). ## real-time aspects all components of the pipeline reside on only one gpu of type titan rtx with 24 gb memory. the number of components is 7 where there are 3 for each s2st direction plus the diacritization component for arabic side. we measured the latency of the system components by using 5 arabic sentences from the test set alj.msa.2h consisting of 16, 9, 8, 12 and 9 words, hence 54 words in total. for asr, we use the synthesis of this sentences which has the total duration of 47.85 seconds. the ratio of the total duration of processing the 5 sentences by the whole pipeline system to the total input speech duration is 7.95 47.85 = 0.17. this means that the processing time is about 6 times faster than the input speech duration. hence, the system is real-time capable and long sentences can be chunked into smaller parts in a stream-like output (see @xcite as an example of using the lstm-based model for asr to process a stream). ## conclusion in this work, we presented the different components of our german-arabic speech-to-speech translation system which is being deployed in the context of interpretation during psychiatric, diagnostic interviews. for this purpose, we have built a pipe-lined speech-to-speech translation system consisting of automatic speech recognition, machine translation, text post-processing, and speech synthesis systems. we also described the problems we faced while building these components and how we proceeded to overcome them. the speech recognition system uses an lstm-based encoder/decoder + attention architecture, the machine translation component is based on the transformer architecture, the post-processing for arabic employs a sequence-tagger for diacritization, and for the speech synthesis systems we use tacotron 2 for generating spectrograms and waveglow as a vocoder. for domain adaptation, we used fine-tuning and mixed-fine-tuning on hand-created in-domain data. we achieved thereby considerable improvements despite the scarce collected domain data. this result confirms the already well established fact about the extreme importance of in-domain data. therefore, collecting more of this domain-related data is on the top of our priorities in the near future. certainly, this data collection process will not prevent us from trying more ways to exploit the data we already have. for instance, more efforts will be put into exploring other adaptation techniques. additionally, we are investigating the exploitation of the large quantities of monolingual data in the translation components.