Language rehabilitation of people with BROCA aphasia using deep neural machine translation (2022)

Abstract: More than 13 million people suffer a stroke each year. Aphasia is known as a language disorder usually caused by a stroke that damages a specific area of the brain that controls the expression and understanding of language. Aphasia is characterized by a disturbance of the linguistic code affecting encoding and/or decoding of the language. Our project aims to propose a method that helps a person suffering from aphasia to communicate better with those around him. For this, we will propose a machine translation capable of correcting aphasic errors and helping the patient to communicate more easily. To build such a system, we need a parallel corpus; to our knowledge, this corpus does not exist, especially for French. Therefore, the main challenge and the objective of this task is to build a parallel corpus composed of sentences with aphasic errors and their corresponding correction. We will show how we create a pseudo-aphasia corpus from real data, and then we will show the feasibility of our project to translate from aphasia data to natural language. The preliminary results show that the deep learning methods we used achieve correct translations corresponding to a BLEU of 38.6.

## introduction stroke represents the 2 nd cause of death in the population, and the 1 st cause of physical handicap in france. according to world stroke organization, 13.7 million people worldwide will suffer their first stroke on 2022 and 5.5 million will die from it. the incidence of stroke increases significantly with age and, in the west, as people are living longer and longer, stroke is almost becoming a pandemic problem. stroke can affect the mechanisms of speech, movement, sensation, an so on. the physical, cognitive and psychic after-effects of a stroke remain, unfortunately, frequent (30 to 50% of cases). many survivors will experience some form of lifelong disability or impairment that they will attempt to cure. they are particularly confronted with the reduction of their movements and isolation, among other things, due to their inability to communicate normally. with rehabilitation and specialist support, however, most stroke survivors can return to a near-normal life. according to the national aphasia association foot_0 , a third of strokes result in aphasia, a major aftereffect that greatly affects quality of life @xcite . aphasia, a term suggested by armand trousseau in 1863, is characterized by a disturbance of the linguistic code, affecting the expression and/or the comprehension, and which can concern the oral and/or the written language. it is a localized or diffuse brain damage, generally in the frontal, parietal and/or temporal area of the left hemisphere, essentially of vascular, traumatic or of tumor origin @xcite . there are several different types of aphasia, all of them coming with their own unique side-effects @xcite . their classification is not a trivial task, however, there is one thing they all share: making communication a difficult task. findings in current theory @xcite suggest frequent misuses of verbs and nouns, either from a character-mismatch or lexical swap perspective, and heavy syntactic alterations @xcite . the discourse abilities might also be limited @xcite . our ultimate goal is to help people with aphasia (pwa) to find their words easily by offering them a speech-to-speech system that corrects mispronounced sentences. to achieve this, we first need a parallel corpus where the source is composed by the altered spoken sentences and the target by what should have been spoken. in our knowledge, this kind of corpus does not exist. what we propose in this article is to create such dataset by starting with sentences pronounced by pwa in speech therapy sessions and their correction, and then augment the corpus with sentence pairs automatically created based on the features of the initial data. we will also perform some preliminary translation experiments to show the overall feasibility of the approach that will lead to an aphasic speech correction system. our focus is on broca patients. we believe that given the nature of popular rehabilitation methods, such as linguistic specific treatment (lst) @xcite and mapping therapy @xcite , both of which are based on repetition of words, similar structures, or giving clues on remembering certain words or phrases, our instant feedback system based on speech translation would be of great help. ## related works in natural language processing (nlp) and also in humanities, the availability of corpora is essential for understanding behavior phenomena and proposing tools or softwares based on nlp techniques or machine learning methods that facilitate the comprehension of such phenomena. in this particular topic, the aphasia data are rare and those that exist are not available. one of the most attractive project developing aphasia corpora is probably aphasiabank @xcite . the objective of aphasiabank project was to provide researchers and clinicians with a large shared multimedia database of uniform discourse from individuals with and without aphasia. the database includes language samples in english, spanish, german, italian, hungarian, mandarin and chinese. the aphasia section of this database contains approximately 180 videos of people with aphasia. the project release @xcite ) refers to the aphasia dataset of individual patient data for the rehabilitation and recovery of people with aphasia after stroke. this project seems to be used by clinicians with the objective to study the rehabilitation. no information is given about the transcription of their utterances. in the moss aphasia psycholinguistics project (mapp) @xcite , the authors provide a searchable database with data from more than 240 patients. the database is made up of the philadelphia naming test (pnt) results. the pnt is a single-word picture naming test developed to collect a large corpus of naming answers from patients. concerning the works that have addressed the problem of aphasia using automatic language processing approaches, we can cite the research below. since the grammatical deficiencies depend on the primary progressive aphasia (ppa), in @xcite the authors propose to classify ppa variants by using part of speech (pos) production and to identify morphological markers that classify them by using machine learning. ppa is a very unique kind of aphasia. it is a form of dementia, and there are no cures available. eventually, the person with this dementia completely loses their ability to comprehend and produce language due to gradual degradation @xcite . the study @xcite combines natural language processing and machine learning methods to predict the severity of pwa, both by score and severity level. the authors used a dataset of 238 participants extracted from aphasiabank. they took the data from its transcript and composed the dataset by removing stop words and other items not necessary for this task. stop word lists differ greatly, but they usually contain non-thematic words, like function words (determiners), prepositions (on, it, under), and so on. this is a very questionable decision, given the importance of the already few words people with aphasia are uttering. stop words could be important indicators. ## building an aphasic-french parallel corpus in this section, we will describe how to build an aphasic-french corpus (aphafrech) which will be used to show the feasibility of developing a communication rehabilitation support system for an aphasic person. to do this, we started by collecting real aphasia data in french that we transcribed, then we developed methods to build a parallel corpus that can be used to develop a machine translation system. we used several sources to build up a corpus for the analysis of aphasic errors. the first source is made up of videos extracted from the web recorded in therapy sessions between speech therapists and pwa. in each video a speech therapist asked several questions to the pwa such as: what is your name? how do you feel today? describe ## a preliminary experience in machine translation of a pseudo aphasic corpus in this section, we study the opportunity to translate an aphasic corpus to its corrected counterpart. for that we use aphafresh, the parallel corpus we described in section 3. to generate the aphasic sentences, we used only the ngram scoring function since it is the one that achieves the best aphasia errors. pseudo-aphasia correct sentences sentences sois juite sois juste j'ai fait sine j'ai fait signe je duis calme je suis calme je me suis révemillé je me suis réveillé je suis detite je suis petite si c ¸a ne vous dérande si c ¸a ne vous dérange pas, pourrions-nous pas, pourrions-nous inspecter votre valise inspecter votre valise ? the input of the encoder is the aphasia sentence and the output is the hidden state and cell state of the lstm. the decoder has the hidden state and cell state of the encoder as inputs in addition to the input sentence. the results of the decoder lstm is passed through a dense layer to predict decoder outputs as shown in figure 2 . in concerning the optimizer, in our experiments we tested several methods, the one which achieves the better results is the adaptive gradient algorithm @xcite . in fact, adaptive gradient algorithms calculate gradient-based updates using the history of gradients, which has the advantage to reduce the inconvenience of manually setting the step size parameter in the stochastic gradient descent optimizer. in addition, adagrad is known also for its computational efficiency @xcite . from figures 3 and 4 , we can conclude that the accuracy is high and the model reduced the value of the loss, which means that the model makes small errors on few data and the model predicts well. the training and the validation curves start with relatively high loss at the beginning and gradually decrease as training and validation examples are added and gradually flatten, indicating that adding more examples does not improve the performance of the model on both data. this leads us to assume that our neural network does not overfit. we tested this model in a test corpus composed of 2,000 aphasia sentences. the results in terms of cumulative bleu are given in figure 5 illustrates the distribution of the bleu score over the 2,000 sentences of the test corpus. we can observe that more than 31% of the sentences have a bleu higher than 50 which means that we achieve a very high quality and fluent translation. only 5% of the translation have a bleu smaller than 10 which corresponds in general to a useless translation. 19% of the translations have a bleu between 10 and 19, which corresponds to sentences that are difficult to understand. in order to make the reading of figure 5 easy, ## conclusion aphasia is a unique and rather complex phenomena. there is a great amount of work trying to understand and explain the underlying structural changes from different perspectives. since there is no general consensus on what the best approach is to therapy, the field remains open for experimentation. we decided to take up the challenge from a machine learning perspective by implementing a method that will eventually allow us to come up with a speech-to-speech system where the input is aphasic speech and the output is a rehabilitated speech. for that, we created an aphasia-like corpus, aphafrech, with correct-incorrect sentence pairs, using three different resources. this required the study of errors from aphasic sources in order to understand certain types of errors and to reproduce them automatically. with the created dataset we trained a neural network machine translation that yields very high quality translations on aphafrech. the next step will concern the introduction of more complex aphasia errors into aphafrech (such as context-dependant errors and semantic based errors) and the study of the quality of the translation by using a more elaborated dnn machine translation.