A uto M e TS : The Autocomplete for Medical Text Simplification (2020)

Abstract: The goal of text simplification (TS) is to transform difficult text into a version that is easier to understand and more broadly accessible to a wide variety of readers. In some domains, such as healthcare, fully automated approaches cannot be used since information must be accurately preserved. Instead, semi-automated approaches can be used that assist a human writer in simplifying text faster and at a higher quality. In this paper, we examine the application of autocomplete to text simplification in the medical domain. We introduce a new parallel medical data set consisting of aligned English Wikipedia with Simple English Wikipedia sentences and examine the application of pretrained neural language models (PNLMs) on this dataset. We compare four PNLMs (BERT, RoBERTa, XLNet, and GPT-2), and show how the additional context of the sentence to be simplified can be incorporated to achieve better results (6.17% absolute improvement over the best individual model). We also introduce an ensemble model that combines the four PNLMs and outperforms the best individual model by 2.1%, resulting in an overall word prediction accuracy of 64.52%.

## introduction the goal of text simplification is to transform text into a variant that is more broadly accessible to a wide variety of readers while preserving the content. while this has been accomplished using a range of approaches @xcite , most text simplification research has focused on fully-automated approaches @xcite @xcite . however, in some domains, such as healthcare, using fully-automated text simplification is not appropriate since it is critical that the important information is preserved fully during the simplification process. for example, @xcite found that fully automated approaches omitted 30% of critical information when used to simplify clinical texts. for these types of domains, instead of fully-automated approaches, interactive text simplification tools are better suited to generate more efficient and higher quality simplifications @xcite . autocomplete tools suggest one or more words during text composition that could follow what has been typed so far. they have been used in a range of applications including web queries @xcite , database queries @xcite , texting @xcite , e-mail composition @xcite , and interactive machine translation, where a user translating a foreign sentence is given guidance as they type @xcite . our work is most similar to interactive machine translation. autocomplete tools can speed up the text simplification process and give full control over information preservation to users, which is required in some domains, such as health and medical. in this paper, we explore the application of pretrained neural language models (pnlms) to the autocomplete process for sentence-level medical text simplification. specifically, given (a) a difficult sentence a user is trying to simplify and (b) the simplification typed so far, the goal is to correctly suggest the next simple word to follow what has been typed. ## medical parallel english wikipedia corpus creation we automatically extracted medical sentence pairs from the sentence-aligned english wikipedia corpus from @xcite . ## approach we had three goals for our analysis: understand the effect of incorporating the additional context of the difficult sentence into autocomplete text simplification models, explore a new application for pnlms, and evaluate our new ensemble approach to the autocomplete text simplification. ## experimental setup we compare the performance of the models on the medical autocomplete text simplification task. we used our medical parallel english wikipedia corpus with 70% of the sentence pairs for training, 15% for development, and 15% for testing. we fine-tuned individual pnlms using huggingface foot_6 with a batch-size of 8, 8 epochs, and a learning rate of 5e -5 . early stopping was used based on the second time a decrease in the accuracy was seen. prediction task sequence of labels (difficult sentence). this (mask) 1 0 1 1 (difficult sentence). this insulin (mask) 0 1 0 0 (difficult sentence). this insulin tells (mask) 1 1 1 1 figure 1 : accuracy for the context-aware models based on the length of the difficult sentences: very short (≤ 5 tokens), short (6 -15), medium (16 -19), and long (≥ 20). we used two metrics to evaluate the quality of the approaches. first, we used standard accuracy, where a prediction is counted correct if it matches the test prediction word. accuracy is pessimistic in that the predicted word must match exactly the word seen in the simple sentence, and as such it does not account for other possible words, such as synonyms, that could be correctly used in the context. since the parallel english wikipedia corpus does not offer multiple simplified versions for a given difficult sentence, accuracy is the best metric that considers automated scoring, simplification quality, and information preservation. accuracy-based metrics can help offset an expensive manual evaluation while providing the best approximation of how the autocomplete systems work. we do not use bleu @xcite and sari @xcite scores, which are widely used in text simplification domain, because the two metrics are specifically designed for fully-automated models that predict an entire sentence at a time. for autocomplete, the models only predict a single word at a time and then, regardless of whether the answer is correct or not, use the additional context of the word that the user typed next to make the next prediction. autocomplete models can suggest just the next word, or they can be used to suggest a list of alternative words that the user could select from (since the models are probabilistic they can return a ranked list of suggestions). to evaluate this use case, and to better understand what words the models are predicting, we also evaluated the models using accuracy@n. accuracy@n counts a model as correct for an example as long as it suggests the correct word within the first n suggestions. ## results and discussion we first analyze the results of the individual pnlms on the medical text simplification autocompletion task and then explore the ensemble approaches. we also include a number of post-hoc analyses to better understand what the different models are doing and limitations of the models. ## conclusions in this paper, we introduced a new medical parallel english wikipedia corpus for text simplification, which contains 3.3k medical sentence pairs. further, we proposed a new autocomplete application for pnlms for medical text simplification. such autocomplete models can assist users in simplifying text with improved efficiency and higher quality results in domains where information preservation is especially critical, such as healthcare and medicine, and where fully-automated approaches are not appropriate. we examined four recent pnlms: bert, roberta, xlnet, and gpt-2, and showed how the additional context of the sentences being simplified could be incorporated into the autocomplete simplification process. further, we introduced automets, an ensemble method that combines the advantages of each of the different pnlms. the automets model outperforms the best individual model, roberta, by 2.1%. longer term, we envision that this new application could lead to other interesting model adaptations and advance text simplification in medical domains.