Biomedical Data-to-Text Generation via Fine-Tuning Transformers (2021)

Abstract: Data-to-text (D2T) generation in the biomedical domain is a promising - yet mostly unexplored - field of research. Here, we apply neural models for D2T generation to a real-world dataset consisting of package leaflets of European medicines. We show that fine-tuned transformers are able to generate realistic, multi-sentence text from data in the biomedical domain, yet have important limitations. We also release a new dataset (BioLeaflets) for benchmarking D2T generation models in the biomedical domain.

## introduction data-to-text (d2t) systems are attracting considerable interest due to their ability to automate the time-consuming writing of data-driven reports. there is a hitherto largely untapped potential for text generation in the biomedical domain. potential applications of natural language generation of patient-friendly biomedical text include preparation of the first draft of package leaflets, patient education materials, or direct-to-consumer promotional materials in countries where this is permitted. here we focus on a d2t task aiming to generate fluent and fact-based descriptions from biomedical data. ## related work recently, neural d2t models have significantly improved the quality of short text generation (usually one sentence long) from input data compared to multi-stage pipelined or template-based approaches. examples include biographies from wikipedia fact tables @xcite , restaurant descriptions from meaning representations @xcite , and basketball game summaries from statistical tables @xcite . still, neural d2t approaches have major challenges, as outlined by @xcite and @xcite which hinder their application to many real-world applications. these include hallucination effects (generated phrases not supported or contradictory to the source data), missing facts (generated text does not include input information), intersentence incoherence, and repetitiveness in the generated text. following the success of leveraging pre-trained large-scale language models for a large variety of tasks, @xcite fine-tuned t5 models @xcite for d2t generation. this strategy achieved state-of-the-art performance on task-oriented dialogue (multiwoz) @xcite , tables-to-text (totto) @xcite and graph-to-text (webnlg) @xcite . to the best of our knowledge, recent neural approaches and transfer learning strategies have not been applied to multi-sentence generation from input data, nor have they been applied in the biomedical domain. our contribution is two-fold: we introduce a real-world biomedical dataset bioleaflets, and demonstrate that transformers can generate high-quality multi-sentence text from data in the biomedical domain. the bioleaflets dataset, fine-tuned models, code, and generated samples are available at https://github. com/bayer-science-for-a-better-life/ data2text-bioleaflets . ## the bioleaflets dataset we introduce a new biomedical dataset for d2t generation -bioleaflets, a corpus of 1336 package leaflets of medicines authorised in europe, which we obtain by scraping the european medicines agency (ema) website. this dataset comprises the large majority (∼ 90%) of medicinal products authorised through the centralised procedure in europe as of january 2021. package leaflets are published for medicinal products approved in the european union (eu). they are included in the packaging of medicinal (a) original section content novonorm is an oral antidiabetic medicine containing repaglinide which helps your pancreas produce more insulin and thereby lower your blood sugar (glucose). type 2 diabetes is a disease in which your pancreas does not make enough insulin to control the sugar in your blood or where your body does not respond normally to the insulin it produces. novonorm is used to control type 2 diabetes in adults as an add-on to diet and exercise: treatment is usually started if diet, exercise and weight reduction alone have not been able to control (or lower) your blood sugar. novonorm can also be given with metformin, another medicine for diabetes. novonorm has been shown to lower the blood sugar, which helps to prevent complications from your diabetes. ## experiments following kale and rastogi (2020), we represent the structured data (i.e., detected entities) as a flat string (linearization). the entities are kept in their order of appearance (table1b). the models are then trained to predict -starting from these entities -the corresponding published leaflet text. we present baseline results on bioleaflets dataset by employing the following state-of-theart approaches: bart and t5 fine-tuning are performed via hug-gingface @xcite . to confirm these findings, human evaluation is performed for section 1 of the test set by two annotators. results are shown in interestingly, specifying the section type in the input records (i.e., explicit conditioning) did not improve model performances ( ## error analysis and limitations after thorough qualitative evaluation of numerous generated samples, the following general issues appear: in the bioleaflets dataset, models perform well in generating section 1, which is 962 characters long on average. however, the quality of section 4 "possible side effects" (3 453 characters long on average) generation is poor. possible improvements to our work are: analysis of the impact of shuffling of entities for the input data generation, introduction of loss functions that explicitly favor factual correctness, usage of specialized biomedical embeddings, inclusion of more source input data (e.g. part-of-speech, dependency tag), generation of longer text (beyond the 512 tokens generated here). ## conclusion in this study, we introduce a new biomedical dataset (bioleaflets), which could serve as a benchmark for biomedical text generation models. we demonstrate the feasibility of generating coherent multi-sentence biomedical text using patientfriendly language, based on input consisting of biomedical entities. these results show the potential of text generation for real-world biomedical applications. nevertheless, human evaluation is still a required step to validate the generated samples. application of the methodology and models used here to different sets of biomedical text (e.g., generation of selected sections of clinical study reports) could be an area for further research.