SINAI at R ad S um23: Radiology Report Summarization Based on Domain-Specific Sequence-To-Sequence Transformer Model (2023)

Abstract: This paper covers participation of the SINAI team in the shared task 1B: Radiology Report Summarization at the BioNLP workshop held on ACL 2023. Our proposal follows a sequence-to-sequence approach which leverages pre-trained multilingual general domain and monolingual biomedical domain pre-trained language models. The best performing system based on domain-specific model reached 33.96 F1RadGraph score which is the fourth best result among the challenge participants. This model was made publicly available on HuggingFace. We also describe an attempt of Proximal Policy Optimization Reinforcement Learning that was made in order to improve the factual correctness measured with F1RadGraph but did not lead to satisfactory results.

## introduction radiology explorations constitute an essential pillar in diagnosis and treatment of many diseases nowadays. the radiology report documents communicate crucial findings in a radiology study to the referring physician and/or patient. a standard radiology report ends with an impression section that summarizes the most relevant findings. this section, due to the communicative purpose of the radiology reports, may be considered its most significant part @xcite . despite the fact of communication being crucial in assuring quality of healthcare, studies have shown that relevant findings may often be eluded @xcite . in the recent years, this problem drew the attention of the research community, allowing the clearness and conciseness of the communication to become of greater focus in radiology. the task 1b at the bionlp workshop held on acl 2023 draws the community effort to designing systems capable of automating the generation of radiology impressions. the work carried out in this growing field in the recent years mostly focused on chest x-ray due to availability of large curated datasets of chest radiography exploration reports, namely mimic-cxr @xcite and open-i chest x-ray @xcite . the goal of the task 1b at the bionlp workshop is to foster development of automatic radiology report summarization system and expanding their applicability by incorporating seven different modalities and anatomies in the provided data. this paper covers participation of the sinai team from the university of jaén in the first challenge of the shared task, which required the development of a mono-modal radiology report summarization system that would maximize the factual correctness of its output. we propose to automate the generation of radiology impressions with "sequence-to-sequence" learning that leverages the power of publicly available pre-trained models, both general domain and biomedical domain-specific. more specifically, we opted for fine-tuning flan-t5-base, a general domain text-to-text transfer transformer fine-tuned with instructions @xcite , and scifive-base-pubmed_pmc, a model with the same architecture trained on large-scale biomedical corpora @xcite . we also cover a proximal policy optimization experiment we carried out with the objective of maximizing factual correctness of generated impressions. our best performing system based on domain-specific model reached 33.96 f1radgraph score which is the fourth best result among the challenge participants. the remainder of the paper is organized as follows: section 2 offers a brief description of the data provided for this task. section 3 discusses the metrics used for system evaluation and section 4 describes the systems presented by our team for the official evaluation with the results that are disclosed in section 5. finally we draw the conclusions of our work in section 6. ## data the data provided for the mono-modal challenge contained 73,255 radiology reports divided into three subsets: training (5,9317 reports), development (7,413) and test (6,526 reports) @xcite . the official evaluation was carried out on a hidden test dataset comprised of 6,531 samples. training and hidden test sets included reports describing 2 different types of radiology explorations, namely ct and mri performed on 7 different anatomic regions: head, abdomen, chest, spine, neck, sinus and pelvis. whereas the development and test datasets lacked samples of spine, abdomen, pelvis and neck mri as well as sinus ct reports. ## evaluation metrics the submissions for the mono-modal challenge were evaluated using bleu4, rouge-l, bertscore and f1radgraph. bleu4 is a metric traditionally used for evaluation of machine translation systems that assesses the proximity of translations to their corresponding labels. it does not quantify the clarity or grammatical accuracy of the model's outputs, but instead employs statistical measures to guarantee that all the words in the generated outputs are also present in the targets @xcite . rouge (recall-oriented understudy for gisting evaluation) is one of the most commonly used metrics for automatic text summarization evaluation by comparing generated summaries to other (ideal) summaries created by human. the rouge-l measures the longest matching sequences of words by looking for the longest common substrings in the generated and reference summaries @xcite . bertscore computes a similarity score for each token in the candidate sentence with each token in the reference sentence @xcite . finally, the f1radgraph leverages a system trained on the radgraph dataset that extracts relevant entities and relations between them. the proposed f1-style reward focuses on entities and whether or not there is a connection between them in the reference summary compared to the prediction @xcite . ## methods with the objective of measuring the impact of domain-specific pre-training on the t5 model's performance on radiology report summarization we carried-out fine-tuning of two models of the same architecture for this task. we also attempted to optimize factual correctness of the impressions generated by the general domain model through proximal policy optimization. ## results our team made a total of three submission on the mimic-iii hidden test set that corresponded to the three systems we have described in 4. a finetuned biomedical domain-specific sci-five model resulted to be the best performing one in terms of f1radgraph, which is the reference metric of this challenge reaching 32.48 points. while the performance of all three models is relatively consistent in terms of bertscore, it can be concluded that the rl experiment resulted in a slight decrease of the quality of model's generations. the results also suggests that domainspecific training is beneficial for such complex tasks as radiology report summarization. ## conclusions this article describes the contribution of the sinai team from the university of jaén to the acl2023-bionlp shared task. more specifically, we introduce an impression section generation systems that follow a "sequence-to-sequence" approach to the problem and leverage pre-trained publicly available models. in our study we compare the performance of general-domain t5 model with sci-five, the result of a domain adaptation process though additional training on biomedical datasets. after finetuning on the datasets provided for this challenge, a system based on sci-five resulted the best performing one, achieving 33.96 f1radgraph score. we made is model available for the research community on hugginface foot_2 . we also describe an experiment of subjecting the fine-tuned general-domain model to a rl learning with ppo method that, unfortunately did not resulted in a performance improvement. thus, the future work will focus on improvement of factual correctness of the output though rl techniques in order to overcome the presented problem.