Fine-tuning BERT Models for Summarizing G erman Radiology Findings (2022)

Abstract: Writing the conclusion section of radiology reports is essential for communicating the radiology findings and its assessment to physician in a condensed form. In this work, we employ a transformer-based Seq2Seq model for generating the conclusion section of German radiology reports. The model is initialized with the pretrained parameters of a German BERT model and fine-tuned in our downstream task on our domain data. We proposed two strategies to improve the factual correctness of the model. In the first method, next to the abstractive learning objective, we introduce an extraction learning objective to train the decoder in the model to both generate one summary sequence and extract the key findings from the source input. The second approach is to integrate the pointer mechanism into the transformer-based Seq2Seq model. The pointer network helps the Seq2Seq model to choose between generating tokens from the vocabulary or copying parts from the source input during generation. The results of the automatic and human evaluations show that the enhanced Seq2Seq model is capable of generating human-like radiology conclusions and that the improved models effectively reduce the factual errors in the generations despite the small amount of training data.

## introduction for patients with cancer, imaging findings are critical for primary diagnosis and treatment guidance * corresponding authors contributed equally. the main goal of this work is to automatically extract information relevant for treatment planning from standardized, real-life radiology reports. expert validation is on the other hand still essential for this clinical routine application. for this purpose, we build a system that merges the information available in the general information and findings sections of the sor radiology reports into a conclusion, which can be compared to conclusions generated by human experts. our main contributions in this work includes: (i) we tested the effectiveness of applying the generic @xcite . each report has a uniform organization: the general section expresses background information on imaging and clinical data, the next section (findings) describes oncology and nononcology findings, and the conclusion section gives oncological and non-oncological impressions. ## related work existing text summarization models can be broadly classified into three categories: extractive, abstractive and hybrid. early extractive approaches relied on human-designed features extracted from texts to identify key sentences. deep learning methods show good performance in various of nlp tasks. the data-driven approaches are able to learn features representations automatically. extractive models have the advantage of producing semantically and syntactically correct summaries. abstractive models employing an encoder-decoder frame-work with attentive recurrent neural networks, e.g. on news article corpus, became a standard architecture in abstractive summarization, which translates the original source content to a concise expression about the main content of the source input @xcite @xcite @xcite . in order to improve the faithfulness of the generated summarization given the facts in the source input, abstractive models are usually enhanced to replicate facts from the source combining extractive and abstractive approaches. @xcite incorporated a pointer network @xcite that selects a word from a predefined vocabulary to replace an unknown word predicted by a rnnbased encoder-decoder model. our work aims to combine both benefits of extractive and abstractive summarization with a transformer-based model. @xcite used the pointer network nallapati et al.2016b as a soft switch to either produce a word from the vocabulary distribution or to select a word from a copy distribution provided by a target-source attention distribution. chen and bansal; @xcite ; 2018 also applied the copy mechanism to the rnn-based model, but decomposed the decoder into a firststage extraction model and a second-stage generator. in the first stage, the encoders in both works processed sequential document representation and provided sentence-level representations to the extractor for selection. in the second stage, @xcite used the language model to rewrite the selected sentences into the summary. chen and bansal, 2018 trained the decoder from scratch by using rouge @xcite scores as a reward strategy for reinforcement learning to generate summaries based on the selected sentences. in our work, we integrate the pointer network to a transformer-based encoder-decoder model. summarizing radiology findings with neural seq2seq learning of @xcite very closely related to our work. @xcite collected a large set of domain-specific training data to train the rnn-based pointer-generator @xcite . because there are usually two sections in radiology reports: background and findings, to provide relevant information for the summary, @xcite incorporated an extra encoder for encoding the background information and findings separately. in contrast, we feed the combination of sequences of the background and findings section as one input and into one encoder. @xcite improved the radiology summarization model by optimizing the factual correctness of the summaries via policy learning. in order to combine extraction and abstraction in one model, we propose two target sequences paired with an input sequence. one target sequence is the reference summary and the other is a sequence consisting of key sentences extracted from the input. our goal with the dual target sequences is to encourage the encoder-decoder model to retain some of the input while generating new phrases for the summaries. pretrained language models have advanced the state-of-the-art when fine-tuned in various nlp tasks, as well as in automatic text summarization @xcite @xcite . @xcite demonstrated the efficacy of warm-starting the encoder and decoder from checkpoints of publicly available large language models, including bert @xcite and gpt-2 @xcite , for text generation task such as machine translation and text summarization. depending on different initialization combinations, they investigated variants of the seq2seq model, such as bert2random, bert2bert, bert2gpt, etc. warm-starting the seq2seq model leveraging these pretrained language models checkpoints can reduce computational resources and time by orders of magnitude, while improving the sequence generation performance. we adopt the warm-starting idea and initialize both the encoder and decoder with a generic pretrained german bert model (deepset.ai, 2019). we fine-tune the model with our german radiology report data and enhance the model by combining extractive and abstractive objectives. ## models the main task of summarizing radiology findings is to transform the salient and clinically significant findings from a source of words and phrases @xmath0, x 2 , ..., x t }, to a sequence of concise expressions @xmath1, y 2 , ..., y ′ t }. background information in the radiology report conveys important information for short-term or long-term examination of each patient in the clinical routine, which is why abstractive models needs to incorporate background information into the summary generation @xcite . the content of the source sequence x contains the background information and imaging findings. these findings convey the information about the location of the primary tumour, the presence of metastases at different body regions, and other non-oncological findings. y is the conclusion of the radiology report, which on the one hand assesses the patient's condition according to the detailed findings and on the other hand concisely summarizes the significant findings from the source sequence x. we use a collection of aligned x and y pairs to train transformer-based seq2seq models to generate y . baseline model warm-starting the seq2seq model leveraging pretrained checkpoints can reduce computational resources and time by orders of magnitude, while improving the sequence generation performance @xcite . we utilize the bert2bert model defined in @xcite , as our abstractive summarization baseline model. the encoder and decoder of the model are initialized from a public available bert checkpoint (deepset.ai, 2019), except the encoder-decoder attention layers in the decoder. taking advantage of the transformer architecture and pretrained language models, among the 221 millions trainable parameters in the bert2bert model, only 26 millions parameters in the encoder-decoder attention layers are initialized randomly, and 195 millions are loaded from the pretrained bert model. the reduction of randomly initialized, trainable parameters, allows for fewer fine-tuning steps, and the model's ability to perform well on small training data sets. bert2bert + extraction most abstractive systems suffer from the problem of creating spurious facts due to their ability to paraphrase. hybrid systems that combine extraction and abstraction are expected to improve the correctness of the generated facts by using more criteria to extract the original facts from the source @xcite @xcite @xcite . different to previous works, which incorporated separate extraction and abstraction stages @xcite @xcite , we propose a new learning scenario with little modification to the architecture of the bert2bert model by adding an extraction learning objective (bert2bert+ext). therefore, during training, we optimize the following combined loss: figure 2: bert2bert model adding extraction loss. in order to train the decoder to extract the key findings through generation, we supply an additional target sequence ("background + key findings"), which consists of the key findings selected from the source sequence. the setup is illustrated in figure 2 . through the extraction objective, the model is trained to reconstruct the key sentences in the generation. in the original setting of bert2bert, we only train the model using our source and target sequence pairs (x, y ). as showed in figure 2 , x symbolizes the source input and contains "background + findings" and y is the target input "conclusion". during training, the decoder of bert2bert+ext is fed with additional target sequences ("background + key findings") including the general section and key sentences from the findings sections as input. section 4.3 explains how to extract these key findings from the finding section from our training data. extractive loss encourages the model to reconstruct key phrases from the source input. abstractive loss prompts the model to generate new formulations that are not from the source sequence. bert2bert + pointer pointer networks allow the model to copy words from the source sequence through an alignment between the target sequence and the source sequence @xcite . the benefits of incorporating the pointer to the generation procedure are not only to reduce the number of tokens, which are not known to bert, but also to ensure factual correctness while generating new phrases. pointer networks have been used for abstract summaries of seq2seq models based on rnns as a standard architecture. however, to the best of our knowledge, there has been little exploration of incorporating pointer networks into the transformer encoder-decoder model for summarization tasks. figure 3 illustrates the combination of bert2bert and the pointer mechanism (bert2bert+ptr). the pointer network consists of one linear layer followed by a sigmoid function which generates a pseudo-probability p gen in the range of [0, 1]. in the original function of @xcite gen is given by: see et al., 2017 recycled attention scores directly from the encoder-decoder attention layer. however, in the bert2bert model, we not only have multiple encoders and decoders, but also multiple heads of the encoder-decoder attention. we can solve the dimension of multiple heads in the attention distribution using the mean of the multi-head attentions @xcite . these hidden states from the final encoder are used as context vectors passed to each decoding step. each decoder state s t used for predicting the next token is also from the last decoder, as well as the multi-head encoderdecoder attention scores a t . h x t in equation 3 represents the hidden output from the final encoder weighted by the sum of the heads of the encoderdecoder attention layers at each decoder step from the last decoder, analogous to the rnn-based context vector. h x t is given by: ## conclusion in this work, we experiment and demonstrate the efficacy of the bert2bert-based abstractive models on summarizing german radiology findings in structured reports. we propose two strategies to improve the bert2bert model with the aim of optimizing the factual correctness in the conclusions generated by the system, bert2bert+ext and bert2bert+ptr. both bert2bert+ext and bert2bert+ptr models have very few modifications to the baseline model and improve the performance of the model. in bert2bert+ext, we train the model to generate summaries, encouraging the model to reconstruct key sentences based on the source text in the training process. bert2bert+ptr incorporates the pointer mechanism to modify the decoder's prediction by copying the salient segments directly from the source sequence. despite the limitations of the models and the imbalanced training data, the issue of unfaithful facts in the conclusions generated by the baseline model is greatly improved by these hybrid models. one pressing issue in the future work is to investigate the potential advantages of these models on free-text radiology data or data in other domains.