Controllable Chest X -Ray Report Generation from Longitudinal Representations (2023)

Abstract: Radiology reports are detailed text descriptions of the content of medical scans. Each report describes the presence/absence and location of relevant clinical findings, commonly including comparison with prior exams of the same patient to describe how they evolved. Radiology reporting is a time-consuming process, and scan results are often subject to delays. One strategy to speed up reporting is to integrate automated reporting systems, however clinical deployment requires high accuracy and interpretability. Previous approaches to automated radiology reporting generally do not provide the prior study as input, precluding comparison which is required for clinical accuracy in some types of scans, and offer only unreliable methods of interpretability. Therefore, leveraging an existing visual input format of anatomical tokens, we introduce two novel aspects: (1) longitudinal representation learning – we input the prior scan as an additional input, proposing a method to align, concatenate and fuse the current and prior visual information into a joint longitudinal representation which can be provided to the multimodal report generation model; (2) sentence-anatomy dropout – a training strategy for controllability in which the report generator model is trained to predict only sentences from the original report which correspond to the subset of anatomical regions given as input. We show through in-depth experiments on the MIMIC-CXR dataset how the proposed approach achieves state-of-the-art results while enabling anatomy-wise controllable report generation.

## introduction a chest x-ray (cxr) is a frequently performed radiology exam (nhs england and nhs improve- report generator current scan prior scan rl rl ll ll there are low lung volumes. the opacities in the left lung are resolved. the right lung remains clear. ## method we first extract anatomical visual feature representations ⃗ @xmath0. to model longitudinal relations, we perform feature extraction for both the scan under consideration and for the most recent prior scan, and then combine region-wise using our proposed longitudinal projection module. we then input the features to the language model (lm) alongside the text indication field, which is trained using our anatomysentence dropout strategy. we show the proposed architecture in figure 2 and describe these steps in more detail below. ## limitations while the proposed method shows state-of-the-art results on cxr automated reporting, end-to-end report generation from cxr images requires further research to reach the clinical accuracy needed to be useful as a diagnostic tool. we note that our eval-uation is itself limited since the chexbert labeller reportedly has an accuracy of only 0.798 f1 @xcite ) and thus we do not expect to measure perfect scores on our clinical efficiency metrics. our method focuses only on cxr and adapting it to other types of medical scans might be challenging. first, due to the 2d nature of cxrs compared to other types of 3d scans (e.g., ct, mri). second, we strongly rely on the chest imagenome dataset and its annotations. these are automatically extracted and similar sentence-anatomy annotations could be extracted for radiology reports from other types of scans. however, as the same authors pointed out, there are some known limitations of their nlp and the region extraction pipelines; for instance, clinical findings may not be properly extracted from a follow-up report which may be as simple as "no change is seen". hence, some refinement of the pipelines with or without additional manual input might be required. ## conclusion this work focussed on two key aspects of cxr automated reporting: controllability and longitudinal cxr representation. we proposed a simple yet effective solution to align, concatenate and fuse the anatomical representations of two subsequent cxr scans into a joint representation used as the visual input to a language model for automated reporting. we then proposed a novel training strategy termed sentence-anatomy dropout, to supervise the model to link each anatomical region to the corresponding output sentences. this gives the user more control and easier interpretability of the model predictions.