Coherent and Concise Radiology Report Generation via Context Specific Image Representations and Orthogonal Sentence States (2021)

Abstract: Neural models for text generation are often designed in an end-to-end fashion, typically with zero control over intermediate computations, limiting their practical usability in downstream applications. In this work, we incorporate explicit means into neural models to ensure topical continuity, informativeness and content diversity of generated radiology reports. For the purpose we propose a method to compute image representations specific to each sentential context and eliminate redundant content by exploiting diverse sentence states. We conduct experiments to generate radiology reports from medical images of chest x-rays using MIMIC-CXR. Our model outperforms baselines by up to 18% and 29% respective in the evaluation for informativeness and content ordering respectively, relative on objective metrics and 16% on human evaluation.

## introduction presenting information in text format has been critical to the development of human civilizations. thus text generation is an important field in artificial intelligence and natural language processing, where the input to such natural language generation models could take on the form of text, graphs, images or database records @xcite @xcite . recent advancements in natural language generation has been propelled by end-to-end neural models (e.g. @xcite ), which has strong capabilities to learn associations within large-scale datasets. however, since it is challenging to exert control over the neural generation process and the corresponding output, the usability of such models in practical scenarios are limited, as the generated content could be erroneous, incoherent, or even socially inappropriate @xcite . it is therefore ideal to include explicit provisions in neural text generation to better model characteristics such as informativeness and topical continuity. it has also been shown that informativeness and textual cohesion are important properties in clinical texts to make them more easily comprehensible @xcite . image to text generation is a natural language generation task that has been popular in communities beyond nlp (e.g. computer vision, machine learning). a general approach is to construct the representation of the entire input image and decode the output text conditioned on the image representation @xcite . such approaches work well for scenarios where only a short generated sentence is needed in the output (e.g. image captioning), as typically what is needed is to identify individual objects and fill in the most probable words to describe the overall situation. however, such approaches might not generalize to scenarios where complex semantics embodied in the input images need further inferencing or where the generated outputs need to articulate detailed or specific information, logical reasoning, or recommendationsall of these cases typically require at least multiple sentences (to form a report) @xcite . medical reports are a classic example of such a scenario where each sentence in a report describes very precise clinical observations or inferences. we present a neural approach for producing radiology reports from images in a sentence-bysentence order to pinpoint more targeted and precise medical information from the input images and at the same time minimize hallucination from neural text generation. the modeling components ensures the generated report is informative, coherent, and concise via gated mechanisms to model topical continuity, orthogonality criteria in sentence state selection to reduce redundancy, and a neural architecture that is pretrained to predict domain entities during each context of sentence generation in order to encourage induction bias. ## method in this section we delineate the following: (1) the proposed neural architecture and the corresponding network computations; (2) how we pre-train the network to predict the context entities from each sentence representation using a multi-label classifier; (3) how we further train the neural architecture to decode the corresponding sentences from each sentence representation to form the report. ## conclusion we presented a technical approach on radiology report generation which ensures global text properties such as informativeness, topical continuity for coherence while reducing redundant content. both objective metrics and human evaluations showed significant performance over competitive baselines.