E cho G en: Generating Conclusions from Echocardiogram Notes (2022)

Abstract: Generating a summary from findings has been recently explored (Zhang et al., 2018, 2020) in note types such as radiology reports that typically have short length. In this work, we focus on echocardiogram notes that is longer and more complex compared to previous note types. We formally define the task of echocardiography conclusion generation (EchoGen) as generating a conclusion given the findings section, with emphasis on key cardiac findings. To promote the development of EchoGen methods, we present a new benchmark, which consists of two datasets collected from two hospitals. We further compare both standard and start-of-the-art methods on this new benchmark, with an emphasis on factual consistency. To accomplish this, we develop a tool to automatically extract concept-attribute tuples from the text. We then propose an evaluation metric, FactComp, to compare concept-attribute tuples between the human reference and generated conclusions. Both automatic and human evaluations show that there is still a significant gap between human-written and machine-generated conclusions on echo reports in terms of factuality and overall quality.

## introduction echocardiography (or echo) is a test that uses sound waves to produce live images of the heart @xcite . it has become routinely used to support the diagnosis, management, and follow-up of patients with suspected or known heart diseases. the echo report documents and communicates the evaluation of cardiac and vascular structures in the echocardiography study. as shown in figure 1 , a standard echo report usually consists of a demographic section, an echocardiographic evaluation section (also called the finding section), and a conclusion section @xcite . in a typical workflow, consultants who interpret echocardiography provide the quantitative measurement and descriptive statements to describe pertinent findings, and then conclude. in this work, we formally study the task of echo conclusion generation (echogen), arising in clinical practice to relieve the clinician of tasks that may contribute to clinician burnout @xcite . a practical system shall be able to generate statements that emphasize abnormal findings, and compare differences and similarities of the current study versus the previous one if available and relevant. we define echogen as a task of learning from the demographic and echocardiographic findings section and generating the conclusion section. neural network-based models @xcite ) are an attractive method for this task, but are difficult to apply without appropriate training data. to address this gap, we present a large-scale echogen benchmark, which consists of two datasets. here we reply on one prexisting mimic-iii dataset (egmimic) and one newly collected dataset from the newyork-presbyterian hospital (egclever) to cover different text genres, data sizes, and degrees of difficulty, and more importantly, highlight common challenges of echogen (figure 1 ). beyond data, a second challenge for echogen is to evaluate the factual correctness of a generated conclusion. automatic metrics such as rouge and meteor only assess content selection but not other quality aspects, such as fluency, grammaticality, and coherence, and are not well-correlated with factuality, leading to the development of separate evaluation measures @xcite @xcite . this study proposes a new evaluation metric to measure factual consistency, called "factcomp" by considering both concept and their attributes in the fact equivalence criteria. ## benchmark results and discussion baseline comparisons randsent and lexrank have similar performances on both datasets. the result is reasonable because lexrank relies on inter-sentence similarity to select sentences, but similarities between conclusion sentences are limited in clinical notes. the tf-idf baseline has contrary performance on two datasets. recall that this approach copies the reference directly from the report with the most similar source in the training data. since the "conclusion" section is written as structured noun phrases in egclever and as complete sentences in egmimic, tf-idf is more likely to achieve a higher rouge score in egclever, which has fewer lexical variations in the "conclusion" section. ## human evaluation to compare the quality of generated text against a human reference, we conduct a human evaluation following @xcite . we randomly sampled 50 echo reports from the development set of egmimic. for each example, we presented echo findings to two neurologist and pulmonary critical care physicians along with the human references and summaries generated from bart in random order. we asked the physicians to compare them in three dimensions (1) fluency, (2) factual consistency, and (3) overall quality. for each metric, we asked the physicians to select the better one, with ties allowed. since it is difficult to reach an agreement between physicians, we show the human evaluation result as confusion matrices in figure 4 . across all three dimensions, both physicians agree that human reference is better among half of the selected samples (the upper-left cell of each figure). further, most of the percentages fall into the top left two-by-two sub-matrices, with the main diagonal being the most frequent. this indicates that physicians have a consensus that generated conclusion is less preferred. there are also uncertainties about whether a reference is better or tied with a generated conclusion (around 20% at off-diagonal). overall, model-generated summaries are still un-desired compared to human reference in terms of fluency, factual consistency, and overall quality. ## limitations while our conducted human evaluation suggests that generated summaries from bart tend to have more factual errors than human reference, the accuracy of factuality comparison between bart and other baselines is still limited by the quality of our proposed system factext. its performance, especially recall, depends on the accuracy of the scis-pacy model we use and the number of common concepts we focus on (55 in this work). for example, we can integrate the recommended phrases that echocardiographers may choose to use to describe pertinent findings by the american society of echocardiography @xcite . we leave continually designing a more robust information extraction system or learning-based models, which both (1) rely less on domain-specific concepts; and (2) generalize to other types of notes, to future works. ## conclusion in this study, we introduce echogen, a new benchmark for evaluating and analyzing models for echocardiography report conclusion generation. we systematically analyze the performance of several baseline methods with our proposed evaluation metric and conclude that there is still a gap between human reference and generated conclusions for echo reports in terms of fluency and factual consistency. detailed analysis shows that our benchmarking can be used to evaluate the capacity of the models to understand the clinical text and, moreover, to shed light on the future directions for developing clinical text generation and summarization systems.