Granularity Matters: Pathological Graph-driven Cross-modal Alignment for Brain CT Report Generation (2023)

Abstract: The automatic Brain CT reports generation can improve the efficiency and accuracy of diagnosing cranial diseases. However, current methods are limited by 1) coarse-grained supervision: the training data in image-text format lacks detailed supervision for recognizing subtle abnormalities, and 2) coupled cross-modal alignment: visual-textual alignment may be inevitably coupled in a coarse-grained manner, resulting in tangled feature representation for report generation. In this paper, we propose a novel Pathological Graph-driven Cross-modal Alignment (PGCA) model for accurate and robust Brain CT report generation. Our approach effectively decouples the cross-modal alignment by constructing a Pathological Graph to learn fine-grained visual cues and align them with textual words. This graph comprises heterogeneous nodes representing essential pathological attributes (i.e., tissue and lesion) connected by intra- and inter-attribute edges with prior domain knowledge. Through carefully designed graph embedding and updating modules, our model refines the visual features of subtle tissues and lesions and aligns them with textual words using contrastive learning. Extensive experimental results confirm the viability of our method. We believe that our PGCA model holds the potential to greatly enhance the automatic generation of Brain CT reports and ultimately contribute to improved cranial disease diagnosis.

## introduction writing diagnostic reports for brain ct imaging with multiple scans is widely applied in medicine, to summarize the findings of cranial diseases. nonetheless, this traditional clinical practice could be time-consuming and error-prone for radiologists due to some subjective factors (e.g. fatigue and distraction) @xcite . a computer-aided ti ss ue g ra ph hi gh de ns ity ba sa l ga ng li a oc ci pi ta l lo be te m po ra l lo be ed em a l es io n g ra ph report: pachy high density with low density around in the right occipital lobe. ...... low-density edema around in the temporal lobe. speckled patchy low density in basal ganglia, parietal ventricle, and centrum semioval. reporting system aims to automatically generate accurate reports, which has the potential to lighten the workload of physicians and economize insufficient clinical resources in populated areas. with the advent of deep neural networks in image captioning @xcite , medical report generation (mrg) methods are increasingly ramping up @xcite @xcite @xcite @xcite @xcite . different from image captioning, mrg task focuses on subtle yet crucial medical terminologies, with the report length typically 4-6 times longer than those of nature object captions. this prompts mrg models to refine the dedicated consistency of salient pathological features between visual and textual modalities. to achieve this, recent studies have employed various cross-modal alignment mechanisms @xcite @xcite , leveraging specific knowledge to effectively improve report generation. despite the promising achievements, previous methods are still thwarted by the following two concerns. 1) coarse-grained supervision: in clinical practice, brain findings are often characterized by some subtle yet vital pathological elements that belong to different attributes (i.e. tissues and lesions) @xcite , see figure 1 . however, medical images and long reports are always treated as coarse-grained supervisory signals in the mainstream methods. although few recent studies have explored the learning of fine-grained signals @xcite in chest x-ray samples, it is unsuitable for brain ct which encompasses more sophisticated pathology patterns. thus, how to leverage the intrinsic fine-grained cranial knowledge to recognize subtle abnormalities remains an open question. 2) coupled crossmodal alignment: current cross-modal alignment methods in mrg inevitably tend to couple visualtextual representation during coarse-grained alignment, which is manifested as highly-similar attention maps @xcite or tangled semantic representations in instance-level contrastive learning @xcite , resulting in inadequate feature learning. in this work, we propose a novel pathological graph-driven cross-modal alignment model, named pgca, which introduces a detailed pathological graph (pg) to seamlessly capture domain knowledge in samples, and explicitly extract finegrained semantic correspondence between visual regions and diagnostic texts for accurate brain ct report generation. specifically, the pg contains heterogeneous nodes of tissue and lesion attributes, which are connected by edges with prior knowledge. as shown in figure 1 , we decompose pg into two fixed tissue graph and lesion graph with intra-attribute knowledge (relations in each attribute), and one dynamic tissue-lesion graph with inter-attribute knowledge (correspondences between two attributes for each case). then, we respectively incorporate tissue and lesion graphs into an attribute-oriented graph convolution network for dedicated node features learning, which is jointly updated via an intra-attribute classification (iac) loss module. while the tissue-lesion graph is updated via an inter-correlation alignment (ica) loss module, which facilitates learning the complex tissue-lesion connections, in turn, benefits node feature representation. finally, a crossmodal contrastive learning (cl) module is pro-posed to reconcile the learned visual features with their corresponding word embeddings in the report, to further boost the visual encoder and textual decoder in report generation. in sum, our main contributions contain: ## methodology as shown in figure 2 , our framework contains two parallel branches, namely brain ct report gener-ation and pathological graph-driven cross-modal alignment (pgca), which interact by the shared visual and textual embedding layers. ## conclusion we propose a pathological graph-driven crossmodal alignment model for brain ct report generation. first, we construct a pathological graph that incorporates detailed medical knowledge, allowing for the injection of this knowledge into dedicated node features through graph embedding and updating, thereby achieving fine-grained visual representations. second, we align the learned node features with the corresponding word embeddings by crossmodal contrastive learning, to further boost report generation. extensive experiments demonstrate that our model achieves superior performance in generating clinically accurate reports.