See Detail Say Clear: Towards Brain CT Report Generation via Pathological Clue-driven Representation Learning (2024)

Abstract: Brain CT report generation is significant to aid physicians in diagnosing cranial diseases.Recent studies concentrate on handling the consistency between visual and textual pathological features to improve the coherence of report.However, there exist some challenges: 1) Redundant visual representing: Massive irrelevant areas in 3D scans distract models from representing salient visual contexts.2) Shifted semantic representing: Limited medical corpus causes difficulties for models to transfer the learned textual representations to generative layers. This study introduces a Pathological Clue-driven Representation Learning (PCRL) model to build cross-modal representations based on pathological clues and naturally adapt them for accurate report generation.Specifically, we construct pathological clues from perspectives of segmented regions, pathological entities, and report themes, to fully grasp visual pathological patterns and learn cross-modal feature representations. To adapt the representations for the text generation task, we bridge the gap between representation learning and report generation by using a unified large language model (LLM) with task-tailored instructions. These crafted instructions enable the LLM to be flexibly fine-tuned across tasks and smoothly transfer the semantic representation for report generation.Experiments demonstrate that our method outperforms previous methods and achieves SoTA performance.Our code is available at https://github.com/Chauncey-Jheng/PCRL-MRG.

## introduction brain computed tomography (ct) imaging is essential for diagnosing various cranial diseases, including cerebral infarction and hemorrhage. however, it is time-consuming and error-prone for radiologists to manually interpret medical findings from these scans and write reports. automated report *corresponding author figure 1 : comparison of different cross-modal alignment paradigms. (a) previous way: the origin ct images contain extraneous information unrelated to diagnosis, and separate training for two semantic tasks makes it challenging to find a shared optimal solution, resulting in inadequate cross-modal alignment. (b) our way: the visual representation is refined to concentrate on pathological clues, and employ joint training with task-tailored instructions via unified llm to find a transferable representation, leading to better adaption for report generation. with the advancement of deep neural networks and their successful application in image captioning tasks @xcite , medical report generation (mrg) has gained more attention. unlike the short sentences of traditional image captioning, mrg aims to generate lengthy and precise reports. to achieve this, various crossmodal alignment methods are required to ensure the consistency between visual and textual information @xcite , including attention mech-anisms @xcite , memory mechanisms @xcite , and knowledge graphs @xcite . recently, learning representations via visualtextual contrastive learning @xcite or using pre-trained large language models (llms) @xcite to strength representations are also proven to be effective. however, as shown in figure 1 (a), learning crossmodal correspondences is still challenging in current methods due to the following concerns: 1) redundant visual representing. different from chest x-ray data, 3d brain ct scans contain extensive redundant information, e.g. background and insignificant areas. with the lack of human-crafted boxes to locate pathology regions, models struggle to capture and interpret the visual pathology patterns for generating reports. although current advanced models use semantic prior knowledge or medical prompts @xcite to automatically learn the salient visual areas, this may introduce noise and unstable representation and cause severe hallucinations in generated texts. 2) shifted semantic representing. compared to natural corpus, limited brain ct report corpus is insufficient to transfer the pathological semantic representations learned by represent learning layers (e.g., contrastive learning layer) to the language model @xcite , since the direct weight-sharing @xcite is prone to degrade the coherence of generated diagnostic sentences. thus, how to uniformly represent cross-modal pathological features and adapt them to report generation still remains an open question. in this paper, we propose a pathological cluedriven representation learning (pcrl) model to seamlessly build cross-modal representations based on diverse pathological clues and transfer them for generating accurate brain ct reports. specifically, we extract pathological clues from perspectives of segmented regions, pathological entities, and report themes to depict clinical scenarios. segmented region clues are automatically generated and filtered by given pathology prompts, enabling the visual encoder to grasp visual pathological patterns. meanwhile, entity and theme clues are respectively extracted by detailed findings and full-text reports, to handle the enriched visual-textual alignment and build cross-modal pathology representations. besides, to adapt the learned representations for the text generation task, we bridge the gap between representation learning and report generation by employing a unified large language model (llm) with task-tailored instructions, which has proven to be more effective than conventional decoders by using appropriate tokens to seamlessly connect different tasks @xcite . as shown in figure 1(b), we craft a representation instruction to prompt the llm to produce high-level pathological semantic features for cross-modal alignment, and a generation instruction to prompt llm to generate accurate brain ct reports based on the learned representations. our main contributions can be summarized as: ## methodology as illustrated in figure 2 , our framework consists of two branches: the brain ct report generation (rg) on the left and the pathological clue-driven representation learning (pcrl) on the right. the interaction between these branches is facilitated through the shared use of a visual encoder and a language model. ## conclusion we propose a novel model to mine pathological clues for enhancing multi-modal representations and seamlessly transfer them into report generation. first, through carefully designed segmentation clue alignment, entity clue alignment, and theme clue alignment, the diverse and precise feature representation can be well-constructed. second, we transfer the learned representation to boost the brain ct report generation via a unified llm prompted by task-tailored instructions. experiments demonstrate the effectiveness of our model in generating pathologically accurate reports.