Overview of M i R eportor: Generating Reports for Multimodal Medical Images (2023)

Abstract: This demo paper presents a brief introduction of MiReportor, a computer-aided medical imaging report generator, which leverages a unified framework of medical image understanding and generation to predict readable descriptions for medical images, and assists radiologists in imaging reports writing.

## introduction in the intelligent-assisted diagnosis scenario, computers are required to present reliable interpretation of medical imaging findings. medical imaging report generation (mirg) integrates advanced technologies such as computer vision and natural language processing for identifying critical information from medical images and giving reasonable explanations @xcite . this demo paper presents a brief overview of mireportor (medical imaging report generator), a prototype system designed for computer-aided imaging report writing, open accessed by http://mireportor.com mireportor generates fluent imaging reports in both chinese and english and provides humancomputer interaction service for radiomics researchers on image reading, report reviewing and editing. ## system overview the initial design of mireportor was derived from the unified framework of medical image understanding and generation that we proposed earlier @xcite . it takes multimodal medical images, such as ct, x-ray, ultrasound, etc. as input, and predicts related semantic labels as well as brief readable descriptions of radiology findings. in the recent work, we refer to the latest progress on vision-language representation (feng, 2022) and update our workflow (see figure 1 ). ## evaluation considering the various linguistic and visual characteristics of different groups of people, we manually annotated a sentence template library of chest x-ray image reports. we used tmrgm to generate image reports for healthy and patient groups respectively. we validated the performance of chest x-ray caption prediction based on the iu chest x-ray dataset, see figure2: an example of imaging report generated by mireportor. ## conclusions this paper briefly introduces mireportor, a prototype system for interactive generation of 3 https://github.com/zhangyudoc/tmrgm medical imaging reports in both chinese and english. experiments based on public chest x-rays revealed the ability of computers on understanding and interpreting medical images. it facilitates the human-computer collaboration practice of imaging diagnosis, which may contribute to the efficient communication between radiologist, clinicians, and patients.