V i LM edic: a framework for research at the intersection of vision and language in medical AI (2022)

Abstract: There is a growing need to model interactions between data modalities (e.g., vision, language) — both to improve AI predictions on existing tasks and to enable new applications. In the recent field of multimodal medical AI, integrating multiple modalities has gained widespread popularity as multimodal models have proven to improve performance, robustness, require less training samples and add complementary information. To improve technical reproducibility and transparency for multimodal medical tasks as well as speed up progress across medical AI, we present ViLMedic, a Vision-and-Language medical library. As of 2022, the library contains a dozen reference implementations replicating the state-of-the-art results for problems that range from medical visual question answering and radiology report generation to multimodal representation learning on widely adopted medical datasets. In addition, ViLMedic hosts a model-zoo with more than twenty pretrained models for the above tasks designed to be extensible by researchers but also simple for practitioners. Ultimately, we hope our reproducible pipelines can enable clinical translation and create real impact. The library is available at https://github.com/jbdel/vilmedic .

## introduction in the past few years, there has been a surge of interest in multimodal problems, especially involving visio-linguistic data that occurs "in the wild", from image captioning @xcite @xcite to visual question answering @xcite @xcite and beyond. multimodal tasks are interesting because many real-world problems are multimodal in nature. this is also the case in medical ai where many repetitive tasks lie at the intersection of vision and language. for example, radiologists must generate and summarize reports from x-ray images, or answer medical questions from patients. as a response, tasks such as radiology report generation @xcite , where assistive systems that take x-ray images of a patient and generate a textual report describing clinical observations or medical visual question answering have been proposed. recent multimodal training techniques, such as contrastive learning, have also enabled powerful multimodal embeddings that contribute to higher quality in-domain image representations that capture the subtlety of visual features required for medical image understanding tasks and annotation-efficient learning @xcite . these recent advances have identified new challenges. first, it is not always clear to what extent truly visio-linguistic reasoning and understanding is required for solving tasks where images and text are available. language can inadvertently impose strong priors that result in seemingly impressive performance without any understanding (or active use) of the visual content. this challenge has been pointed out for tasks involving natural images, such as visual question answering @xcite or multimodal machine translation @xcite , but also involving medical images, such as slice discovery @xcite or multimodal radiology report summarization @xcite . secondly, multimodal training has been identified as a challenging learning task by nature: multimodal networks are prone to overfitting due to their increased capacity and modalities overfit and generalize at different rates @xcite . another challenge in medical ai is transparency. despite much promising research currently being undertaken, particularly in imaging, the literature as a whole lacks clear reporting to facilitate replicability, exploration for potential ethical concerns, and clear demonstrations of effectiveness @xcite . @xcite evaluated 511 scientific papers across several machine learning subfields and found that machine learning for health compared poorly to other areas regarding reproducibility metrics, such as dataset and code accessibility. as an example, mckinney et al. (2020) demonstrated a system that improves the speed and robustness of breast cancer screening, while highlighting the challenges of making such work reproducible. this absence of sufficiently documented methods and computer code underlying the study has effectively undermined its scientific value @xcite . to address the aforementioned shortcomings related to multimodal training and medical ai, we propose vilmedic ( §3), an open-source vision-and-language medical library. vilmedic emphasizes technical reproducibility by packaging common operations (e.g., linguistic or visual encoding) as "blocks" (3.1), and by defining "solutions" (3.2), the full pipeline of a certain multimodal technique published in the literature, as an assembly of blocks. with the abstraction of blocks and example configurations of these blocks to form solutions, vilmedic gives the user an easy to use interface to (i) reproduce the results reported in the literature, and (ii) investigate novel multimodal techniques quickly. in addition to blocks and solutions, vilmedic hosts a model-zoo (3.3) containing trained solutions usable in one line of code. as of 2022, vilmedic contains dozen of solutions replicating the state-of-the-art results for problems that range from medical visual question answering and radiology report generation to multimodal representation learning on widely adopted medical datasets, and more than twenty pretrained models for the above tasks. ## related work recent progress in natural language processing and computer vision has been driven by advances in both model architecture and model pretraining. namely, transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of medical tasks. the library huggingface transformers @xcite provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio. vilmedic takes advantage of the available medical language representation models hosted on the huggingface model hub, such as clinicalbert @xcite and biomed-roberta @xcite , to fine-tune their representations on downstream medical multimodal tasks. another related work is the multimodal framework (mmf, @xcite al. (2018) ), a deep learning library for vision and language multimodal research for natural images. mmf has the same philosophy as vilmedic: the library replicates the architectures and results from the literature and provides baseline implementations for machine learning challenges. however, the mmf framework is oriented towards handling data that occurs "in the wild" (vqa, textvqa) or on the internet (hateful memes detection). as of today, no medical tasks are addressed. nevertheless, mmf contains state-of-the-art visio-linguistic architectures, such as visualbert @xcite and vilt @xcite whose efficiency does not straightforwardly translate in the medical domain but that could be useful for vilmedic in the future. in the medical field, torchxrayvision @xcite ) is an open-source software library for working with chest x-ray datasets and deep learning models. it also provides a common interface and common pre-processing chain for a wide set of publicly available chest x-ray datasets. it differs from vilmedic in two ways: first, it offers very limited tools and details to re-train the models and second, it primarily focuses on releasing pretrained visual encoder (namely, densenet @xcite ). vilmedic focuses on training and releasing pretrained models that are multimodal across four major medical tasks and also emphasizes technical reproducibility. nonetheless, we have integrated torchxrayvision as a component of our library. ## vilmedic vilmedic consists of blocks, solutions and a model-zoo. a block ( §3.1) is a common operations defined as a snippet of code. blocks can be a piece of a neural network architecture (e.g., a resnet @xcite encoding the image in a multimodal solution), a loss function, or an evaluation metric. therefore, a block can be suitable for several solutions. solutions ( §3.2) are defined as the full pipeline of a certain multimodal technique published in the literature. that is, a solution contains pre-processed data (or the pre-processing scripts), open-source architecture implementations, proper training parameters, and evaluation scoring. a solution is composed of independent blocks and defined in a configuration file. finally, most of our solutions are trained and stored in a model-zoo ( §3.3), saving researchers and practitioners time and effort (appendix f). our blocks, solutions and model-zoo are supported by a documentation available at https://vilmedic.readthedocs.io/ en/latest/ . ## conclusion and future work we presented vilmedic, a framework for research at the intersection of vision and language in medical ai. we have reproduced and make publicly available state-of-the-art medical ai models, as well as implemented custom solutions that exceed their performance. our goal is to maintain the library up-to-date with new blocks and solutions that can serve as a standard for benchmarking results across vision and language medical ai tasks. we also hope our library will be used to generate new ideas and publications.