Medical Visual Textual Entailment for Numerical Understanding of Vision-and-Language Models (2023)

Abstract: Assessing the capacity of numerical understanding of vision-and-language models over images and texts is crucial for real vision-and-language applications, such as systems for automated medical image analysis. We provide a visual reasoning dataset focusing on numerical understanding in the medical domain. The experiments using our dataset show that current vision-and-language models fail to perform numerical inference in the medical domain. However, the data augmentation with only a small amount of our dataset improves the model performance, while maintaining the performance in the general domain.

## introduction vision-and-language models have made great progress on complex tasks, going beyond image recognition and towards reasoning over images and texts @xcite @xcite . following the success of pre-trained language models @xcite , recent advances in vision-and-language models have been made by the introduction of large-scale pretraining @xcite @xcite . however, as with pre-trained language models, it is unclear what information pretrained vision-and-language models learn and use in their predictions, and what their limitations are. while a large body of research @xcite @xcite has provided challenging reasoning tasks to probe the reasoning ability of pre-trained language models, such work has been more limited for vision-and-language models. furthermore, previous visual reasoning datasets are usually provided by the general domain of images, and analysis across different domains is desirable. our focus is to investigate whether current vision-and-language models have the ability to infer numerical relationships between images and texts in the medical domain, which is crucial for real vision-and-language applications such as systems for automated medical image analysis. consider the example of images and textual descriptions in a medical article presented in figure 1 . the lesion number affects the treatment strategy for diseases such as brain metastasis. if systems can automatically judge whether the lesion number in given images matches that in arbitrary query texts, they can support medical decision-making. recently, a vision-and-language model focusing on the medical domain @xcite ) has begun to be provided but is not yet fully developed. with this motivation, we provide a visual reasoning dataset focusing on numerical inference in the medical domain by adding annotations to the previous medical image and caption dataset medicat @xcite . we call our dataset medvte, which will be publicly available at https://github.com/ ynklab/medvte . using medvte, we investigate the extent to which current pre-trained visionand-language models have the ability of numerical understanding on visual reasoning tasks across images and texts in the medical domain. the experiments show that current models have much room to perform numerical inference in the medical domain. ## medvte datasets we introduce medvte, visual reasoning datasets in the medical domain involving numerical expressions. medvte is composed of pairs of medical images, captions, and three-class entailment labels (entailment, contradiction, or neutral). medvte focuses on the relationship between the number of lesions, such as cancer in an image and the numerical expression in a text. we created medvte by selecting examples involving numerical expressions from medicat dataset @xcite . medicat contains 217,060 figure-caption pairs in medical articles, whose captions sometimes refer to the number of the depicted lesions (e.g., tumors or nodules). the selection is conducted by one medical expert. ## conclusion we created the visual reasoning dataset medvte, focusing on numerical understanding in the medical domain. the experiments using medvte showed that current vision-and-language models struggled with performing numerical inference in the medical domain. however, the data augmentation with only a small amount of our medvte dataset improved the model performance, while maintaining the performance in the general domain. in future work, we increase the size of our med-vte dataset and make further analysis of visionand-language models to investigate the extent to which the size of a fine-tuning dataset affects the performance of numerical inference in the medical domain. improving automated numerical visionand-language understanding in the medical domain could aid therapeutic decision-making that depends on lesion numbers.