Towards Visual Question Answering on Pathology Images (2021)

Abstract: Pathology imaging is broadly used for identifying the causes and effects of diseases or injuries. Given a pathology image, being able to answer questions about the clinical findings contained in the image is very important for medical decision making. In this paper, we aim to develop a pathological visual question answering framework to analyze pathology images and answer medical questions related to these images. To build such a framework, we create PathVQA, a VQA dataset with 32,795 questions asked from 4,998 pathology images. We also propose a three-level optimization framework which performs self-supervised pretraining and VQA finetuning end-to-end to learn powerful visual and textual representations jointly and automatically identifies and excludes noisy self-supervised examples from pretraining. We perform experiments on our created PathVQA dataset and the results demonstrate the effectiveness of our proposed methods. The datasets and code are available at https://github.com/UCSD-AI4H/PathVQA

## introduction pathology @xcite studies the causes and effects of diseases or injuries. it underpins every aspect of patient care, such as diagnostic testing, providing treatment advice, preventing diseases using cutting-edge genetic technologies, to name a few. given a pathology image, being able to answer questions about the clinical findings contained in the image is very important for medical decision-makings. in this paper, we aim to develop a pathological visual question answering framework to analyze pathology images and answer medical questions related to these images. we first need to col- * equal contribution lect a dataset containing questions about pathology imaging. one possible way to create a pathology vqa dataset is crowdsourcing, which is used successfully for creating general domain vqa datasets @xcite @xcite @xcite . however, it is much more challenging to build medical vqa datasets than general domain vqa datasets via crowdsourcing. first, medical images such as pathology images are highly domain-specific, which can only be interpreted by well-educated medical professionals. it is rather difficult and expensive to hire medical professionals to help create medical vqa datasets. second, to create a vqa dataset, one first needs to collect an image dataset. while images in the general domain are pervasive, medical images are very difficult to obtain due to privacy concerns. to address these challenges, we resort to pathology textbooks, especially those that are freely accessible online, as well as online digital libraries. we extract images and captions from the textbooks and online digital libraries. given these images, question-answer pairs are created based on image captions. these qa pairs are verified by medical professionals to ensure clinical meaningfulness and correctness. in the end, we created a pathology vqa dataset called pathvqa, which contains 32,795 questions asked from 4,998 pathology images. to our best knowledge, this is the first dataset for pathology vqa. given the pathology vqa dataset, the next step is to develop a pathology vqa system, which is also very challenging, due to the following reason. the medical concepts involved in pathvqa are very diverse while the number of question-answer pairs available for training is limited. learning effective representations of these diverse medical concepts using limited data is technically difficult. poorly learned representations lead to infe-q1: what are dilated and congested? q2: are the sinuses dilated and congested? q3: is there increased fibrosis in the red pulp, capsule and the trabeculae? q4: where is increased fibrosis? q5: is gamna-gandy body also seen? q1: what is slightly depressed on the surface? q2: where is the wedge-shaped infarct slightly depressed? q3: is the wedge-shaped infarct slightly depressed on the surface? q4: what is on the surface? q5: what is pale while the margin is haemorrhagic? rior vqa performance. to address this challenge, we propose a three-level optimization framework which performs cross-modal self-supervised pretraining (tan and bansal, 2019) and vqa finetuning of a pathology image encoder and a question encoder end-to-end to learn powerful visual and textual representations jointly and automatically identifies and excludes noisy self-supervised examples from pretraining. experiments on our developed pathvqa dataset demonstrates the effectiveness of our proposed methods. the major contributions of this paper are as follows: ## the pathvqa dataset the pathvqa dataset consists of 32,795 questionanswer pairs generated from 1,670 pathology images collected from two pathology textbooks: "textbook of pathology" @xcite and the question-answer pairs are generated using a semi-automated pipeline with linguistic rules. figure 1 shows some examples. on average, each image has 6.6 questions. the maximum and minimum number of questions for a single image is 14 and 1 respectively. the average number of words per question and per answer is 9.5 and 2.5 respectively. there are eight different categories of questions: what, where, when, whose, how, why, how much/how many, and yes/no. ## method we propose a three-level optimization based framework to perform vqa on pathvqa. in our framework, there are three learning stages, which are performed end-to-end jointly. in the first stage, self-supervised learning @xcite is performed to pretrain the image encoder and text encoder. in the second stage, we finetune the image encoder and text encoder on the pathvqa dataset. in the third stage, the trained model is validated on the validation set. in the first stage, we perform cross-modal self-supervised learning @xcite of an image en-1 http://peir.path.uab.edu/library/ index.php?/category/2 coder w and a text encoder t . the image encoder is used to extract visual features of pathology images. the text encoder is used to extract semantic features of questions and answers. self-supervised learning @xcite is an unsupervised representation learning approach where pretext tasks are defined solely based on the input data, and representations are learned by solving these pretext tasks. there are many ways to construct pretext tasks. in our work, following @xcite , we define a simple yet effective pretext task: in the pathvqa dataset, given a pathology image and a question, judge whether this question is about this image. from the pathvaq training set d, we create another dataset @xmath0, y i , t i )} m @xmath1. there are m tuples, each containing a pathology image x from d and a question y from d. t i is a binary variable where t @xmath2. given d , we develop a model to map (x i , y i ) to t i . in this model, an image encoder is used to encode x i and a text encoder is used to encode y i ; the concatenation of these two encodings is fed into a linear layer to predict whether the image matches with the question. in self-supervised learning @xcite , the labels are typically constructed automatically without human supervision. as a result, they contain a lot of noises. for example, in d , t is determined simply based on whether x and y are from the training example in d. it is totally possible that a question y asked about an image x is appropriate to be a question for another image x as well if x and x are pathologically similar. in this case, the correct label t for (x, y) should be 1. however, it is set to 0 in d . training the encoders using these noisy and incorrect labels may confuse the encoders and result in poor-quality representations. to address this problem, we aim to develop a method to automatically identify incorrectly autolabeled examples in the training data of the ssl task. for each example (x, y, t) in d , we associate a selection variable @xmath3, 1] with it. if a is close to 1, it means this example is correctly labeled; if a is close to 0, it means this example is incorrectly labeled. let l(f (x, y; w, t ), t) denote the ssl loss defined on (x, y, t), where f (x, y; w, t ) is the predicted probability that @xmath4. we multiply a with l(f (x, y; w, t ), t) so that if (x, y, t) is incorrectly labeled, its loss will be down-weighted to 0 and effectively (x, y, t) is excluded from the ssl pretraining process. in the end, only correctly-labeled examples are used for pretraining the encoders. to this end, in the first stage, we solve the following optimization problem: in this problem, the selection variables @xmath5. {a i } m @xmath6. w and t are trained by minimizing the sum of weighted losses. note that the optimal solutions w * (a) and t * (a) are functions of a since w * (a) and t * (a) are functions of the loss function, which is a function of a. in the second stage, we finetune the image encoder and text encoder in the vqa task defined on the pathvqa dataset d. let v , u , r denote the network weights of the image encoder, text encoder, and qa network respectively. we train v , u , r by minimizing the vqa loss: n (tr) @xmath7 in the third stage, we apply the optimally trained vqa model including v * (w * (a)), u * (t * (a)), and r * to make predictions on the validation dataset. putting all these pieces together, we have the following three-level optimization framework: ## conclusion in this paper, we build a pathology vqa dataset -pathvqa -that contains 32,795 question-answer pairs of 8 categories, generated from 4,998 images. majority of questions in our dataset are open-ended, posing great challenges for the medical vqa research. our dataset is publicly available. to address the challenges that the self-supervised training data may contain errors and the effective representations of pathology images and questions are difficult to learn on limited data, we propose a threelevel optimization framework to automatically identify and remove problematic ssl training examples and learn sample-efficient visual and textual representations. experiments on the pathvqa dataset demonstrate the effectiveness of our method.