MIMOQA : Multimodal Input Multimodal Output Question Answering (2021)

Abstract: Multimodal research has picked up significantly in the space of question answering with the task being extended to visual question answering, charts question answering as well as multimodal input question answering. However, all these explorations produce a unimodal textual output as the answer. In this paper, we propose a novel task - MIMOQA - Multimodal Input Multimodal Output Question Answering in which the output is also multimodal. Through human experiments, we empirically show that such multimodal outputs provide better cognitive understanding of the answers. We also propose a novel multimodal question-answering framework, MExBERT, that incorporates a joint textual and visual attention towards producing such a multimodal output. Our method relies on a novel multimodal dataset curated for this problem from publicly available unimodal datasets. We show the superior performance of MExBERT against strong baselines on both the automatic as well as human metrics.

## introduction multimodal content is at the heart of digital revolution happening around the world. while the term modality has multiple connotations, one of its common usage is to indicate the content modality i.e. images, text, audio etc. it has been shown that multimodal content is more engaging and provides better cognitive understanding to the end user @xcite @xcite . with recent improvements in visionlanguage grounding and multimodal understanding @xcite @xcite , several works have explored beyond unimodal machine comprehension @xcite @xcite towards a holistic multimodal comprehension @xcite @xcite @xcite ) with significant improvements. however, all these explorations on multimodal understanding, question answering in particular, have limited their focus to unimodal outputs even with multimodal inputs. for example -visual question answering (vqa) task takes a textual query and an image to produce a textual answer. the multimodal question answering tasks @xcite @xcite take multiple input modalities, but the output is limited to text only. even the recently proposed manymodalqa @xcite relies on multimodal understanding to produce a textual answer. these works implicitly assume that the textual answers can satisfy the needs of the query across multiple input modalities. we posit that such an assumption is not always true; while textual answer can address several queries, a multimodal answer almost always enhances the cognitive understanding of the end user; understanding the answer through visuals is faster and provides enhanced user satisfaction. in this paper, we propose a new task, multimodal input multimodal output question answering (mimoqa), which not only takes multimodal input but also answers the question with a multimodal output. our key contributions are: 1) we introduce the problem of multimodal input multimodal output question answering. we establish the importance of such multimodal outputs in question-answering for enhanced cognitive understanding via human experiments. ## multimodal output multimodal output not only provides better understanding to the end user but also provides grounding to the actual answer. for e.g., the multimodal output for the question in figure 1 (a) aids in better comprehension of the answer, while also providing grounding to words like 'stick', 'knob'. in some cases, textual answer might even be insufficient, especially, for questions which seek explicit visual understanding (questions about colors, structures, etc). in such cases, existing systems apply image understanding on top of the images to arrive at a 'textual description' of the desired answer. while this might suffice in some cases, a multimodal output can almost always enhance the quality of such answers. in fig. 1(b) , the textual answer is insufficient and gets completed only with the help of the final image-text combination. ## dataset & pretraining since there is no existing dataset which satisfies the requirements of the task, we curate a new dataset (refer to supplementary for details on curation strategy and data samples) by utilizing the existing public datasets. we observe that several qa datasets contain answers that come from a wikipedia article. since most wikipedia articles come with a set of related images, such images could feature as the input i in our setup. extending this heuristic, we use two qa datasets -ms-marco @xcite and natural question (nq) @xcite , to extract those question-answer pairs which are originally extracted from wikipedia and scrape all images from the original article. more details about the curation process and examples of the images scraped for questions can be found in the appendix. figure 3 : calculation of proxy supervision scores for an image. we compute the tf-idf similarities of the caption with the question, answer and relevant paragraph, and also compute the distance of the image from the answer paragraph. these are summed in a weighted fashion to get the final score. ## experiments we conduct extensive experiments and ablations for the proposed mexbert framework and compare it against the e&m baseline. we divide our curated dataset into train, development and test sets as shown in for the e&m baseline, we pretrain the text extraction with the squad dataset @xcite and finetune it on our dataset. for the image matching, we use image ranking using the input query (q), input passage p and the extracted input answer a all concatenated together. for mexbert, we tested different variants with and without proxy supervision (ps); with different pre-training setups -pretraining the textual stream alone, visual stream alone and both -to test the independent value of different pre-training. except pretraining experiments and baseline experiments, all our experiments on mexbert have been conducted with 3 random seeds and the reported scores have been averaged over the 3 seeds. we use bert pretrained embeddings for the textual stream of mexbert and use n @xmath4. for finetuning mexbert, we use adam optimizer initialized with a learning rate of 0.0001 and train it till the validation loss saturates. the model was trained over 4 v100 machines using a batch size of 8 for finetuning and 64 for pretraining. for pretraining, we use an adam optimizer with a learning rate of 0.0001 for 2 epochs over 3.2 million image-text pairs for all our ablations during pretraining stage. we use 768 dimensional textual embeddings with a vocabulary size of 30, 522 and intermediate hidden embedding size 3072 for both textual and visual features. we project 4096 dimensional vgg-19 image features into 2048 dimensions and use it as input to the visual stream. evaluation metrics: we independently evaluate the text and image part of the extracted answer using various metrics. for the text, we considered standard metrics like rouge, bleu popularly used in the literature for textual question answering task. for images, we use the precision @1,2 and 3 in which we measure if the predicted image is in top-1,2 or 3 images as selected in the ground truth. although these metrics are standard, we verify their utility in the multi-modal case by conducting a human experiment and calculating their correlations with human judgments. to further validate the choice of our metrics, we collated a subset of 200 examples which have their ground truth available (collected as discussed later). we, then, apply our best performing model for these examples and generate the multimodal answers. for each of 200 pairs, we have both its predicted as well as ground truth counterparts. we conduct a human experiment where the annotators are asked to rate the quality of both textual and image part of the answer on relevance r and user satisfaction s. the overall quality of the answer is high if it is both relevant and provides high user satisfaction. for each pair, 5 different annotators rate the answers resulting in independent ratings for both predicted and ground truth answers. we calculate the overall quality of a predicted answer q a with respect to the ground truth by calculating the ratio between the quality (which we represent by r*s) of predicted answer and the ground truth answer, q @xmath5. we compute the pearson correlation between different metrics and q a . we observe that rouge-1, rouge-2, rouge-l and bleu yielded a correlation scores of 0.2899, 0.2716, 0.2918 and 0.2132 -indicating a moderate correlation and reassuring their viability for evaluating textual answer even in our multimodal setup. for image metrics, we found precision@1 to be most strongly correlated with human judgement (0.5421). while the expectation might be that such a metric has a perfect correlation, the user judgement is also biased by the corresponding textual answer leading to different scores even if the image is same in actual and predicted answer. evaluating textual outputs: to overcome unimodal reasoning, there are attempts at truly multimodal reasoning with the datasets such as manymodalqa @xcite , recipeqa @xcite , and tvqa @xcite . while recipeqa aims reasoning over recipes and the associated pictures, tvqa involves multimodal comprehension over videos and their subtitles. the recently proposed manymodalqa goes a step further by adding tables to the multimodal reasoning as well. however, these datasets provide responses in a single modality only, either an mcq or textual response. with the rate at which multimodal consumption is taking place in our lives, it is important that the answering systems also enable multimodal output which, as discussed, already can provide better cognitive understanding when combined with textual modality. ## conclusion we presented one of the first exploration, to the best of our knowledge, of multimodal output question answering from multimodal inputs and proposed usage of publicly available textual datasets for it. we proposed strong baselines by utilizing the existing frameworks for extract textual answers and independently match them with an appropriate image. we demonstrate the value of a joint-multimodal understanding for multimodal outputs in our problem setup by developing a multimodal framework mexbert which outperformed the baselines significantly on several metrics. we also developed a proxy supervision technique in absence of labelled outputs and showed its effectiveness for improved multimodal question answering. we used some existing metrics to compare the different models and justified the usage of these metrics based on a human experiment. while it is an interesting and challenging task even in its current shape, we believe there are several limitations in our proposed framework. while our datasets had multimodal elements, modeling multimodal reasoning from multimodal inputs and using it to arrive at a multimodal answer calls for a more careful question curation that includes these challenges. recently proposed datasets such as multimodalqa have created questions explicitly aimed at reasoning across multimodal input, but however, lack the multimodal output component. future works could include questions which specifically aim for a visual elements making the output requirement multimodal. also, free form answer generation in the multimodal input/output context is another interesting subject of further research. junnan zhu, haoran li, tianshang liu, yu zhou, jiajun zhang, and chengqing zong. 2018. msmo: multimodal summarization with multimodal output. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 4154-4164.