Visuo-Linguistic Question Answering ( VLQA ) Challenge (2020)

Abstract: Understanding images and text together is an important aspect of cognition and building advanced Artificial Intelligence (AI) systems. As a community, we have achieved good benchmarks over language and vision domains separately, however joint reasoning is still a challenge for state-of-the-art computer vision and natural language processing (NLP) systems. We propose a novel task to derive joint inference about a given image-text modality and compile the Visuo-Linguistic Question Answering (VLQA) challenge corpus in a question answering setting. Each dataset item consists of an image and a reading passage, where questions are designed to combine both visual and textual information i.e., ignoring either modality would make the question unanswerable. We first explore the best existing vision-language architectures to solve VLQA subsets and show that they are unable to reason well. We then develop a modular method with slightly better baseline performance, but it is still far behind human performance. We believe that VLQA will be a good benchmark for reasoning over a visuo-linguistic context. The dataset, code and leaderboard is available at https://shailaja183.github.io/vlqa/ .

## introduction question answering (qa) is a crucial way to evaluate the system's ability to understand text and images. in recent years, a large body of natural language qa (nlqa) datasets and visual qa (vqa) datasets have been compiled to evaluate the ability of a system to understand text and images. for most vqa datasets, the text is used merely as a question-answering mechanism rather than an actual modality that provides contextual information. on the other hand, deriving inference from combined visual and textual information is an important ⇤ corresponding author skill for humans to perform day-to-day tasks. for example, product assembly using instruction manuals, navigating roads while following street signs, interpreting visual representations (e.g., charts) in various documents such as newspapers and reports, understanding concepts using textbook-style learning, etc. the importance of joint reasoning has also been emphasized in the design of standardized / psychometric tests like pisa (oecd, 2019) and gre foot_0 , as evident from figure 2 . pisa assessments conducted post 2018 take into account "the evolving nature of reading in digital societieswhich requires an ability to compare, contrast and integrate information from multiple sources". the gre has 'data interpretation' questions that assess a student's ability to "analyze given data as a combination of text and charts." both the aforementioned evidence motivate the need to develop visuo-linguistic qa (vlqa) system, posing a further challenge to state-of-the-art vision and language research. there are no benchmarking datasets that focus on reasoning over both images and text to our best knowledge. we formalize the task of deriving joint inference, where a system must utilize both visual and textual information to correctly answer the question, as demonstrated in figure 1 . to create a benchmark for this task, we develop and present a new dataset: vlqa (visuo-linguistic question answering) foot_2 as our main contribution. vlqa dataset consists of text together with a diverse range of visual elements. since manuals, documents and books containing texts and visuals are ubiquitous, the vlqa dataset is very much grounded in the real world. the dataset is curated from multiple resources (books, encyclopedias, web crawls, existing datasets, etc.) through combined automated and manual efforts. the dataset consists of 9267 image-passage-qa tuples with detailed annotation, which are meticulously crafted to assure its quality. we then evaluate the best existing visionlanguage architectures with respect to our vlqa dataset. this includes lxmert @xcite , vl-bert @xcite , vilbert @xcite and @xcite . our results demonstrate that despite a significant improvement over vision and language tasks separately, the best existing techniques cannot reason well on the joint tasks. we then propose a modular method hole (hopping and logical entailment), which demonstrates slightly better baseline performance and offers more transparency for the interpretation of intermediate outputs. the results indicate that vlqa task is relatively harder compared to existing vision-language tasks due to diversity of figures and additional textual component, demanding the need of better approaches to tackle multi-modal question answering. the vlqa challenge thus has the potential to open new research avenues spanning language and vision. ## related work we identify image-text multi-modality, multi-hop reasoning and variants of visual question answering (vqa) closest to vlqa and compare with relevant datasets in these areas (refer appendix a.1 for comprehensive comparison with more datasets). ## vlqa dataset we formally define the vlqa task, explain our approach to curate this dataset and necessary measures for quality assurance below; ## benchmarking human performance we performed human evaluation on 927 test samples with a balanced variety of questions by image types, answer types, knowledge/reasoning types and hardness. first, we ask 3 in-house experts to take tests in isolation. we also ask them to rate questions based on the difficulty levels (easy/medium/hard) and an option to mark a dataset sample 'ambiguous'. then we match their predictions against ground-truth answers, which turned out to be 84%. random baseline vlqa dataset contains 4way and 2-way multiple choice questions (mcqs) where each answer choice is likely to be picked with 25% and 50% chance. based on the answertype distribution provided in ## results & discussion multi-modality brings both pros and cons while developing new artificial intelligence (ai) benchmarks. the presence of multiple modalities provide natural flexibility for varied inference tasks, simultaneously making the reasoning process more complex as information is now spanned across them and requires cross-inferencing. in this work, we focused on joint reasoning over image-text multimodal context and developed a visuo-linguistic question answering (vlqa) dataset. our proposed vlqa dataset has important distinctions from existing vqa datasets. firstly, it incorporates a text passage that contains additional contextual information. secondly, it offers various figure types including natural images, templated images and free-form images (unstructured), which is not so common for other vqa datasets. thirdly, it tests diverse reasoning capabilities, including crossinferencing between visual and textual modalities. we then use several baselines and benchmark their performance over the resulting vlqa dataset. as vlqa has multiple choice questions with exactly one correct answer, we use standard accuracy as an evaluation metric. from the results in 3, we can observe that pre-trained vision-language models fail to solve a significant portion of the vlqa items. our proposed modular method hole slightly outperforms them and is more interpretable for analysis. we also report the performance of question-only, image-only and passageonly baselines which we used for quality check. the poor performance of these baselines indicate that the vlqa dataset requires models to jointly understand both image and text modalities and is relatively harder than other vision-language tasks. for human evaluation of the vlqa test-set, the reported accuracy is 84.0%. for 148 wrongly predicted answers, we group them according to 4 reasons for failures, which are listed in 4. the results demonstrate a room for significant improvement in existing vision-language models that are far behind the human performance. this stimulates the underlying reason for incorrect answer provided by test-taker #incorrect/148 (%incorrect) lacked necessary knowledge 27 (18.2%) misunderstood the provided info 47 (31.7%) mistake in deduction/calculation 63 (42.5%) felt that data item is ambiguous 11 (7.4%) ## conclusion in this work, we introduced the visuo-linguistic question answering (vlqa) challenge that we believe has the potential to open new research avenues in areas of joint vision & language. our experiments show that a system equipped with state-of-the-art vision-language pre-training does not perform well on the task that requires joint image-text inference. there is a room for significant improvement in capability of these models to tackle multi-modal contexts. our future work would include further expansion of this dataset and building generic ai models that can learn novel visual concepts from a small set of examples.