Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering (2020)

Abstract: Image text carries essential information to understand the scene and perform reasoning. Text-based visual question answering (text VQA) task focuses on visual questions that require reading text in images. Existing text VQA systems generate an answer by selecting from optical character recognition (OCR) texts or a fixed vocabulary. Positional information of text is underused and there is a lack of evidence for the generated answer. As such, this paper proposes a localization-aware answer prediction network (LaAP-Net) to address this challenge. Our LaAP-Net not only generates the answer to the question but also predicts a bounding box as evidence of the generated answer. Moreover, a context-enriched OCR representation (COR) for multimodal fusion is proposed to facilitate the localization task. Our proposed LaAP-Net outperforms existing approaches on three benchmark datasets for the text VQA task by a noticeable margin.

## introduction visual question answering (vqa) has attracted much interest from the communities and witnessed tremendous progress. however, lacking the ability to generate answers based on texts in the image limits its applications. recently, many new datasets @xcite and new methods @xcite are proposed to tackle this challenge and refer it as text vqa. the earliest method for text vqa is lorra @xcite , which provides an optical character recognition (ocr) module for the vqa input and proposes a dynamic copy mechanism to select the answer from both fixed vocabulary and ocr words. the following work m4c @xcite inspired by lorra, uses rich representations of ocr as input and utilizes dynamic pointer network to deal with out-of-vocabulary answers, leading to state-of-the-art performance. however, m4c simply concatenates all modalities as transformer input and does not consider the high-level interaction among modalities of text vqa. moreover, it is unable to provide evidence for the answer since the text is not localized in the image. another recent work @xcite proposes a new dataset for evidence-based text vqa, which suggests intersection over union (iou) based evaluation metric to measure the evidence. our work follows the spirit of evidence-based text vqa. more specifically, we generate the answer text bounding box during the answer prediction process as supplementary evidence for our answer. we propose a localization-aware answer prediction module (laap) that integrates the predicted bounding box with our semantic representation for the final answer. besides, we propose a multimodal fusion module with context-enriched ocr representation, which uses a novel position-guided attention to integrate context object features into ocr representation. the contributions of this paper are summarized as follows: 1) we propose a laap module, which predicts the ocr position and integrates it with the generated answer embedding for final answer prediction. ## experiments we evaluate our laap-net on the three challenging benchmark datasets: textvqa @xcite , @xcite and ocr-vqa @xcite . we show that the proposed laap-vqa network outperforms state-of-the-art works on these datasets. we further perform the ablation study to investigate the proposed context-enriched ocr representation (cor) and the localization-aware answer prediction (laap) on textvqa dataset. ## conclusion this paper proposes a localization-aware answer prediction network (laap-net) for text vqa. our laap-net not only generates the answer to the question, but also provides a bounding box as an evidence of the answer generated. moreover, a context-enriched ocr (cor) representation is proposed to integrate object related features. the proposed laap-net outperforms existing approaches on three benchmark datasets for the text vqa task by a noticeable margin with new state-of-the-art performance: textvqa 41.41% , st-vqa 0.485 (anls) and ocr-vqa 64.1%.