Multiple-Question Multiple-Answer Text- VQA (2024)

Abstract: We present Multiple-Question Multiple-Answer (MQMA), a novel approach to do text-VQA in encoder-decoder transformer models. To the best of our knowledge, almost all previous approaches for text-VQA process a single question and its associated content to predict a single answer. However, in industry applications, users may come up with multiple questions about a single image. In order to answer multiple questions from the same image, each question and content are fed into the model multiple times. In contrast, our proposed MQMA approach takes multiple questions and content as input at the encoder and predicts multiple answers at the decoder in an auto-regressive manner at the same time. We make several novel architectural modifications to standard encoder-decoder transformers to support MQMA. We also propose a novel MQMA denoising pre-training task which is designed to teach the model to align and delineate multiple questions and content with associated answers. MQMA pre-trained model achieves state-of-the-art results on multiple text-VQA datasets, each with strong baselines. Specifically, on OCR-VQA (+2.5%), TextVQA (+1.4%), ST-VQA (+0.6%), DocVQA (+1.1%) absolute improvements over the previous state-of-the-art approaches.

## introduction the task of text-based visual question answering (text-vqa) requires answering questions related to a given image by understanding the text and visual contents in the image. unlike generic vqa @xcite , where the task is to answer questions mainly using visual information, the text-vqa task involves multiple modalities (i.e., visual, language, and layout) to answer questions (biten et al., 2022; hu et al., 2020; appalaraju et al., 2021; huang et al., 2022; kant et al., 2020; mathew et al., 2021, * equal contribution. 2020; @xcite @xcite @xcite @xcite . the task needs a model to not only consume multiple modalities (text and image) but also to reason within and across modalities to answer a question (see figure 1 ). in recent years, the text-vqa task has attracted a lot of attention @xcite @xcite @xcite @xcite . almost all text-vqa approaches known to us, consume a single question and associated content to predict a single answer. we call these approaches single-question single-answer (sqsa) text-vqa, see figure 2 (a) . typical sqsa approaches @xcite @xcite @xcite @xcite first extract text in a given image using an ocr engine. then the entire content -image, ocr text and in some cases bounding box information @xcite @xcite , along with the text of a single question are fed to a multi-modal transformer model which then predicts an answer. industry text-vqa applications often involve multiple questions. for example, a user may ask multiple questions about a single image, or a group of users may ask different questions about the same image (e.g., shipped date, order no., address, etc. in figure 1 (a) ). existing text-vqa models are not well-equipped for answering multiple questions. these models typically process a single question and its associated content to predict a single answer. in order to answer multiple questions from the same image, each question and content are fed into the model multiple times. this is inefficient and can lead to sub-optimal performance (sec. 5). mqma can address the limitations of existing text-vqa models. mqma takes multiple quesanswering questions for text-vqa requires multimodal information, including visual, language, and layout information. zoom in to see better. ## related work text-vqa has attracted more and more attention recently @xcite @xcite @xcite @xcite @xcite . focusing on different types of images with texts, several works introduce various text-vqa datasets, including ocr-vqa @xcite for book and movie covers, textvqa @xcite and @xcite for scene-text images, docvqa @xcite for document images, etc. unlike generic vqa @xcite which answers questions by reasoning visual contents, text-vqa reasons from both text and visual contents in images to answer questions, which introduces more challenges to the text-vqa task compared with the generic vqa. the most common text-vqa pipeline first extracts texts and bounding boxes using ocr, and then feed multi-modal inputs (i.e., texts, bounding boxes, and image) into multi-modal models ## mqma model architecture in this section, we discuss in detail the mqma model architecture. our choice of architecture for mqma is an encoder-decoder transformer model (see figure 3 ). this architecture is chosen due to its popularity, versatility, and state-of-the-art text-vqa accuracy @xcite @xcite . in addition, using a vocabulary-free generative decoder lends itself as a generic vqa architecture over approaches which are designed for closed-vocabulary vqa @xcite . the use of decoder elicits additional challenges for mqma as it is not obvious how the model can auto-regressively generate multiple answers for arbitrary number (> 1) of input questions for a content. our mqma model is built on top of the state-ofthe-art multi-modal encoder-decoder model doc-formerv2 @xcite which is termed as the single-question single-answer (sqsa) baseline in the experiment section 5. the input questions and content -image, ocr text, layout information are vectorized and fed into the transformer encoder. so the model can process multiple modalities at the same time. see section 3.1 for more details. the transformer encoder processes these inputs with a series of self-attention layers, feed-forward layers, and layer normalization layers to get transformer encoder representations. this representation is then fed into the transformer decoder, consisting of a series of self-attention layers, cross-attention layers, feed-forward layers, and layer normalization layers, decoding answers as predictions in an auto-regressive manner. in order to support mqma functionality, the model needs to be made aware of that the input has multiple questions and that at the decoder, the model needs to appropriately align each question with the predicted answer. to facilitate this behavior, we introduce two key changes to the above described sqsa multi-modal encoder-decoder transformer architecture: a) question distinguishing multi-modal encoder -in order to distinguish different questions and content in the inputs, we introduce a question index embedding layer which uses different embeddings for different questions and content, where the embedding of index i is used for the i-th question and the embedding of index 0 is used for content (see section 3.1). b) learnable prompt at the decoder -traditionally, a decoder is trained to auto-regressively predict a token beginning with a fixed [start] token @xcite . instead, in our approach, we introduce n learnable prompts corresponding to the n questions we fed into the model at the encoder. the decoder autoregressively predicts n answers beginning with these learnt prompts instead of the [start] token. each question uses a separate prompt to decode the corresponding answer (see section 3.2). ## mqma unsupervised pre-training it is well established that pre-training followed by task specific fine-tuning almost always leads to superior performance when compared with models trained with just supervised fine-tuning @xcite @xcite @xcite @xcite . ability to train on vast amounts of unsupervised data has a key role to play in the success of this training strategy. in language domain, a number of pre-training strategies inspired by cloze task @xcite have been designed, e.g., masked language modeling (kenton and toutanova, 2019). more recently, a denoising language modeling pre-training task was proposed in the t5 model @xcite and this pre-training task has been successfully used in previous text-vqa models like docformerv2 @xcite and @xcite . the denoising language modeling task is unsupervised. the task masks spans of original text and the objective is to reconstruct the masked text during training (see "standard denoising" in we experimentally show that this novel pretraining task is better aligned with the downstream text-vqa task and benefits the model for text-vqa even if the mqma setting is not desired. we also tried "before" style question formulation and found it to be not as beneficial when compared with the "after" style. so in experiments we stick to the "after" style questions only. there could be other ways to formulate the questions to get more benefits.