Multimodal Learning for Accurate Visual Question Answering: An Attention-Based Approach (2023)

Abstract: This paper proposes an open-ended task for Visual Question Answering (VQA) that leverages the InceptionV3 Object Detection model and an attention-based Long Short-Term Memory (LSTM) network for question answering. Our proposed model provides accurate natural language answers to questions about an image, including those that require understanding contextual information and background details. Our findings demonstrate that the proposed approach can achieve high accuracy, even with complex and varied visual information. The proposed method can contribute to developing more advanced vision systems that can process and interpret visual information like humans.

## introduction as computer vision research moves beyond "bucketed" identification and toward resolving multimodal problems, language and visual problems like picture captioning and visual question answering (vqa) have become prominent @xcite . issues in the nexus of vision and language are complex due to the complicated compositional structure of language @xcite . however, recent research has shown that language can also provide a strong prior that can lead to good performance on the surface even when the underlying models do not fully comprehend the visual information. our approach to solving the vqa problem involves the development of three distinct models, each with its strengths and limitations: the first model is a simple baseline model that utilizes a pre-trained convolutional neural network (cnn) and recurrent neural network (rnn) architectures to extract visual and textual features from the input image and question, respectively. these features are then concatenated and fed into a simple feed-forward neural network that outputs the final answer. the second is an attention-based model that builds upon the baseline model by incorporating attention mechanisms to selectively focus on relevant parts of the image and question during the feature extraction process. this allows the model to attend to different regions of the image and words in the question, depending on their relevance to the answer. the third model is a more complex, multi-modal transformer-based model that uses a pre-trained transformer architecture to extract visual and textual features from the input image and question. the transformer model incorporates selfattention mechanisms that allow it to learn the relationships between different input parts and selectively attend to the most relevant information. this model also incorporates a visual-linguistic transformer (vilt) module that learns joint representations of both the image and question, allowing for a more seamless integration of visual and textual information. the experimental results show that our models employ different approaches to feature extraction and utilize various neural network architectures to tackle the vqa problem. in another work, @xcite proposed a method to enable contrastive language-image pretraining (clip), a computer vision model, to generate multimodal outputs from a single prompt using distillation techniques that transfer knowledge from a separate multimodal generator model. their proposed method achieves state-of-the-art performance on various multimodal tasks, including image captioning, text-to-image synthesis, and image synthesis from textual prompts. however, our proposed method differs from clip in several ways; the attentional long short term memory (lstm) selectively attends to specific parts of the input sequence, while inception v3 effectively extracts visual features from the input image. combining these models leverages both strengths and provides better representations for multimodal understanding. additionally, the multimodal system is trained on smaller and more targeted datasets, making it more effective in scenarios where the training data is limited or biased. @xcite introduced a framework called "kosmos-1" for vqa task that aligns perception with language models. their approach involves a two-stage training process where a pretrained image encoder is fine-tuned on a small set of vqa tasks before being integrated into a multimodal transformer architecture. additionally, the authors showed that their approach improved the interpretability of vqa models, allowing for a better understanding of model decision-making processes. our proposed method introduces three different architectures. the approach allows for a more direct and intuitive way to associate image information with the textual inputs and exploit the interactions between visual and textual inputs in a more finegrained manner. kosmos-1 uses a single-stream architecture that processes textual and visual information in separate streams, leading to information loss and incomplete modeling of the interactions between the two modalities. ## dataset description the microsoft common objects in context (mscoco) vqa v2 dataset is a large-scale vqa task dataset @xcite . it is a subset of the mscoco dataset, comprising over 330,000 images and 2.5 million object instances. the mscoco vqa v2 dataset contains 265,016 images, and each image is accompanied by at least three open-ended questions and ten humangenerated answers for each question. this dataset evaluates various visual reasoning and language understanding capabilities, including object recognition, spatial reasoning, counting, and reasoning about actions and events. the questions in the dataset cover a wide range of topics, from ordinary objects and scenes to more complex and abstract concepts. using the mscoco vqa v2 dataset for vqa tasks enables researchers to develop and evaluate new visual reasoning and language understanding techniques essential in fields such as autonomous vehicles, robotics, and human-computer interaction. ## models and experiments we propose three vqa models, utilizing recurrent neural networks (rnns) and image embeddings to answer questions based on visual content. the models differ in adding the image to the input question tensor. we provide an overview of our approach before describing each step in detail in the following subsections. the first model, appending image as word model, appends the image features to the end of the question features, creating a concatenated vector fed into an lstm layer for prediction. the second model is prepending image as word model, which prepends the image features to the beginning of the question features, creating a concatenated vector fed into an lstm layer for prediction. the third model is the co-attention model, which utilizes a co-attention mechanism, where the image and question features are combined at every time step using attention weights. this model then feeds the integrated features into an lstm layer for prediction. all three models are evaluated on the mscoco vqa v2 dataset and compared to the state-of-theart approaches @xcite . an ablation study is conducted to investigate the impact of different hyperparameters and variations of the models on the vqa task's performance. the experiments show that adding image features to the input question tensor can significantly improve the model's performance and highlight the importance of the rnn's architecture and the number of image features utilized. ## results we evaluate the three models trained on the train splits of both the unbalanced and balanced datasets by testing on the balanced test set as done in @xcite . training on the balanced dataset works well. this may be because the models trained on flat data must learn to extract visual information to answer the question correctly since they can no longer exploit language biases in the training set. whereas models trained on the unbalanced set are blindsided into learning strong language priors, which are then not available at the test step. the results of model-1, model-2, and model-3 are summarized in ## conclusions our proposed framework addresses the limitations of existing vqa models by combining the attentional lstm and inception v3 models to create three different models for vqa. by appending or prepending the image as a word in the question sentence or using a co-attention model, we can better capture the relationship between images and questions, improving vqa performance.