M ini VQA - A resource to build your tailored VQA competition (2021)

Abstract: MiniVQA is a Jupyter notebook to build a tailored VQA competition for your students. The resource creates all the needed resources to create a classroom competition that engages and inspires your students on the free, self-service Kaggle platform. “InClass competitions make machine learning fun¡‘.

## task overview beyond simply recognizing what objects are present, vision-and-language tasks, such as captioning @xcite or visual question answering @xcite , challenge systems to understand a wide range of detailed semantics of an image, including objects, attributes, spatial relationships, actions and intentions, and how all these concepts are referred to and grounded in natural language. vqa is therefore viewed as a suitable way to evaluate a system reading comprehension. all implemented techniques can be evaluated on the difficult task that is vqa. because answering questions about a visual context is a hard cognitive task, using different nlp approaches can lead to significant performance differences. ## resource overview the source code is split into several sections, each section containing tunable parameters to modulate the dataset to match your needs. for example, you can choose the number of possible answers, the sample size per answer or the balance of labels between splits. minivqa built upon two datasets, the vqa v2 dataset @xcite and the vqa-med dataset @xcite . you can choose to create a competition based on natural images or medical images. minivqa proposes 443k questions on 82.7k images and 4547 unique answers for the former, and 7.4k unique image-question pairs and 332 answers for the latter foot_2 . finally, minivqa provides a second jupyter notebook that trains and evaluates a baseline vqa model on any dataset you create. you are free to share it or not amongst your class. ## resource presentation the following sections present the features of minivqa. we illustrate these features using the vqa v2.0 dataset as a matter of example. the same can be applied to the vqa-med dataset. ## baseline model the provided baseline consists of a dataloader that opens images and resize them to 112 × 112 pixels and that embeds questions to a feature vector of size 768 from a pre-trained distilbert @xcite . the network consists a modified resnet @xcite that takes as input an rbg image of size 112 × 112 and outputs a feature map of size 512 × 4 × 4 that is flattened and then concatenated with the question representation of size 768. finally, a classification layer projects this representation to probabilities over answers. the network is trained until no improvements is recorded on the validation set (early-stopping).