What’s Different between Visual Question Answering for Machine “Understanding” Versus for Accessibility? (2022)

Abstract: In visual question answering (VQA), a machine must answer a question given an associated image. Recently, accessibility researchers have explored whether VQA can be deployed in a real-world setting where users with visual impairments learn about their environment by capturing their visual surroundings and asking questions. However, most of the existing benchmarking datasets for VQA focus on machine “understanding” and it remains unclear how progress on those datasets corresponds to improvements in this real-world use case. We aim to answer this question by evaluating discrepancies between machine “understanding” datasets (VQA-v2) and accessibility datasets (VizWiz) by evaluating a variety of VQA models. Based on our findings, we discuss opportunities and challenges in VQA for accessibility and suggest directions for future work.

## introduction much research has focused on evaluating and pushing the boundary of machine "understanding" -can machines achieve high scores on tasks thought to require human-like comprehension, including image tagging and captioning (e.g., @xcite , and various forms of reasoning (e.g., @xcite . in recent years, with the advancement of deep learning, we saw great improvements in machines' capabilities in accomplishing these tasks, raising the possibility for deployment. however, adapting machine systems in real-life is non-trivial as real-life situations and users can be significantly different from synthetic and crowdsourced dataset examples @xcite . in this paper we use the visual question answering (vqa) task as an example to call more attention to shifting from development on machine "understanding" to building machines that can make positive impacts to the society and people. ## related work to the best of our knowledge, this is the first work that attempts to quantify and understand the gap in performance vqa models have between the vqa-v2 dataset collected by sighted people and the vizwiz dataset that contains images and questions from people with visual impairments and an-swers from sighted people. @xcite conduct a thorough study on the types of questions people with visual impairments would like answered, and provide a taxonomy for the types of questions asked and the features of such questions. this work was a significant step in understanding the need in people with visual impairments for vqa systems. in combination with our own work, this gives a more complete picture of what kinds of questions not only contribute to better model performance, but actually help individuals with visual impairments. additionally, @xcite seek to understand the task of answering questions about images from people with visual impairments (i.e., vizwiz) and those from sighted people (i.e., vqa-v2). the authors identified the common vision skills needed for both scenarios and quantified the difficulty of these skills for both humans and computers on both datasets. @xcite , who published a very first visual question answering (vqa) dataset, "vizwiz" containing images and questions from people with visual impairments, pointed out the artificial setting of other vqa datasets that include questions that are artificially created by sighted people. the vizwiz challenge is based on real-world data and directs researchers working on vqa problems toward real-world vqa problems. this dataset was built on data collected with a crowdsourcing app, where users with visual impairments share an image and a question with a sighted crowdworker who answers the question for them @xcite . other existing datasets, such as vqa @xcite , daquar @xcite , and ok-vqa @xcite , are different in that their questions were not provided by those who took images. instead, the images were first extracted from web searches, and then questions were later provided by sighted crowdworkers who viewed and imagined questions to ask about those images. here, we see that people with visual impairments can benefit the most from vqa technology but most of the existing vqa datasets do not involve people with visual impairments. some prior work has investigated vqa datasets further, focusing on assessing diversity in answers to visual questions. for instance, @xcite looked at answers to visual questions created by blind people and sighted people and worked on anticipating the distribution of such answers. predicting the distribution of answers asked, they helped crowdworkers create as many unique answers as possible for answer diversity. @xcite tackle the same issue by looking at images of vqa. they proposed a taxonomy of nine reasons that cause differences in answers and developed a model predicting potential reasons that can lead to differences in answers. however, little work explores discrepancies between questions from actual users of vqa applications (i.e., users with visual impairments) and contributors who helped develop data for vqa applications. our work aims to understand this gap by assessing the discrepancies between the dataset containing artificially created data and the dataset containing real-world application data present across different vqa models. more specifically, we assess the performance of vqa models that were proposed in different times and delve into the old model and the state-of-the-art model with individual datapoints to identify patterns where the models perform poorly for the accessibility dataset. ## experiment setup to evaluate how existing vqa models' performance on machine understanding dataset align with performances on the accessibility dataset, we select two vqa datasets and seven vqa models. one of the datasets, vqa-v2, was proposed for machine understanding, whereas the other dataset, vizwiz, was collected to improve accessibility for visually-impaired people. the seven vqa models, selected from the vqa-v2 leaderboard foot_0 , include mfb @xcite , mfh @xcite , ban @xcite , butd @xcite , mcan @xcite , pythia @xcite , and albef @xcite . we assess all seven models on both of the datasets to investigate and understand the model progress across the machine understanding and accessibility datasets foot_1 . ## findings and discussion our objective in this section is to investigate challenges of the vqa task on two different datasets. we assess the performance progress of vqa models and delve into errors. then, we discuss research directions that future work could take. ## limitations this work aims to understand the degree to which progress on machine "understanding" vqa has, and has the ability to, improve performance on the task of accessibility vqa. our findings should be interpreted with several limitations in mind. first, while we analyzed many models across several years of vqa research, our analysis is limited to two datasets. moreover, as discussed in §4.2.2, the "ground truth" in these datasets, especially when combined with the standard evaluation metric, is not always reliable. second, our analysis is limited to english, and may not generalize directly to other languages. finally, blind and low-vision users are not a monolithic group, and the photos taken and questions asked in the vizwiz dataset are representative only of those who used the mobile app, likely a small, unrepresentative subset of the population. ## conclusion and future directions in this paper, we have shown that, overall, performance improvements on machine "understanding" vqa have translated into performance improvements on the real-world task of accessibility vqa. however, we have also shown evidence that there may be a significant overfitting effect, where significant model improvements on machine "understanding" vqa translate only into modest improvements in accessibility vqa. this suggests that if the research community continues to only hill-climb on challenge datasets like vqa-v2, we run the risk of ceasing to make any process on a pressing humancentered application of this technology, and, in the worst case, could degrade performance. we have also shown that along with the overall model improvement, the accessibility vqa system have improved on almost all of the challenge classes though some challenges remain difficult. in general, we observe the models struggle most on questions that require text recognition skill as well as ambiguous questions. future work thus may wish to pay more attention on these questions in both data collection and model design. finally, we have seen that we are likely reaching the limit of the usefulness of the standard vqa ac-curacy metric, and that more research is needed to develop automated evaluation protocols that are robust and accurately capture performance improvements. on top of this, vqa systems are reaching impressive levels of performance, suggesting that human evaluation of their performance in ecologically valid settings is becoming increasingly possible. as ecological validity would require conducting such an evaluation with blind or low-vision users, research is needed to ensure that such evaluation paradigms are conducted ethically and minimize potential harms to system users.