  The problem of knowledge-based visual question answering involves answering
questions that require external knowledge in addition to the content of the
image. Such knowledge typically comes in a variety of forms, including visual,
textual, and commonsense knowledge. The use of more knowledge sources, however,
also increases the chance of retrieving more irrelevant or noisy facts, making
it difficult to comprehend the facts and find the answer. To address this
challenge, we propose Multi-modal Answer Validation using External knowledge
(MAVEx), where the idea is to validate a set of promising answer candidates
based on answer-specific knowledge retrieval. This is in contrast to existing
approaches that search for the answer in a vast collection of often irrelevant
facts. Our approach aims to learn which knowledge source should be trusted for
each answer candidate and how to validate the candidate using that source. We
consider a multi-modal setting, relying on both textual and visual knowledge
resources, including images searched using Google, sentences from Wikipedia
articles, and concepts from ConceptNet. Our experiments with OK-VQA, a
challenging knowledge-based VQA dataset, demonstrate that MAVEx achieves new
state-of-the-art results.
