Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQA (2024)

Abstract: Knowledge-based Visual Qustion-answering (K-VQA) often requires the use of background knowledge beyond the image. However, we discover that a single knowledge generation strategy is often insuffcient for all K-VQA questions. To this end, we propose Diversifcation, Evidence Truncation, and Combination for Knowledge-based Elucidation (DietCoke), which utilizes a bundle of complementary question-answering tactics and aggregates their answers using textual rationales. DietCoke comprises of three stages: diversifcation, rationalization, and ensemble. The diversification stage generates three distinctive decision contexts, each leading to its own answer candidate. The rationalization stage generates two rationales, the automatic rationale and the mechanistic rationale, for each answer candidate using decorrelated techniques. Finally, in the ensemble stage, an LLM informed by the rationales selects one answer from the three candidates. Experiments show that DietCoke significantly outperforms state-of-the-art LLM-based baselines by 2.8% on OK-VOA and 4.7% on A-OKVOA and that the strategies in the ensembles are highly complementary.

## introduction knowledge-based visual question-answering (k-vqa) requires background knowledge beyond the image content. for example, to answer the question in fig. @xcite , it is necessary to know the carbohydrate content of different types of food. zero-shot k-vqa provides an effective evaluation of ai models in applying their knowledge to answer novel vision-informed questions. an effective technique for zero-shot vqa is to translate an image to a textual decision context for a text-based model, such as an llm, which the answers the question using the context @xcite . for example, @xcite translate the image to captions; @xcite further generate question-answer pairs to demonstrate the vqa task. to cater to the knowledge-intensive nature of k-vqa, @xcite generate a paragraph describing relevant background knowledge, from which an llm extracts the answer. these approaches rely on a single question-answering strategy using a single type of decision context. interestingly, we observe that one llm strategy is often not sufficient for k-vqa datasets. for some questions, merely the captions are sufficient for finding the right answer. other questions benefit from knowledge extracted from llms, but it is often not clear if a short, one-sentence knowledge statement or a paragraph would beget the right answer from the llm. as an illustration, in fig. 1 we show three questions and three ai-generated decision contexts for each question, including captions, short-form knowledge as a single sentence, and long-form knowledge, which contains multiple sentences. in each question, only one decision context leads to the right answer. in light of this, we propose diversification, evidence truncation, and combination for knowledge-based elucidation (dietcoke) to achieve dynamic ensemble of different questionanswering strategies. dietcoke comprises a diversification phase, a rationalization phase, and an ensemble phase. in each phase, we use an incontext-learning llm without training. in diversification, we generate three decision contexts for each k-vqa question, including image captions, short-form knowledge, and long-form knowledge, with an increasing amount of background knowledge. the llm generates three answer candidates, one from each decision context. in the ensemble phase, the llm selects one correct answer from the three candidates. to encourage informed decisions, we design the rationalization phase, which generates two complementary figure 1 : the three k-vqa questions are best answered using distinctive decision contexts, including image captions, (a), and two forms of generated knowledge statements, (b) and (c). this is due to (1) the difficulty in controlling the generation of captions and knowledge statements and ensuring they contain relevant information, and (2) the inability of llms to identify correct answers from noisy contexts. on two popular k-vqa datasets, okvqa and a-okvqa, dietcoke outperforms state-of-theart baselines that use frozen llms without training by 2.6% to 4.7%. ablation studies reveal that the fusion of three answers improves performance over the best single answer strategy by 1.1% on ok-vqa and 1.6% on a-okvqa. further, adding both the automatic and mechanistic rationales obtains gains of 1.2% and 1.4%, respectively on the two datasets. the success of dietcoke can be understood from multiple perspectives. first, dietcoke creates an effective ensemble by employing decorrelated qa strategies and rationalization strategies. classic theory on ensemble learning @xcite indicates that an ensemble of weak classifiers becomes more powerful as the individual classifiers become less correlated and more diverse. intuitively, different qa contexts lead to diverse answers. also, disparate mechanisms used by the two rationalization strategies utilize reduce correlation. in §4.3, we perform a series of ablation studies that demonstrate the synergy among the three qa strategies and the two rationalization strategies. second, rationales in dietcoke can be understood from a chain-of-thought (cot) perspective. recently, @xcite prove that chain-ofthought prompting expands the circuit complexity of problems solvable by an llm from the class ac 0 with constant circuit depth to the class p/poly with polynomial circuit depth. this is made possible by saving intermediate computational results as cot tokens. analogically, providing the qa contexts (i.e., intermediate results) to the answer-selecting ensemble llm should be beneficial. however, in practice we observe that long decision contexts may contain many irrelevant facts that can mislead the ensemble llm. by summarizing the contexts into rationales, we provide abridged chains-of-thoughts that expand circuit depth while avoiding misleading tokens. our contributions include: • we identify the need for combined use of multiple question-answering strategies in k-vqa, and propose dietcoke, which fuses multiple answers and answer strategies dynamically using frozen in-context learning llms, achieving state-of-the-art performance on zero-shot • we propose that rationales serve an important role in answer ensemble. further, we devise two rationale generation techniques, automatic llm summarization, and mechanistic llm interpretation. these rationales complement each other in enhancing vqa performance. end-to-end lvlms and lvlm prompting for zero-shot k-vqa. end-to-end training is adopted to align the vision and text modalities, so that a pretrained visual encoder can work seamlessly with a pretrained llm, resulting in lvlms @xcite @xcite @xcite @xcite @xcite . while these lvlms achieve high performance on zero-shot k-vqa, they require training on an enormous quantity of image-text pairs and are hence compute-intensive. to enable the lvlms to better understand questions and perform reasoning, one concurrent method explore question rephrasing in the prompt @xcite . however, it is difficult for this method to change to a different llm, which provides crucial knowledge for k-vqa, without expensive retraining of the lvlms. in contrast, our method translates images into textual descriptions and directly utilizes frozen llms to answer questions, enabling us to easily change the knowledge source at any time. ## method the goal of our method is to jointly utilize multiple decision contexts to answer each question and achieve a dynamic ensemble of these different strategies. to accomplish this, we propose dietcoke, which consists of three phases: diversification, rationalization and ensemble. this section provides detailed descriptions of them. ## conclusion we propose dietcoke, which ensembles several qa strategies for knowledge-based vqa. diet-coke first generates three types of decision contexts: image captions, short-form knowledge and long-form knowledge, and answers the question from each decision context separately. after that, dietcoke generates two types of rationales for each answer. an ensemble llm selects the best answer from the rationales. dietcoke achieves state-of-the-art results on ok-vqa among comparable methods and showcases llm-informed, rationale-based ensemble as an effective vqa method. dietcoke allows potential trade-off between compute and answer quality and may contribute to research on scaling llms with inferencetime compute, a research direction pioneered by @xcite .