Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems (2022)

Abstract: For vision-and-language reasoning tasks, both fully connectionist, end-to-end methods and hybrid, neuro-symbolic methods have achieved high in-distribution performance. In which out-of-distribution settings does each paradigm excel? We investigate this question on both single-image and multi-image visual question-answering through four types of generalization tests: a novel segment-combine test for multi-image queries, contrast set, compositional generalization, and cross-benchmark transfer.Vision-and-language end-to-end trained systems exhibit sizeable performance drops across all these tests. Neuro-symbolic methods suffer even more on cross-benchmark transfer from GQA to VQA, but they show smaller accuracy drops on the other generalization tests and their performance quickly improves by few-shot training. Overall, our results demonstrate the complementary benefits of these two paradigms, and emphasize the importance of using a diverse suite of generalization tests to fully characterize model robustness to distribution shift.

## introduction widely used multi-modal pretrained models @xcite @xcite have exhibited great performance when fine-tuned on downstream vision-and-language tasks like vqa @xcite and gqa @xcite . these models often generalize poorly to out-of-distribution (ood) data, suggesting shortcomings in the vle2e pipeline. neurosymbolic methods @xcite try to address this issue by disentangling grounding and reasoning mechanisms in multi-modal systems. ns methods generate grounded visual representations, parse the language into executable programs for reasoning, and execute the programs on the visual representations. previous work @xcite figure 1 : we build segment-combine tests, contrast sets and compositional generalization splits for multi-image question answering in the covr dataset. the above question requires counting both within and across images. segment-combine test: the multi-image query enables considering each image in isolation, pairing them with random unrelated images and feeding to the model, doing an or operation of per-image answers. contrast set: language perturbation by replacing phrases in query with synonyms or antonyms. compositional generalization: the evaluated query is a compositional variant of questions train a and train b, involving reasoning on both counting and relations. @xcite has shown the effectiveness of neuro-symbolic methods for ood compositional generalization on single-image vl reasoning tasks. however, we still lack a compre-hensive understanding of the generalization differences between these two paradigms under various setups. given recent work suggesting that ood accuracy often strongly correlates with in-distribution accuracy @xcite , we might expect vle2e and ns systems to often have similar generalization abilities. but do they? in this work, we conduct the first comprehensive comparison of generalization behavior between vle2e and ns systems for vl reasoning tasks. our study spans single-image and multi-image settings with natural images and includes four distinct types of generalization tests, three of which are shown in figure 1 . we introduce a novel segmentcombine test for multi-image settings that requires models to make consistent predictions when some input images are replaced with irrelevant ones. we evaluate on contrast sets @xcite , including new contrast sets we construct for covr that test understanding of quantifiers. we also measure compositional generalization as defined by compositional splits from covr @xcite and cross-benchmark transfer between vqa and gqa. we also develop improved ns systems for gqa by handling mismatches between program and scene graph object descriptors, and for covr by refining the original logical language. overall, we find that vle2e and ns systems exhibit distinct and complementary generalization patterns. the ns systems are more robust than the vle2e systems in the first three testing situations. the vle2e systems exhibit overstability to meaning-altering perturbations, suggesting they overfit to spurious correlations in the training data and do not learn precise reasoning skills. we further find that the semantic parsing module of ns systems can quickly improve on generalization tests given a few training examples, whereas vl models do not adapt as quickly. on the other hand, while vle2e systems lose more than 10% in accuracy on transfer between vqa and gqa, the ns methods perform even worse. taken together, our findings underscore the need for a diverse suite of generalization tests to fully compare different modeling paradigms. the different behavior of these two systems could guide the community to design more robust vl reasoning systems. we release our code for generating test data, and we encourage future vl models to be evaluated on these tests. foot_0 vl ood generalization. many efforts have been made to evaluate the generalization ability of vle2e systems and task-specific methods on compositionality @xcite , language perturbations (ribeiro et al., 2019) and visual perturbations @xcite . @xcite showed vle2e systems exhibit better robustness than task-specific methods. we are the first to comprehensively compare the generalization differences between vle2e and ns systems across different ood tests. ## models next, we formally define the vl reasoning tasks and vle2e and ns methods we study. we also discuss a new ns system for covr and associated changes to the original covr logical forms. ## evaluation methods we evaluate vle2e and ns systems on four generalization tests. we create a new multi-image perturbation test called the segment-combine test, and create new contrast sets for covr by perturbing quantifiers. we also test models on compositional generalization and cross-benchmark transfer. segment-combine test. we introduce the segment-combine test to test model generalization on multi-image perturbations. for a multi-image query (q, i) where @xmath0, ..., i |i| ), we first perform a segmentation phase. we make |i| queries, where the k-th query uses the original question q and an image set formed by the union of the original image i k plus |i| -1 random images unrelated to q. we feed these to the model to get |i| predictions. next, in the combination phase, we apply an aggregation function (e.g., sum or or) based on the question type to fuse these predictions (figure 1 ). a robust model should return the same answer on the segment-combine test and the original example. we run the segment-combine test on covr, sampling random images from all images in the covr validation set. to confirm that we only sample images unrelated to the original image set (i.e., will not change the answer after fusion), we execute ground truth programs on ground truth scene graphs for each query in the segmentation phase, and find that the accuracy is 100%. we focus on two templates in covr for which there is an appropriate fusion function. for the template countgroupby (e.g., "how many images have 2 bottles?"), the fusion function is sum. that is to say, the answer on the original input should be equal to the sum of the |i| answers from the segmentation phase. for the template verify-countgroupby, the fusion function is logical or, as shown in figure 1 . contrast sets. for vl reasoning, we define a contrast set @xcite of an example (q, i, y) ∈ d test to be a set of similar examples (q ′ , i, y ′ ), where q ′ is similar to q and y ′ may or may not be the same as y, depending on q ′ . q ′ could be constructed by replacing specific words or phrases in q with synonyms or antonyms, or by substituting objects with other objects. given n contrast set examples (q ′ 1 , i, y ′ 1 ), . . . , (q ′ n , i, y ′ n ), we primarily evaluate models on the average accuracy on these n examples. we also measure the average local coherency as 1 n n @xmath1, which measures how much the model ignores perturbations. we use the single-image contrast sets created by @xcite for gqa. their contrast sets involve object substitutions from scene graphs and mainly test the robustness of vle2e systems for grounding objects. for multi-image covr, we design new contrast sets that target perturbations involving cross-image reasoning for multi-image queries. we replace quantifiers in the testing data with phrases of the equivalent and opposite meanings and change the labels accordingly. we focus on examples generated by 4 templates, where quantifiers (e.g., at least, all) play the role of introducing cross-image reasoning: one counting question template count-groupby, and three binary question templates, verifycountgroupby, verifycount (e.g., "at least 2 bottles on the table?") and quanti-fier (e.g., "no bottles are on the table?"). we test meaning-preserving perturbations such as replacing at least with no less than on counting and binary questions. we also test meaning-altering perturbations such as replacing no with some on binary questions and flipping the answer. we do not apply meaning-altering perturbations to counting questions as it is non-trivial to determine what the new answer y ′ should be. ## experiments we present our experimental setup and results on four types of generalization tests below. the results indicate the complementary robustness of vle2e and ns systems in ood settings. ## discussion and conclusion in conclusion, vle2e training systems do not learn precise reasoning, which inhibits their generalization ability under small perturbations to either language or vision. though the in-domain results of ns systems are usually slightly worse than vle2e systems, the ns methods are more robust on most of the generalization tests we de-velop here. even when the performance of ns methods drops on some ood data, they can still quickly recover by few-shot training. nonetheless, vle2e systems still achieve better performance on cross-benchmark transfer, while ns methods struggle when test questions require novel program constructs or scene graph object types. our work highlights the importance of evaluating on a diverse set of metrics besides indistribution accuracy, in line with recent work on improving leaderboards @xcite . our analysis suggests that we should not expect in-domain and out-of-domain accuracies to be strongly correlated when evaluating very different types of models, such as vle2e and ns models, in contrast with @xcite . finally, we hope our observation that endto-end and neuro-symbolic systems have complementary generalization advantages will inspire the community to design more robust vl reasoning systems that share the benefits of both approaches.