Grounded Graph Decoding improves Compositional Generalization in Question Answering (2021)

Abstract: Question answering models struggle to generalize to novel compositions of training patterns. Current end-to-end models learn a flat input embedding which can lose input syntax context. Prior approaches improve generalization by learning permutation invariant models, but these methods do not scale to more complex train-test splits. We propose Grounded Graph Decoding, a method to improve compositional generalization of language representations by grounding structured predictions with an attention mechanism. Grounding enables the model to retain syntax information from the input that significantly improves generalization to complex inputs. By predicting a structured graph containing conjunctions of query clauses, we learn a group invariant representation without making assumptions on the target domain. Our model performs competitively on the Compositional Freebase Questions (CFQ) dataset, a challenging benchmark for compositional generalization in question answering. Especially, our model effectively solves the MCD1 split with 98% accuracy. All source is available at https://github.com/gaiyu0/cfq .

## introduction can neural networks "make infinite use of finite means" with language @xcite ? the ability of humans to reason compositionally enables us to form novel complex compound sentences by combining constituent concepts. toki pona @xcite is an engineered language with only 120 words but can express a wide variety of concepts through composition. compositionality specifically refers to the phenomenon that the meaning of an expression is given by combining the meanings of its parts @xcite figure 1 : an illustration of grounded graph decoding with the question "who directed and produced goldfinger and inception?". the predicates "direct(ed)" and "produce(d)", and the entities "goldfinger" and "inception" are grouped together by the sequence encoder. black lines represent linear transforms between latent spaces, while the red line represents the attention that grounds the prediction. 1970). for example, after understanding the questions "who directed inception?" and "did christopher nolan produce goldfinger?", one can understand "who produced inception?". however, it's unclear whether neural networks truly reason compositionally; for example, @xcite argue that compositional reasoning is not necessary to answer complex composite questions. to benchmark the performance of neural models under compositional generalization, lake and @xcite propose the scan dataset containing sequences of instructions. scan uses length of input as a proxy for compositional complexity. however, scan's length-based splits are now considered solved @xcite . as a more realistic benchmark, @xcite introduced the compositional freebase questions (cfq) dataset. cfq is a challenging knowledge graph question answering task with ad-versarial splits (mcd1, mcd2 and mcd3) that maximize compositional divergence between train and test sets. while a simple lstm model achieves near-perfect accuracy on an i.i.d. split, large models such as t5-11b @xcite performs catastrophically poorly on the mcd splits. we hypothesize current models perform poorly on the cfq task as they fail to take semantic and syntactic structures into account. we propose grounded graph decoding to incorporate these structures with graph decoding and grounding. the encoder often fails to match common syntactic patterns from the input text question. we modify the decoder with attention over the input sequence to enable referencing specific input syntactic structures. moreover, we find the semantics of knowledge graph queries (sparql) is a conjunctive query graph. sequence decoders fail to represent this structure well. we integrate semantic structure by decoding output predictions as a graph. our work makes the following contributions: ## preliminaries we formally present an overview of compositional generalization in deep learning to motivate grounded graph decoding. ## methodology: grounded graph decoding in the following sections, we first describe how to decode conjunctive query graphs from natural language questions, then how to incorporate syntactic compositions in graph decoding, and finally introduce a grounding mechanism by explicitly conditioning graph decoding on syntactic compositions. ## experiments we evaluate grounded graph decoding to understand how grounding as well as graph decoding improve semantic and syntactic understanding. ## related work various approaches to the compositional generalization challenge posted by the cfq dataset has been explored in prior or contemporary works, including guo et al. (2020), @xcite , and @xcite . these approaches are discussed in more detail in section 4. another promising approach that has received relatively less attention so far is grammar induction, which can potentially derive grammar rules directly from question-query pairs. grammar induction methods, such as @xcite , typically assumes a limited set of grammar rules to bootstrap the model, and then search some grammar spaces to find grammars that can lead to successful parsing of observed questions. the idea of grammar induction has inspired various work that to different extent solved the scan dataset, such as @xcite or @xcite . the advantage of grammar induction methods is that they can potentially identify the complete set of transformation rules and thus attain perfect compositional generalization. however, grammar induction methods are generally search-based, which limits their scalability to long sentences due to the size of search spaces. additionally, there has been considerable research in the semantic parsing literature to design neural network architectures that incorporate different query structures, including tree @xcite , graph @xcite @xcite . however, these architectures only incorporating query structures without incor-porating syntactic structures in questions. our ablation study ( similar to our work, @xcite also proposes to improve the compositional generalization of seq2seq models using attention. however, their work only studies token-level attention without consideration of syntactic or semantic structures. both russin et al. (2019) and gordon et al. (2020) finally, in the domain of semantic parsing, prior to @xcite proposed to split datasets such that training and test sets contain no common sql patterns. although this approach increases task difficulty, different sql query patterns may still share similar substructures, which enables neural networks to solve the tasks relatively easily using the "mix-andmatch" strategy (lake and baroni, 2018). ## conclusion in this paper we propose grounded graph decoding to make compositionally generalizable predictions of conjunctive query from natural language questions. our model consists of a graph decoder that captures permutation invariance in conjunctive queries, a sequence encoder that is aware of syntactic composition, and an attention mechanism that enables strong association between syntactic and semantic compositions. the proposed method solves the mcd1 split of the challenging cfq dataset, and improves the state-of-the-art of the other two splits. notably, grounded graph decoding significantly outperforms competitive baselines including large pre-trained models (such as t5) as well as domain-specific models. careful ablations of our method demonstrate the importance of both graph decoding and grounding.