SLOG : A Structural Generalization Benchmark for Semantic Parsing (2023)

Abstract: The goal of compositional generalization benchmarks is to evaluate how well models generalize to new complex linguistic expressions. Existing benchmarks often focus on lexical generalization, the interpretation of novel lexical items in syntactic structures familiar from training; structural generalization tasks, where a model needs to interpret syntactic structures that are themselves unfamiliar from training, are often underrepresented, resulting in overly optimistic perceptions of how well models can generalize. We introduce SLOG, a semantic parsing dataset that extends COGS (Kim and Linzen, 2020) with 17 structural generalization cases. In our experiments, the generalization accuracy of Transformer models, including pretrained ones, only reaches 40.6%, while a structure-aware parser only achieves 70.8%. These results are far from the near-perfect accuracy existing models achieve on COGS, demonstrating the role of SLOG in foregrounding the large discrepancy between models’ lexical and structural generalization capacities.

## introduction compositional generalization benchmarks that test the ability to understand novel utterances based on composition of known parts @xcite @xcite have emerged as a useful tool for model evaluation in semantic parsing. cogs @xcite in particular has become a widely-used benchmark, as it is designed to expose a generalization gap between training and testing data that many recent semantic parsers still struggle with. cogs distinguishes two distinct types of generalization challenges: lexical generalization tests a model's ability to interpret novel combinations of known lexical items and known linguistic structures (figure 1a ), whereas structural generalization * * this work was conducted during bingzhi li's visit to nyu. the middle authors are listed in alphabetical order. 1 . to facilitate a more comprehensive evaluation of structural generalization, we introduce slog, a structural long-distance dependencies generalization benchmark. slog extends cogs to include 17 cases of structural generalization in total (14 new cases and 3 existing cases from cogs) ( §2). the novel generalizations we introduce target two key structural features of human language ( §3): recursion and filler-gap dependencies. training generalization cogs emma saw the dog. ; * dog(x3); see.agent(x1,emma) ∧ see.theme(x1, x3) the cat ran. ; * cat(x1); run.agent(x2, x1) the dog ran. ; * dog(x1); run.agent(x2, x1) slog emma saw the dog that max held. ; * dog(x3); see.agent(x1,emma) ∧ see.theme(x1, x3) ∧ dog.nmod(x3, x6) ∧ hold.agent(x6,max) ∧ hold.theme(x6, x3) the cat ran. ; * cat(x1); run.agent(x2, x1) the dog that max saw ran. ; * dog(x1); dog.nmod(x1, x4) ∧ see.agent(x4,max) ∧ see.theme(x4, x1) ∧ run.agent(x5, x1) from scratch @xcite , two pretrained transformers (t5-base; @xcite , and a structureaware foot_0 model (am-parser; @xcite . in comparison to their overall performance on cogs, all models exhibit considerably lower performance on slog ( §5). an error analysis reveals that the structure-aware am-parser generalizes well on the existing structural generalization cases in cogs but struggles with the gap constructions introduced in slog due to inherent structural limitations, which we discuss in §5.3. transformers tend to erroneously repeat frequent meaning representation subsequences observed during training. even with pretraining, they struggle with unseen long-distance dependencies, which we attribute to their bias towards shorter predicate-argument dependencies. overall, the discrepancy in performance between slog and cogs demonstrates the utility of slog in exposing the overall limitations of current semantic parsing models shown to achieve high performance on existing generalization benchmarks, as well as highlighting the different weaknesses of these models. the slog benchmark slog follows the semantic parsing format used in cogs, where the task is to translate english expressions into logic-based meaning representations. as in cogs, there is a systematic gap between the training set and the generalization set: there are constructions in the generalization set that are not included in the training set, but pieces of constructions included in the training set can be recombined to arrive at their correct meanings. for example, as illustrated in slog contains 17 structural generalization cases grouped into four categories. these generalization cases are primarily motivated by frequency asymmetries in natural language, where simpler structures are more common than complex ones; in other words, slog assesses whether nlp models can extrapolate from frequent patterns to rare ones. we describe the four categories below; see ## experimental setup models we evaluate the performance of seq2seq, autoregressive, and structure-aware models on slog. the seq2seq models we evaluate are a transformer we train on slog from scratch (vanilla transformer henceforth; @xcite , and a finetuned pretrained transformer (t5; @xcite ) that has demonstrated strong performance on multiple compositional generalization tasks @xcite . the autoregressive transformer model we evaluate is llama @xcite . finally, the structure-aware model we evaluate is the am-parser @xcite , which achieves near-perfect accuracy on cogs @xcite . previous work has shown that structure-aware models perform well on compositional generalization tasks, specifically those involving structural generalization @xcite . following @xcite , we first have the am-parser predict an intermediate dependency tree, and then convert it to a graph-based representation of the slog logical form. we use the a* am-parser from lindemann et al. (2020) for our experiments, as it yields the best overall results compared to alternative versions of am-parser, such as the one in @xcite . 9 we run each experiment with five different random seeds. see appendix a for more details. evaluation metric most studies report exact match accuracy on cogs. this metric has two limitations that may lead to an underestimation of a model's generalization capacity. first, because the cogs lf is conjunctive, reorderings of the conjuncts are semantically equivalent; yet, under exact match accuracy, only a single order is considered correct. second, the cogs lf uses skolem constants with a naming scheme tied to the linear indices of phrasal heads in the input. while a commitment to a systematic naming scheme is necessary for consistent evaluation, different naming schemes up to the renaming of the constants in the gold lf yield equivalent lfs (e.g., (15a) vs. (15b)). such lfs would be considered incorrect under exact match. to incorporate semantic equivalence up to conjunct reordering and constant renaming, at evaluation time, we alphabetically sort the conjuncts of the gold lfs, and subsequently index variables based on their appearance order in the sorted lfs. the same modifications are applied to the model outputs. this process results in the reformatted output as shown in (16); applying these modifications to ( 15a ) and (15b) yields the same outcome. then, computing exact match on these postprocessed lfs captures the targeted semantic equivalence. this reformatted exact-match metric is used for all results reported in the main text; see appendix c.1 and ## results overall, seq2seq transformers, both trained from scratch and pretrained, display low accuracy on slog (figure 2 ), in line with earlier studies on structural generalization in seq2seq models @xcite . this is also the case for the more recent autoregressive transformer llama, whose performance is similar to that of t5. as figure 2 shows, high accuracy on the full cogs dataset, where 86% of the generalization cases are lexical, can obscure low performance on structural generalization, highlighting the need for the expanded structural generalization tests included in slog. slog additionally reveals weaknesses in the am-parser that cogs did not. while the am-parser achieves 90% accuracy on the structural generalization subset of cogs (figure 2 ), it faces systematic difficulties with several generalization types introduced in slog (figure 3 ). performance varied substantially across generalization categories (figure 3 ); in particular, all models achieved near-perfect accuracy on active subject wh-questions and shallower pp recursion. these cases were the least structurally complex in their respective categories ( §2.3 and §2.1).we highlight specific error types in the rest of this section; see appendix c for full results and additional error analysis. ## related work previous research has shown that recurrent neural network (rnn) architectures often struggle with learning complex long-range relations from simpler formal languages @xcite . our results on slog reveal that unseen long-distance predicateargument dependencies pose considerable difficulty for transformer-based models as well ( §5.2). for filler-gap dependencies, prior work has centered on syntactic tasks involving wh-questions or relative clauses @xcite @xcite . these studies primarily use language modeling as the task and do not require mapping to semantic representations. slog incorporates both long-distance predicateargument and filler-gap dependencies within a semantic parsing setup. generalizing recursive patterns to deeper structures has been investigated in both artificial neural networks and humans using artificial languages @xcite @xcite . our findings underscore transformer-based models' limitations with deeper recursive structures, corroborating the observations of @xcite ; @xcite . in contrast, human studies have shown that they can learn and extrapolate center-embedding patterns to greater depth in artificial languages @xcite . generalization cases in slog draw inspiration from the frequency gaps in natural language, where common patterns serve as a foundation for generalizing to rarer structures. this has connections to language acquisition in children, who have limited exposure to complex, less frequent structures, yet need to generalize to novel complex utterances by extrapolating from familiar linguistic elements @xcite @xcite . human proficiency in such generalizations is attributed to inductive biases rooted in systematic compositional rules. however, the transformer-based models we tested, despite excelling in lexical generalization scenarios, face challenges when presented with unfamiliar linguistic structures requiring such rule induction, hinting at potentially different or inadequate underlying mechanisms. more broadly, how the compositional generalization cases proposed in this work can be connected to human language acquisition is an interesting area of future study. ## conclusions we introduce slog, a semantic parsing dataset that extends the cogs benchmark with a focus on structural generalization, which is often underrepresented in current benchmarks for compositional generalization. using slog, we assess the structural generalization capacity of transformer models (both pretrained and trained from scratch), as well as am-parser, a structure-aware model. while all models achieve good overall accuracy on cogs (≥ 78%), their performance on slog is substantially lower, especially for transformer models (≤ 41%). furthermore, even the structure-aware am-parser, which achieved strong performance on all structural generalization cases of cogs, performs poorly on several of the newly introduced generalization types in slog. our error analysis shows that all transformer models tested struggle with interpreting unseen long-distance dependencies and deeper recursive constructions than observed in training. on the other hand, the am-parser, despite its stronger overall performance (71%), displays categorical failures on gap generalization due to its inherent parser design limitations. overall, slog exposes the limitations of a range of models that have previously been claimed to achieve good compositional generalization, and can serve as a useful analytic tool for guiding future improvements.