R e COGS : How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation (2023)

Abstract: Compositional generalization benchmarks for semantic parsing seek to assess whether models can accurately compute meanings for novel sentences, but operationalize this in terms of logical form (LF) prediction. This raises the concern that semantically irrelevant details of the chosen LFs could shape model performance. We argue that this concern is realized for the COGS benchmark (Kim and Linzen, 2020). COGS poses generalization splits that appear impossible for present-day models, which could be taken as an indictment of those models. However, we show that the negative results trace to incidental features of COGS LFs. Converting these LFs to semantically equivalent ones and factoring out capabilities unrelated to semantic interpretation, we find that even baseline models get traction. A recent variable-free translation of COGS LFs suggests similar conclusions, but we observe this format is not semantically equivalent; it is incapable of accurately representing some COGS meanings. These findings inform our proposal for ReCOGS, a modified version of COGS that comes closer to assessing the target semantic capabilities while remaining very challenging. Overall, our results reaffirm the importance of compositional generalization and careful benchmark task design.

## introduction compositional generalization benchmarks have emerged as powerful tools for diagnosing whether models have learned to systematically interpret natural language @xcite @xcite @xcite . the core task is to map sentences to logical forms (lfs), and the goal is to make accurate predictions for held-out examples that include novel grammatical combinations of elements. these tasks are guided by the assumption that natural languages are governed by the principle of compositionality: the meanings of complex phrases are determined by the meanings of their parts and the way those parts combine syntactically, and so the meanings of novel combinations of familiar elements are fully determined as well @xcite @xcite . the cogs (compositional generalization challenge based on semantic interpretation) benchmark of @xcite is among the most widely used compositional generalization benchmarks at present, and it is noteworthy for containing assessment splits that almost no present-day models get traction on ( we argue that this conclusion is hasty. the core issue is one of semantic representation. the goal of cogs is to assess whether models can compute meanings compositionally, but this is operationalized as the task of predicting lfs in a specific format. there will in general be innumerable lf formats that express the desired meanings, and we have no reason to privilege any particular one. in this paper, we focus on three incidental details of syntactic form that profoundly impact model performance and thus challenge the validity of cogs. first, we find that when we make trivial, meaning-preserving modifications to the lfs by removing redundant symbols, we see substantial improvements in model performance (section 4.2). we also identify subtler issues related to variable binding. in cogs, all variables are bound. 1 ## related work approaches to cogs researchers have adopted a variety of approaches to solving cogs, including grammar-based rules @xcite , lexicons or lexicon-style alignments incorporated into seq2seq models @xcite , modified transformer models for betterstructured representations @xcite @xcite , meta-learning @xcite , tree-like neural parsers @xcite , a grammar-enhanced seq2seq learner @xcite , and various data augmentation techniques @xcite . ## experiments we report on experiments studying semantic representations for cogs. for any modifications we apply to the dataset, we ensure there is no data leakage, as such leakage trivializes the generalization tests (as evidenced by the consistently very high results for cogs iid test splits in prior work). ## recogs: a revised version of cogs we now propose a revision to cogs called recogs. we assemble the insights provided by the above experiments and use them to inform a benchmark that comes closer to assessing models purely on their ability to handle semantic generalizations. as a metric for recogs, we propose semantic exact match (sem). for sem, we exhaustively check whether there is a semantically consistent conversion of the variables in the predicted lfs into those of the actual lf. for cogs, this amounts to checking whether there is bijective map between variables in the predicted and actual lfs, where corresponding variables participate in exactly the same predications. 6 for instance, the predicted lf while both our models still achieve near 100% performance on the iid testing split, they struggle on the structural and lexical generalization tests. however, they now show signs of life. for instance, the object @xmath0. this shows how the meaning-preserving modifications of ''um'' insertion and preposing can remove artifacts that were preventing models from learning before. in addition, our results show that lstms perform better than transformers on the same split (13.4% vs. 4.5% on average). future work may investigate the reason why lstms seem to generalize better in this scenario. we also see the effects of breaking the relationship between variable names and linear position. without this non-semantic pattern to rely on, models show degraded performance. to verify this, we revert recogs back to the original variable name bindings of cogs where the order of lfs mirrors the word order in the input sequence, and generate a new variant, recogs pos . as shown in figure 5 , performance of both models increases significantly on recogs pos . consequently, we believe that recogs poses additional challenges in terms of long-term coindexing over entities, challenges that were partly obscured by the variable naming scheme chosen for cogs. ## discussion and conclusion with compositional generalization benchmarks, we hope to gain reliable information about whether models have learned to construct the meanings of natural language sentences. however, meanings are highly abstract entities that we do not know how to specify directly, so we are compelled to conduct these assessments using lfs, which are syntactic objects that we presume can themselves be mapped to meanings in the relevant sense. the central difficulty here is that there is no single privileged class of lfs to use for this purpose, and different lfs are likely to pose substantially different learning problems and thus may lead to very different conclusions about our guiding question. we explored these issues in the context of cogs, a prominent compositional generalization benchmark. cogs includes sub-tasks that even our best present-day models cannot get any traction on. our central finding is that there are two major factors contributing to this abysmal performance: redundant symbols in cogs lfs and the requirement imposed by cogs that models predict the exact numerical values of bound variables. these details cannot be justified semantically, and they play a large role in shaping model performance. we showed that simple, meaning-preserving modifications to cogs fully address these problems and allow models to succeed. in turn, we propose recogs, a modified version of cogs that incorporates these insights. our models are able to get some traction on all the sub-tasks within recogs, but it remains a very challenging benchmark. one of our reviewers raised an important question relating to recogs and benchmarking efforts that might follow from it: could this encourage an unproductive sort of ''lf hacking'' in which people continually reformat the data in an effort to incrementally boost performance? this is certainly a risk. however, for now, we venture that such experimentation can be productive. if the modified lfs are truth-conditionally identical to the originals, we can be confident that the generalization splits are not being compromised by this lf hacking. thus, consistent improvements in performance may lead to lasting insights about effective meaning representation for our models, and persistent failures are likely to point to deep limitations. overall, then, some lf hacking could help us get incrementally closer to truly evaluating the capacity of models to compute meaning. our results also raise important questions about compositional generalization itself. it is in the nature of compositional generalization tasks that models will be assessed on examples that are meaningfully different from those they have seen in training. when is this fair, and when does it implicitly contradict the train set itself? our discussion focused on the distribution of pp modifiers across different grammatical positions. cogs models are trained on examples that might suggest these are limited to object noun phrases. this may seem implausible for pp modifiers, but many natural languages do display subject/object asymmetries with regard to phenomena like case marking, definiteness, and pronominalization. in our experiments, we indirectly instructed our model about our intended generalization via meaning preserving modifications to the training data, but ultimately these issues may call for deeper changes to how we pose compositional generalization tasks.