Semantically Informed Data Augmentation for Unscoped Episodic Logical Forms (2023)

Abstract: Unscoped Logical Form (ULF) of Episodic Logic is a meaning representation format that captures the overall semantic type structure of natural language while leaving certain finer details, such as word sense and quantifier scope, underspecified for ease of parsing and annotation. While a learned parser exists to convert English to ULF, its performance is severely limited by the lack of a large dataset to train the system. We present a ULF dataset augmentation method that samples type-coherent ULF expressions using the ULF semantic type system and filters out samples corresponding to implausible English sentences using a pretrained language model. Our data augmentation method is configurable with parameters that trade off between plausibility of samples with sample novelty and augmentation size. We find that the best configuration of this augmentation method substantially improves parser performance beyond using the existing unaugmented dataset.

## background ulfs are trees written in parenthesized list form. the leaves of these trees, which we will refer to as atoms, can be: ## system description our system can be broadly broken into two parts: a sampler that takes a single seed ulf as input and generates one new ulf-english pair, and a handler which uses the sampler repeatedly to augment a given dataset. pseudocode for the salient parts of this process is given in appendix e. ## experiments we evaluate our sampler on the hand-annotated ulf 1.0 dataset by @xcite , the only dataset of gold ulf annotations that we are aware of. this dataset has 1,378, 180, and 180 sentences of ulf-english pairs in the training, development, and test sets, respectively. metrics. following prior work on this dataset, we use sembleu as the primary evaluation metric and use el-smatch secondarily for analysis, since it is broken down into f1, precision, and recall components. the sembleu score better reflects the the parser's ability to generate coherent ulfs because it takes into account chains of multiple nodes and edges that el-smatch does not. thus, sembleu is used as the primary evaluation metric for ulf parsing. @xcite describes el-smatch in detail, which includes a method for representing ulfs as a set of triples similar to amrs. when sembleu is run on ulf, the same set-of-triples representation is used for ulfs so that the metric designed for amr can be run on ulf. ## conclusions we presented a data augmentation method for ulfs that leverages ulf's underlying semantic type structure. this method helps alleviate the data sparsity problem that currently exists for ulf parsing, leading to a new state-of-the-art in this task without any change in the parsing model. though we tested our data augmentation method on ulfs, this technique is applicable to any semantic parsing task with an underlying tree-structured compositional type system. for example, parsing in combinatory categorical grammar @xcite is another appropriate candidate for this sampling technique. some details of the sampling procedure can also be improved in obvious, but not trivial ways. for example, our ulf atom sampling procedure uses word frequencies without ulf type information.this leads to an over-representation of type-ambiguous words in our generated samples. we think that type system-driven data augmentation for ulf is a promising way to further improve ulf parser performance. we expect further parsing improvements through refinement of the sampling parameters and expansion of the sampler to include macros. the additional data provided by such augmentation would support more general neural network-based semantic parsers as have been successful in other semantic representations @xcite @xcite . we are hopeful to see an improved semantic parser find utility in ulf-related tasks such as those mentioned at the end of section 1. a baseline ulf parser description @xcite ulf parser is a transition system-based parser where the transition actions are selected using an lstm. this parser modifies the cache transition parser @xcite to better model ulfs. at a high level, the modification introduces methods of generating ulf symbols on the fly from input words, rather than assuming a sequence of symbols as input. these symbol generation methods are further designed to reflect the relationship between ulf symbols and their corresponding words rather than assuming that an arbitrary mapping can exist between them. the cache transition system oracle, which is needed for training, is similarly modified to support these changes in the possible actions.