Improving Generalization in Language Model-based Text-to- SQL Semantic Parsing: Two Simple Semantic Boundary-based Techniques (2023)

Abstract: Compositional and domain generalization present significant challenges in semantic parsing, even for state-of-the-art semantic parsers based on pre-trained language models (LMs). In this study, we empirically investigate improving an LM’s generalization in semantic parsing with two simple techniques: at the token level, we introduce a token preprocessing method to preserve the semantic boundaries of tokens produced by LM tokenizers; at the sequence level, we propose to use special tokens to mark the boundaries of components aligned between input and output. Our experimental results on two text-to-SQL semantic parsing datasets show that our token preprocessing, although simple, can substantially improve the LM performance on both types of generalization, and our component boundary marking method is particularly helpful for compositional generalization.

## introduction pre-trained language models (lms) foot_1 such as t5 @xcite have now been more and more widely adopted for semantic parsing due to their promising performance and straightforward architectures @xcite @xcite @xcite . however, recent work revealed that these lms still struggle to generalize on outof-distribution (ood) samples (lake and @xcite @xcite . for example, if a parser has learned "how many heads are in the department" and "how many people are older than 56", it is expected to generalize to "how many heads of the departments token preprocessing (applied to database schema) before: department_management | department :| id , budget_in_billions , num_employees after: department_management | department :| id , budget _ in _ billions , num _ employees token preprocessing (applied to sql) before: select avg ( flight.price) where flight.@xmath0: select average ( flight . price ) where flight . @xmath1: how many heads of the departments are older than 56 ? select count (head.*) where head.@xmath2: [sep0] how many heads of the departments [/sep0] [sep1] are older than 56 ? [/sep1] [sep0] select count (head.*) [/sep0] [sep1] where head.@xmath3 in this paper, we investigate these two types of generalization of lms in text-to-sql semantic parsing, i.e., given a natural language (nl) input and the database schema, producing a sql query that can be executed against the database for desired output. we conduct experiments using the cross-database spider benchmark @xcite and its derivation spider-cg @xcite . compared with existing benchmarks @xcite , this task setting is both more realistic (e.g., containing larger language variations) and more challenging (e.g., requiring grounding to the database context). although previous work tackling the two types of generalization all requires non-trivial engineering effort (see section 2), in this work, we present two simple yet effective techniques, which are extremely easy to implement with lms ( on five evaluation sets, the experimental results based on t5-base show that, albeit simple, our token-level technique dramatically improves both types of lm generalization, and our sequence-level technique is particularly helpful for compositional generalization. combining them together leads to further improvements. our additional experiments further demonstrate the generalizability of our approaches (e.g., to text-to-lisp expression parsing @xcite ). ## related work text-to-sql semantic parsing. this task has received considerate attention since the creation of the wikisql @xcite and spider @xcite datasets. while a large amount of existing work designed specialized architectures for this task @xcite @xcite , there has been a trend of directly fine-tuning pre-trained sequenceto-sequence models as semantic parsers @xcite @xcite . our work follows the same line and proposed approaches to further improve the lm performance. on the other hand, @xcite ; @xcite ; @xcite showed that simplifying the sql representation in a way that the new representation can semantically better align with the nl can dramatically improve the parsing performance. in our work, we follow the natsql representation @xcite as it has better alignments with the nl. ## conclusion in this paper, we present two simple yet effective techniques to improve the domain and compositional generalization of lms in text-to-sql semantic parsing. our techniques aid lms in preserving the semantic boundaries of tokens and components in their input and output. we also demonstrate their potential to be generalized to other semantic parsing tasks.