Leveraging Code to Improve In-Context Learning for Semantic Parsing (2024)

Abstract: In-context learning (ICL) is an appealing approach for semantic parsing due to its few-shot nature and improved generalization. However, learning to parse to rare domain-specific languages (DSLs) from just a few demonstrations is challenging, limiting the performance of even the most capable LLMs.In this work, we show how pre-existing coding abilities of LLMs can be leveraged for semantic parsing by (1) using general-purpose programming languages such as Python instead of DSLs and (2) augmenting prompts with a structured domain description that includes, e.g., the available classes and functions. We show that both these changes significantly improve accuracy across three popular datasets; combined, they lead to dramatic improvements (e.g., 7.9% to 66.5% on SMCalFlow compositional split) and can substantially improve compositional generalization, nearly closing the performance gap between easier i.i.d. and harder compositional splits. Finally, comparisons across multiple PLs and DSL variations suggest that the similarity of a target language to general-purpose code is more important than prevalence in pretraining corpora. Our findings provide an improved methodology for building semantic parsers in the modern context of ICL with LLMs.

## introduction semantic parsing, the task of translating natural language utterances to structured meaning representations @xcite is a core requirement for building task-oriented dialog systems and voice assistants. this task is primarily addressed with two approaches: fine-tuning models on labeled datasets of utterances mapped to domain-specific language (dsl) programs @xcite @xcite ; when prompted with python, the model leverages its pre-existing knowledge of coding to produce the correct program and answer. @xcite and employing in-context learning (icl; @xcite to prompt a large language model (llm) with a few demonstrations. however, both strategies present significant limitations. fine-tuning requires substantial pools of labeled data, which can be expensive and timeconsuming to obtain. crucially, fine-tuned models also struggle to compositionally generalize, e.g., to decode programs longer than seen during training or to emit unseen structures @xcite @xcite . while icl can improve compositional generalization in some cases @xcite @xcite , learning from a few demonstrations is challenging: llms need to not only understand the meaning of the input utterance but also learn how to correctly use a typically rare domainspecific language (dsl), given only few demonstrations. this makes icl sensitive to demonstration selection @xcite , which may not cover all functionalities and subtleties of a dsl. while prior work has tried to alleviate this with a better selection of demonstration @xcite @xcite , such approaches require access to a large pool of labeled demonstrations to select from and are not applicable in a true few-shot settings. given that llms show remarkable coding abilities in general-purpose programming languages @xcite @xcite , in this work, we ask two main questions: (1) how can we leverage these abilities to improve icl-based semantic parsing? (2) can llms compositionally generalize better with pls rather than dsls? to investigate this, first, we replace dsls with equivalent code written in popular programming languages such as python or javascript. this helps better align the output space with pretraining corpora, obviating the need for llms to learn new syntax, basic operations, or other coding practices from scratch. for example, consider figure 1 : to select a state that has the most major cities, an llm prompted with a dsl needs to use the operator most, for which it might not be given an example. in contrast, with python, the llm can leverage its pre-existing knowledge of code to find such a state. second, we augment the icl prompt with a structured description of the output meaning representation, which we refer to as domain description (dd). this provides domain-specific information such as types of entities, attributes, and methods (e.g., state and its attributes in figure 1 ). while such descriptions can also be added to dsls, we find that domain descriptions for pls are easier to precisely define with explicit declarations of objects, methods, their signatures, etc. furthermore, llms are more likely to leverage descriptions with pls rather than dsls, as using previously defined objects and methods is a common coding practice. we evaluate our approach on both chatgpt 2 and the open-source starcoder model @xcite , by implementing python-executable environments for three complex semantic parsing benchmarks, namely geoquery @xcite , @xcite , and smcalflow @xcite , and annotating them with python programs and dds. in a true few-shot setting, where only a few (e.g., 10) labeled examples are available to use as demonstrations, we find that pl prompts with dds dra-2 https://chat.openai.com/ matically improve execution-based accuracy across the board, e.g., 49.7 points absolute improvement (31.0% to 80.7%) on the length split of geoquery, compared to the standard icl approach of a dslbased prompt with no dd. prompting a model with python and domain description can often even eliminate the need for many demonstrations: with just a single demonstration, accuracy on a compositional split of geoquery reaches 80%, compared to 17% for dsl prompting with no dd. in fact, for two datasets, a single pl demonstration with dd outperforms dsl prompts with as many as 25 demonstrations and an equivalent dd. interestingly, we find that employing python with a dd substantially improves compositional generalization, almost entirely closing the compositionality gap, i.e., the performance difference between an i.i.d. split and harder compositional splits. one might hypothesize that the strong performance of python is due to its prevalence in the pretraining corpus @xcite . to investigate this, we evaluate the performance of pls whose popularity differs from that of python. surprisingly, we find that prevalence in pretraining corpora does not explain superiority: both scala, a pl much rarer than python, and javascript, which is much more prevalent, perform roughly similarly. sql, a common query language, performs better than dsls, but worse than the other more generalpurpose pls. further analyses with simplified versions of dsls indicate that even rare dsls, as long as they resemble general-purpose code, might perform nearly as well as pls, provided a detailed dd is used. in conclusion, we demonstrate that using popular pls instead of dsls and adding domain descriptions dramatically improves icl for semantic parsing while nearly closing the compositionality gap. further, we show that when llms are used for semantic parsing, it is better to either prompt them with pls or design dsls to resemble popular pls. overall, these findings suggest an improved way of building semantic parsing applications in the modern context of in-context learning with llms. effect of meaning representations. to address specific challenges with dsls, previous work has proposed to work with simpler meaning representations (mrs) (herzig et al., 2021; li et al., 2022; wu et al., 2023) or synthetic nl utterances (shin et al., 2021), or prompting models with the grammar of the dsl (wang et al., 2023). recently, jhamtani et al. (2023) used python to satisfy virtual assistant requests. differently from that, our work provides an extensive study exploring the advantage of using code and domain descriptions in semantic parsing, across different datasets and pls. ## setup given a natural language request x and an environment e, our task is to "satisfy" the request as follows by executing a program z: if x is an information-seeking question, program z should output the correct answer y; if x is an action request, z should update the environment e appropri-3 class person: name: str def find_team_of() -> list[person]: … def find_reports_of() -> list[person]: … def find_manager_of() -> person: … class event: attendees: list[person] = none subject: optional[str] = none location: optional[str] = none starts_at: optional[list[datetimeclause]] = none ... class api: def find_person(name: str) -> person: … def get_current_user() -> person: … def add_event(event: event) -> none: … ... ately. for example, an information-seeking question could be "what is the longest river?", where e contains a list of facts about rivers and lengths, and the answer y should be returned based on these facts. an action request could be "set a meeting with john at 10am" where e contains a database with a list of all calendar items, and the task is to update e such that the requested meeting event is created. the environment e can be any type of database or system that provides a way to retrieve information or update it using a formal language program. each environment accepts a different formal language for z, and has its own specific list of accepted operators (see ## domain-augmented pl prompts semantic parsing studies have traditionally used dsls. we posit that using general-purpose pls with a structured description of the domain in hand could better exploit the potential of modern llms, which are pretrained on a mix of code and natural language. leveraging existing coding knowledge. while dsls tailored to specific domains can be valuable for trained domain experts, their rarity makes it challenging for llms to learn them from just a few demonstrations. in contrast, pls are prevalent in pretraining corpora; by prompting llms to generate pls rather than dsls, llms can leverage their existing coding knowledge without the need to learn the syntax and standard operations for a new language from scratch. for instance, consider the operator most in figure 1 . llms with no prior knowledge of the given dsl struggle to correctly apply this operator without sufficient demonstrations. however, with python, the model can exploit its parametric knowledge to perform this operation by employing the built-in max and len operators of python, along with list comprehension. another example is filtering sets of items in λ-dcs ( ## results we first compare python-based prompts with dslbased prompts and the effect of dds ( §6.1). we then experiment with several other pls and variations of dsls to better understand how the design of the output language affects performance ( §6.2). ## conclusions in this work, we have shown that leveraging pls and dds does not only improve the effectiveness of in-context learning for semantic parsing, leading to substantial accuracy improvements across various datasets, but also significantly narrows the performance gap between i.i.d. and compositional splits and reduces the need for large demonstration pools. our findings carry significant implications for the development of semantic parsing applications using modern llms.