On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex (2023)

Abstract: Semantic parsing is a technique aimed at constructing a structured representation of the meaning of a natural-language question. Recent advances in language models trained on code have shown superior performance in generating these representations compared to language models trained solely on natural language text. The existing fine-tuned neural semantic parsers are vulnerable to adversarial attacks on natural-language inputs. While it has been established that the robustness of smaller semantic parsers can be enhanced through adversarial training, this approach is not feasible for large language models in real-world scenarios, as it requires both substantial computational resources and expensive human annotation on in-domain semantic parsing data. This paper presents the first empirical study on the adversarial robustness of a prompt-based semantic parser based on CODEX, a stateof-the-art (SOTA) language model trained on code. Our results demonstrate that the large language model of code is vulnerable to carefully crafted adversarial examples. To overcome this challenge, we propose methods for enhancing robustness without requiring substantial amounts of labelled data or intensive computational resources.

## introduction semantic parsing is a technique that transforms natural-language utterances (nls) into machinereadable logical forms (lfs) and has been widely applied in various research fields, such as code generation, question-answering systems, and dialogue systems @xcite . most current state-of-the-art semantic parsers are deep-learning models trained in a supervised manner using indomain data. however, this approach requires a large amount of in-domain semantic parsing data, which can be costly to obtain @xcite . ## robustness evaluation for prompt-based semantic parsing this section gives an overview of our evaluation framework, including the methods of constructing the evaluation corpora and the evaluation metrics to evaluate the robustness of the prompt-based semantic parser. ## conclusion this study examines the robustness of semantic parsers in the context of prompt-based few-shot learning. to achieve this objective, robustness evaluation sets were curated to evaluate the robustness of the prompt-based semantic parser, codex. the research aims to identify methods to improve the robustness of codex. the results of our comprehensive experiments demonstrate that even the prompt-based semantic parser based on a large pretrained language model is susceptible to adversarial attacks. our findings also indicate that various forms of in-context learning can improve the robustness of the prompt-based semantic parser. it is believed that this research will serve as a catalyst for future studies on the robustness of promptbased semantic parsing based on large language models.