k NN - ICL : Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning (2024)

Abstract: Task-Oriented Parsing (TOP) enables conversational assistants to interpret user commands expressed in natural language, transforming them into structured outputs that combine elements of both natural language and intent/slot tags. Recently, Large Language Models (LLMs) have achieved impressive performance in synthesizing computer programs based on a natural-language prompt, mitigating the gap between natural language and structured programs. Our paper focuses on harnessing the capabilities of LLMs for semantic parsing tasks, addressing the following three key research questions: 1) How can LLMs be effectively utilized for semantic parsing tasks? 2) What defines an effective prompt? and 3) How can LLM overcome the length constraint and streamline prompt design by including all examples as prompts? We introduce k Nearest Neighbor In-Context Learning (kNN-ICL), which simplifies prompt engineering by allowing it to be built on top of any design strategy while providing access to all demo examples. Extensive experiments show that: 1) Simple ICL without kNN search can achieve a comparable performance with strong supervised models on the TOP tasks, and 2) kNN-ICL significantly improves the comprehension of complex requests by seamlessly integrating ICL with a nearest-neighbor approach. Notably, this enhancement is achieved without the need for additional data or specialized prompts.

## introduction task-oriented parsing (top), which aims to translate the natural-language commands from users into specific actions, such as booking a restaurant, is an essential component in conversational systems @xcite @xcite . over the past few years, a wide range of methods have been developed, ranging from rule-based methods @xcite to neural program synthesis methods @xcite @xcite . the recent advancements in top involve formulating the task as a sequence-to-sequence problem, relying on a wealth of labeled data. this process typically entails feeding user utterances into transformer models @xcite @xcite and designing specialized transformer architectures to generate structured outputs that combine natural language with intent and slot tags like "in:" and "sl:". some alternative approaches attempt to reframe top as conventional tasks, such as canonical paraphrasing @xcite and abstractive question answering @xcite . these approaches aim to reduce the model's burden by eliminating the need to rationalize non-linguistic labels that were not part of pre-training. however, in real-world scenarios, obtaining high-quality training examples with expert annotations can be challenging. additionally, api documentation, which contains valuable information, often remains underutilized within the finetuning paradigm. on the contrary, large language models (llms) excel in challenging few-shot scenarios, where they only need a few examples to generate the desired output and can also understand extensive api documentation. in this paper, we delve into an examination and analysis of the performance of icl in the context of top. besides, following a comprehensive review of prompt design strategies, we introduce knn-icl, which offers adaptable integration with any prompt design strategy, further augmenting model performance. we conducted experiments using three representative models: gpt-neox-20b @xcite , codegen-16b-multi @xcite , and codex (code-davinci-002) @xcite to assess the performance of llms in top. our research revolves around three key questions: question 1: how can we effectively leverage llms for top tasks? we transform top into a code generation task, mapping semantic parse trees to python code (referred to as 'api' hereafter) to align with llms' output format, as shown in figure 1 . question 2: what constitutes proper prompt design for top using llms? we analyze icl performance under various prompt design strategies, including factors: api documentation and exemplar selection methods. our findings include: api documentation benefits the more powerful codex model but can be distracting for the lesscapable codegen and gpt-neox models. unsupervised demo selection proves most effective with icl, thanks to its flexibility in capturing semantically similar examples holistically. question 3: how can we overcome the llm's length constraint and simplify prompt design by including all examples as prompts? instead of solely focusing on llm performance through complex prompt design strategies, we draw inspiration from the retrieval language model approach @xcite and introduce knn-icl. knn-icl enables llms to access all available demo exemplars during inference, harnessing synergy between the copy mechanism and the labeling task. we tackle two key challenges. first, how to infuse top knowledge into llms, considering their limited domain expertise due to the scarcity of labeled data? we address this by including a few demos as knn-icl prompts, guiding llms to follow the desired output pattern. second, how to retrieve the nearest neighbor when there's a representation discrepancy between the knn-icl datastore and the llm prompt? to address this challenge, instead of using the hidden state h t directly from the llm as the current step representation, we utilize a combination of the target utterance and previously generated words from the llm at time step t. this ensures consistency in representation between the knn-icl datastore and llms, offering flexibility in adapting to various prompt design strategies. in summary, the contributions of this paper are three-fold: 1) we examine prompt design strategies for llms in the context of top by framing top as a code generation task. we conclude that similaritybased demo selection is effective for both blackbox and open-source models, with stronger llms benefiting more from documentation; 2) we introduce knn-icl to address the length constraints of llms, offering flexibility to integrate with any prompt design strategy; and 3) extensive experimental results showcase that knn-icl can enhance the generation of better-nested api structures by leveraging guidance from nearest neighbors. ## methodology in this section, we outline the methodology employed in this study. our approach consists of two key components: (1) prompt design for semantic parsing (cf. section 2.3). we begin by crafting an effective prompt for top. this involves varying two crucial elements within the prompt: api documentation and exemplar selection strategies. (2) integration with knn-icl (cf. section 2.4). we then integrate all the exemplars to llm in knn-icl. this integration allows us to harness the collective knowledge from all exemplars within the demo pool, enhancing the generation of the semantic parse api. ## experiments in this section, we address the following key research questions through our experiments: (1) what constitutes an effective prompt design to enhance the performance of llms? (2) how does icl compare to state-of-the-art supervised models on top tasks? (3) what impact does knn-icl, which incorporates all available demo examples from the demo pool through interpolation, have on prediction quality? ## conclusion in this paper, we have delved into the intricacies of prompt design and introduced knn-icl to include all demo examples on the task-oriented semantic parsing task, addressing input length challenges during decoding and lessening prompt engineering efforts. our findings highlight the substantial impact of prompt design on top, with models like codex deriving more benefits from structured documentation than less capable models. the similarity-based exemplar retrieval strategy enhanced model performance significantly. furthermore, knn-icl's integration with a large language model and k nearest neighbor search ensures comprehensive use of training data, offering more consistent results across tasks. this study underscores the critical role of prompt design, pointing to promising avenues for future research.