Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking (2023)

Abstract: Zero-shot Dialogue State Tracking (DST) addresses the challenge of acquiring and annotating task-oriented dialogues, which can be time-consuming and costly. However, DST extends beyond simple slot-filling and requires effective updating strategies for tracking dialogue state as conversations progress. In this paper, we propose ParsingDST, a new In-Context Learning (ICL) method, to introduce additional intricate updating strategies in zero-shot DST. Our approach reformulates the DST task by leveraging powerful Large Language Models (LLMs) and translating the original dialogue text to JSON through semantic parsing as an intermediate state. We also design a novel framework that includes more modules to ensure the effectiveness of updating strategies in the text-to-JSON process. Experimental results demonstrate that our approach outperforms existing zero-shot DST methods on MultiWOZ, exhibiting significant improvements in Joint Goal Accuracy (JGA) and slot accuracy compared to existing ICL methods.

## introduction dialogue state tracking (dst) is crucial in task-oriented dialogue (tod) systems to understand and manage user intentions @xcite @xcite @xcite . collecting and annotating dialogue states at the turn level is challenging and expensive @xcite , and commercial applications often need to expand the schema and include new domains. therefore, it is vital to develop dst learning strategies that can adapt and scale effectively with minimal data. most of the existing fine-tuning methods for zero-shot dst have primarily focused on domaintransfer approaches @xcite , which have not yielded satisfactory * the first two authors contribute equally. weiran xu is the corresponding author. decode state directly system: the address of it is xxx street, do you want the xxx hotel with a low price? user: no, i want an attraction xxx instead. hotel-name: xxx hotel price-range: cheap attraction-name: xxx attraction-name: xxx {sys: {… info: {hotel:{ address: [xxx street], name: [xxx hotel], price-range: [cheap]}}, …} {user: { reject: {hotel: { name: [xxx hotel]}}, request: {attraction: { name: [xxx]}}} state updating strategies prompt (instruction + schema description + example) context state hotel-name: xxx hotel only achieve slot filling task but hard to understand intricate state updating strategies in dst … golden label! translate to json mistake not an entity/ not in context out of schema [delete] current turn dialogue black box controllable, interpretable performance. recent studies have showcased the impressive and adaptable expertise possessed by large language models (llms) @xcite @xcite . however, with the discovery of the emergent ability of large language models in downstream tasks @xcite , the in-context learning (icl) method @xcite has garnered more attention in dst research. icl offers a more flexible and cost-effective approach as it eliminates the need for retraining when new slots or domains are introduced. some recent studies have explored the effectiveness of icl in dst @xcite @xcite such as the ic-dst method, which has demonstrated remarkable performance surpassing previous fine-tuning methods. in this work, we proposed a new icl method named parsingdst for zero-shot dst. specifically, we reformulate the dst task by semantic parsing that translates the origin dialogue text into the designed json format (text-to-json) as the intermediate state through the llms the translated json data will include the information about the speaker, interaction action, domain, slot, and corresponding values in a formatted and structured manner, which enables the introduction of manual state updating strategies later. we illustrated the significance of updating strategies by presenting an intricate scenario in figure 1 . previous icl approaches typically decode the dialogue to directly extract the values of all slots, such as "hotel-name", "price-range", and "attractionname" in the example, without considering the updating strategies, which leads to errors in the result. in contrast, our method applies additional updating strategies after the text-to-json conversion. we delete non-entity slots like "price_range" when it appears in the system but not in the context, and we set the value of the "hotel name" to "[delete]" when it is mentioned in the user's "reject" action, which makes our result aligns with the golden label. based on our observations, we noticed that when merging context, system, and user utterances as dialogue input like the previous methods, the llms will often exhibit confusion between these different types of information, which will invalidate the updating strategies. to address this, we also designed a companion framework with more process modules to ensure the effectiveness of updating strategies and conducted experiments to validate them. 1 we choose to test and compare icl meth-1 codes in github.com/tolightupthesky/parsingdst ods with the powerful llms by openai foot_0 : gpt-3.5turbo-0301 foot_1 and text-davinci-003 @xcite on multiwoz 2.1&2.4 dataset. : in summary, our work makes the following contributions: (1) to our best knowledge, we are the first to apply text-to-json by semantic parsing as the transition between dialogue to state, making it possible to tool additional state updating strategies and other modules, making the process of dst more controllable and interpretable. (2) we proposed a novel framework with modules in the process of text-to-json to maintain the effectiveness of updating strategies. ## conclusion in this paper, we reformulate dst to semantic parsing with llms that translate the dialogue text into json as the intermediate state, which enables us to introduce updating strategies and makes dst process more controllable and interpretable. furthermore, we present a novel framework that includes more modules within the text-to-json conversion process to ensure the effectiveness of updating strategies. through experiments on the mul-tiwoz 2.1&2.4 dataset, our system achieves sota performance in zero-shot settings. our analyses showed that our method surpasses the previous icl method and the innovative modules and framework benefit the overall performance. in future work, we plan to explore and evaluate more variety of updating strategies and modules to further enhance our framework and we will test our method in more open-source llms like llama @xcite .