C o F - C o T : Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Tasks (2023)

Abstract: While Chain-of-Thought prompting is popular in reasoning tasks, its application to Large Language Models (LLMs) in Natural Language Understanding (NLU) is under-explored. Motivated by multi-step reasoning of LLMs, we propose Coarse-to-Fine Chain-of-Thought (CoF-CoT) approach that breaks down NLU tasks into multiple reasoning steps where LLMs can learn to acquire and leverage essential concepts to solve tasks from different granularities. Moreover, we propose leveraging semantic-based Abstract Meaning Representation (AMR) structured knowledge as an intermediate step to capture the nuances and diverse structures of utterances, and to understand connections between their varying levels of granularity. Our proposed approach is demonstrated effective in assisting the LLMs adapt to the multi-grained NLU tasks under both zero-shot and few-shot multi-domain settings.

## introduction natural language understanding (nlu) of dialogue systems encompasses tasks from different granularities. specifically, while intent detection requires understanding of coarse-grained sentencelevel semantics, slot filling requires fine-grained token-level understanding. moreover, semantic parsing entails the comprehension of connections between both token-level and sentence-level tasks. large language models (llms) possess logical reasoning capability and have yielded exceptional performance @xcite . however, they remain mostly restricted to reasoning tasks. on the other hand, mutlistep reasoning can take place when solving multiple interconnected tasks in a sequential order. in practical nlu systems, as coarse-grained tasks are less challenging, they can be solved first before proceeding to fine-grained tasks. therefore, the coarse-grained tasks' outcomes can provide valuable guidance towards subsequent fine-grained tasks, allowing for deeper semantic understanding of diverse utterances across different domains within nlu systems @xcite @xcite . for instance, consider the utterance "remind john the meeting time at 8am" under reminder domain, recognizing get_reminder_date_time intent is crucial for correctly understanding the existence of person_reminded slot type rather than contact or attendee slot type. chain-of-thought (cot) @xcite provides an intuitive approach to elicit multi-step reasoning from llms automatically. however, there remain two major challenges with the current cot approach: (1) llms entirely rely on their uncontrollable pre-trained knowledge to generate stepby-step reasoning and could result in unexpected hallucinations @xcite ), (2) additional beneficial structured knowledge cannot be injected into llms via the current cot. on the other hand, structured representation demonstrates the effectiveness in enhancing the capability of pre-trained language models (plms) @xcite @xcite . in dialogue systems, the dependencies among different dialogue elements together with the existent diversely structured utterances necessitate the inte- step 1: generate amr input: sentence, domain output: amr step 2: generate intent input: sentence, intent vocabulary, amr, domain output: intent step 3: generate slot values input: sentence, amr, intent, domain output: slot values step 4: generate (slot value, slot type) pairs input: sentence, amr, slot vocabulary, slot values, intent, domain output: (slot value, slot type) pairs step 5: generate logic form input: sentence, intent, domain, (slot value, slot type) pairs output: logic form show me dates for music festivals in 2018 [in:get_event [sl:category_event music festivals ] [sl:date_time in 2018 ] ] (c/ show-01 :arg1 (m /me) :arg2 (d/ date-entity :mod (f/ music-festival) :year 2018)) get_event show me, dates, music festival, in 2018 (show me, o),(dates, o), (music festival, category_event), (in 2018, date_time) [in:get_date_time_event [sl:title_event music festivals] [sl:date_time 2018]] in our work, we explore the capability of llms in nlu tasks from various granularities, namely multi-grained nlu tasks. motivated by cot, we propose an adaptation of cot in solving multigrained nlu tasks with an integration of structured knowledge from amr graph. our contribution can be summarized as follows: structured representation structured representation has been widely incorporated in language models to further enhance the capability across various nlp tasks @xcite . structured representation can be either in the syntax-based structure @xcite such as dependency parsing (dp) graph, constituency parsing (cp) graph or semantic-based structure @xcite such as abstract meaning representation (amr) graph @xcite . unlike previous works, we aim at leveraging structured representation as an intermediate step in the multi-step reasoning approach to extract essential concepts from diverse utterances in the multi-domain dialogue systems. ## proposed framework in this section, we introduce our proposed coarseto-fine chain-of-thought (cof-cot) approach for nlu tasks as depicted in figure 2 . specifically, we propose a breakdown of multi-grained nlu tasks into 5 sequential steps from coarse-grained to finegrained tasks. at each step, llms leverage the information from the previous steps as a guidance towards the current predictions. as domain name could provide guidance to nlu tasks @xcite , at each step, we condition the domain name of the given utterance in the input prompt. the model's output is in the format of logic form (kamath and das) which encapsulates coarse-grained intent label, fine-grained slot labels and slot values. further details of logic form's structure and its connections with multi-grained nlu tasks are provided in the appendix a. our multi-step reasoning is designed in the following sequential order: ## result & discussion in addition, we observe consistent improvements of different cot variants over the direct prompt. it implies that cot prompting allows the model to reason over multiple steps and learn the connections between different nlu tasks more effectively. in comparison with massive, performance of all methods is significantly lower on mtop. it is mainly due to the more complex logic form structures existent in mtop. it is noticeable that mas-sive datasets contain samples of fewer average number of slots, leading to significantly better performance on semantic parsing tasks (i.e. frame accuracy and exact match). our cof-cot shares certain degrees of similarities with least-to-most (zhou et al., 2022), planand-solve prompting @xcite . however, unlike the two aforementioned baselines that rely heavily on the existent pre-trained knowledge of llms, cof-cot provides a controllable number of sequential steps and conditioning inputs for each step, allowing for flexible adaptations and customizations to future downstream tasks that llms might not be familiar with. comparison with ft under zsl and fsl settings, the ft model suffers from the aforementioned domain gap issues. specifically, as observed in ## conclusion in this work, we conduct a preliminary study of llms' capability in multi-grained nlu tasks of dialogue systems. moreover, motivated by cot, we propose a novel cof-cot approach aiming to break down nlu tasks into multiple reasoning steps where (1) llms can learn to acquire and leverage concepts from different granularities of nlu tasks, (2) additional amr structured representation can be integrated and leveraged throughout the multi-step reasoning. we empirically demonstrate the effectiveness of cof-cot in improving llms capability in multi-grained nlu tasks under both zero-shot and few-shot multi-domain settings.