Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding (2024)

Abstract: State-of-the-art task-oriented dialogue systems typically rely on task-specific ontologies for fulfilling user queries. The majority of task-oriented dialogue data, such as customer service recordings, comes without ontology and annotation. Such ontologies are normally built manually, limiting the application of specialised systems. Dialogue ontology construction is an approach for automating that process and typically consists of two steps: term extraction and relation extraction. In this work, we focus on relation extraction in a transfer learning set-up. To improve the generalisation, we propose an extension to the decoding mechanism of large language models. We adapt Chain-of-Thought (CoT) decoding, recently developed for reasoning problems, to generative relation extraction. Here, we generate multiple branches in the decoding space and select the relations based on a confidence threshold. By constraining the decoding to ontology terms and relations, we aim to decrease the risk of hallucination. We conduct extensive experimentation on two widely used datasets and find improvements in performance on target ontology for source fine-tuned and one-shot prompted large language models.

## introduction state-of-the-art task-oriented dialogue (tod) systems still rely on a fixed ontology to model their scope @xcite . a tod ontology comprises three levels of hierarchy: domains, slots and values. domains are general topics of interest, slots are types of information about entities in a domain, and values are concrete instantiations of slots. ontology thus forms a hierarchy: it is a directed graph where slots belong to domains and values in turn belong to slots. note that slots can be shared across domains, and so can values. an ontology is typically a prerequisite for generating api calls that access the underlying databases for entity retrieval. further, the ontology defines the dialogue state, which is tracked by the system to determine the next actions given the evolving discourse. the dependency on an ontology poses a significant challenge in transferring existing tod systems to new domains and use cases. although ontology-agnostic approaches do exist, their transfer capabilities are limited and their performance remains sub-par on novel data @xcite . large quantities of domain-specific tod data, e.g. customer service recordings, are frequently available, but tend to come without annotation, rendering direct use for system development difficult @xcite . manual labelling is error-prone, does not scale well and quickly becomes prohibitively expensive @xcite @xcite . despite topical or domain mismatch, existing annotated datasets may provide information about tod that can be leveraged to harness new data. for this reason, we are interested in utilising existing labelled tod datasets to automatically generate a full ontology for new, yet-unlabelled, data. automatic dialogue ontology construction typically consists of two steps, dialogue term extraction @xcite and hierarchy establishment. although hierarchy establishment is often done via clustering @xcite we approach it via relation extraction (re), which is more similar to common information extraction pipelines @xcite . we call this task dialogue ontology relation extraction @xcite . a hierarchy is established by inferring in which level extracted terms lie, and by connecting terms across levels. although large language models (llms) have demonstrated considerable task transfer abilities @xcite , dialogue input: "user": "am looking for a place to to stay that has cheap price range it should be in a type of hotel" "system": "okay , do you have a specific area you want to stay in ?" "user": "no , i just need to make sure it ' s cheap . oh , and i need parking" "system": "i found 1 cheap hotel for you that includes parking . do you like me to book it ?" "user": "yes , please . 6 people 3 nights starting on tuesday ." "system": "i am sorry but i wasn ' t able to book that for you for tuesday . is there another day you would like to stay or perhaps a shorter stay ?" "user": "how about only 2 nights ." "system": "booking was successful . reference number is : 7gawk763 . anything else i can do for you ?" "user": "no , that will be all . good bye ." "system": "thank you for using our services ." prompt: for the listed terms in the dialogue predict the "has value" relation between slots and values in the format @xcite . llm output: branch 0: [hotel, has value, price range] ... branch 1: here are the relations between slots and values: [price range, has value, cheap] ... branch 2: [system, is serving, user] constrain to given terms and relations à [nights, has value, 3] ... low confidence high confidence prompt llm generate completions for top-! most likely first tokens of the generation @xcite . domains are highlighted in green, slots in yellow and values in red. branch 0 predicts an incorrect relation (hotel misclassified as slot) with lower confidence. branch 1 has the highest confidence in the relation prediction, which is why it is selected as the final response. also, it contains a form of reasoning that stresses the type of terms that are part of the relations to be predicted, i.e., slots and values. branch 2 visualises constrained decoding, where the prediction of terms and relations is not possible if they are not present in the input. in this work, we assume that some labelled outof-domain source dialogue data is available to facilitate transfer learning. we examine two strategies of providing source data to an instructiontuned llm; 1) as one-shot examples in the prompt, and 2) as data for an additional round of supervised fine-tuning. we establish a challenging transfer setup by conducting experiments on two wellestablished medium to large scale multi-domain task-oriented dialogue benchmark datasets: mul-tiwoz 2.1 @xcite and the schema-guided dialogue (sgd; @xcite dataset. since our focus is solely on dore, we assume that the results of the first step of ontology construction, namely term extraction, are provided. we propose to improve the decoding mechanism of an llm in order to better leverage task-specific knowledge. concretely, we constrain the generation to terms and relation types given in the model input to force the model to consider terms from the target data and output the desired format. we further adapt chain-of-thought (cot) decoding @xcite , which was recently proposed for logical reasoning, for dore. traditionally, cot methods prompt or train the model to generate reasoning paths before giving the final answer @xcite . cot-decoding on the other hand exploits the observation that the presence of cot-paths is correlated with higher confidence in the predicted answer in logical reasoning. we extend cot-decoding to dore by selecting the final answer based on the confidence of predicted relations in multiple generated model answer branches. our final proposal, constrained cot-decoding for dialogue ontology extraction, is the combination of our cot-decoding approach to re with constrained decoding, see fig. 1 . empirically, this new decoding mechanism significantly outperforms both source one-shot and source finetuned baselines on the target data. our contributions are as follows: dialogue ontology construction we divide dialogue ontology construction into term extraction and relation extraction. vukovic et al. (2022) improve out-of-domain generalisation of a dialogue term extraction model by making use of topological properties of the language model embedding space. nguyen et al. (2023) improve phrasal segmentation of ontology terms via language model probing and contrastive learning. since we evaluate the hierarchy on a global level based on relations, our approach is not directly comparable to clusteringbased approaches such as hudeček et al. (2021); yu et al. (2022). in contrast to these methods, we view hierarchy establishment in isolation as a relation extraction task. yu et al. (2020) present dialogre, a popular dataset for re in short chit-chat dialogues. closest to our approach, albalak et al. (2022) jointly optimise re and explanation generation to improve performance with a model-agnostic framework. xu and chen (2023) propose a zero-shot approach for extracting trigger words for dialogue relation extraction on dialogre. however, these works focus on chit-chat dialogues, which do not include ontology relations. relation extraction with llms llms show promising transfer capabilities out of the box @xcite . direct application to our task however is not promising, as it has been shown that aligned llms such as chatgpt @xcite do not perform well on extracting multiple relations at once @xcite . this shortcoming has been linked to the influence of pre-training data distribution on downstream task performance @xcite . re data in particular amounts to a mere 0.5% of instruction-tuning datasets, and is hardly utilised for model selection @xcite . traditionally, re is performed in a pairwise manner @xcite , resulting in quadratic complexity given the number of terms. this becomes intractable for generative llms when querying the llm separately for each pair of terms. alternatively, one may extract all relations present in a given input with a single llm query, as is common in multi-relation extraction tasks such as documentlevel re. for example, @xcite extract relations by either predicting relations directly, or first predicting possible head entities in a docu-ment. @xcite align llms for zeroshot re by transforming re into a question answering (qa) task, which is more frequent in the instruction-tuning data. constrained decoding constrained decoding limits the tokens that can be generated. it is typically applied to llms to improve downstream task performance, reduce hallucination and ensure certain output formats. @xcite constrain decoding for translation to ensure that certain terminology is used. @xcite use constrained decoding with a lookahead heuristic to speed up adaptation of llms to plan generation according to a given api in tod. we want to force the model to use its inherent task knowledge while transferring abilities to new data. chain-of-thought reasoning llm performance on complex reasoning tasks improves when the model generates a chain of thought (cot). @xcite include examples of multi-step reasoning in the prompt, and @xcite prompt the model in a zero-shot fashion to "think step by step". reasoning capabilities can be further enhanced via specific training on @xcite , or via teaching the model to reason @xcite . in contrast to this, we focus on eliciting model-inherent reasoning capabilities, without the need for specific prompts or training. as described in sec. 3.2, we leverage the fact that a top-k decoding beam usually contains a cot @xcite . ## discussion although the performance of the fine-tuned model is improved by constrained cot-decoding, it is not clear where the improvement comes from based on qualitative analysis alone, as this model generates no reasoning paths. the workings and explainability of constrained cot-decoding for fine-tuned models have not been investigated yet, but are relevant future research directions. our results imply that for tasks that are not frequently used in instruction-tuning data, it can be useful to utilise existing tod data for training an llm rather than annotating a few examples on the target data for the dore task (see section 4.3). due to the length of examples in the dore task, the amount of annotated examples that can fit in the prompt is highly limited, aggravating the applicability of few-shot approaches. the results strengthen the finding that annotated data should be used if available @xcite . constrained cot-decoding only improves performance on the target data, where task knowledge is more useful than distributional knowledge about the source data. @xcite state that for learning tasks where language is used in a functional way to accomplish certain goals, such as dore, self-supervised next-token prediction is not sufficient. instead, the model needs to learn taskspecific information via specialised fine-tuning to solve such tasks, which are not frequently present in pre-training data and involve task-specific reasoning. dore can only be solved by knowing the specific relationship definition provided in the task descriptions, which the model cannot handle if it was not trained on the task. in summary, the presented results reinforce the observation that finetuning and specialised architectures are still needed to solve functional language-based tasks that cannot be solved by pattern matching alone. the computational complexity of cot-decoding is k times higher than regular greedy decoding, however, decoding of the different branches can be done in parallel. nonetheless, one should note that only one run of ontology construction is needed, as the ontology can be repeatedly used for other dialogue modelling tasks such as state tracking. it is worth stressing that cot-decoding only increases inference cost, while training cost remains the same. compared to few-shot prompting, which also increases inference cost due to longer input context, there is no additional annotation cost. approach response baseline -[price per night, has domain], [best western plus toronto north york hotel & suites, has domain], '@xmath0 63" @xmath1 117] -[hotel,has slot,price per night] -[best western plus toronto north york hotel, has domain, hotel] -[hotel,has slot,address] -[best western plus toronto north york hotel, has domain, restaurant]... [hotel,has slot,date] ... cot-decoding [1, has domain, hotel] [best western plus toronto north york hotel & suites, has domain, hotel] [1, has domain, address] [best western premier toronto airport carlingview hotel, has domain, hotel] [hotel, has slot, day] [best western plus toronto north york hotel, has domain, hotel] [hotel, has slot, name] [hotel, has slot, room] [hotel, has slot, area] [hotel, has slot, reference number] ... constrained cot-decoding [1, has domain, hotel] ... [hotel,has slot,price per night] [best western plus toronto north york hotel, has domain, restaurant] [best western plus toronto north york hotel, has domain, hotel] [hotel,has slot,address] [hotel,has slot,check in date] [hotel,has slot,street address] [hotel,has slot,date] [hotel,has slot,tuesday next week] [hotel,has slot,check in date] ... ## conclusion we propose constrained chain-of-thought (cot)decoding, a new decoding mechanism for dialogue ontology generation (dore) in a transfer set-up. an llm using a one-shot example from the source data is significantly improved using the proposed constrained cot-decoding mechanism. fine-tuning an llm on the source data and using constrained cot-decoding for inference on the target data outperforms a one-shot target data model significantly. the results warrant further research into dore in particular, and into eliciting reasoning in llms by adapting the decoding mechanism in general. moreover, we offer a method for applying llms to tasks that are underrepresented in pre-training and where the vanilla llms perform poorly. our method is appealing as it does not necessitate labelling new examples. future research directions include explainability of constrained cot-decoding in fine-tuned llms and including cot-decoding during fine-tuning.