SQATIN : Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU (2024)

Abstract: Task-oriented dialogue (TOD) systems help users execute well-defined tasks across a variety of domains (e.g., flight booking or food ordering ), with their Natural Language Understanding (NLU) components being dedicated to the analysis of user utterances, predicting users’ intents ( Intent Detection , ID) and extracting values for informational slots ( Value Extraction , VE). In most domains, labelled NLU data is scarce, making sample-efficient learning – enabled with effective transfer paradigms – paramount. In this work, we introduce SQATIN, a new framework for dialog NLU based on (i) instruction tuning and (ii) question-answering-based formulation of ID and VE tasks. According to the evaluation on established NLU benchmarks, SQATIN sets the new state of the art in dialogue NLU, substantially surpassing the performance of current models based on standard fine-tuning objectives in both in-domain training and cross-domain transfer, and it also surpasses off-the-shelf large language models for the same task, both in terms of performance and inference efficiency. Furthermore, SQATIN yields particularly large performance gains in cross-domain transfer, owing to the fact that our QA-based instruction tuning leverages similarities between natural language descriptions of classes (i.e., slots and intents) across domains.

## introduction task-oriented dialogue (tod) systems support users in execution of specific, well-defined tasks through natural language interaction (e.g., ordering food or purchasing tickets) @xcite . fine-grained understanding of user's utterances, commonly referred to as (dialogue) natural language understanding (nlu) is necessary for successful tod @xcite . nlu modules of tod systems typically solve two complementary tasks: (1) intent detection (id) aims to recognise the purpose (i.e., intent) of the user's utterance, classifying utterances into a set of predefined classes (e.g., the intent lost_luggage in flight booking); (2) value extraction (ve) aims to extract spans that express values for any of the predefined informational slots (e.g., a dialog system for booking flights would have slots such as origin, destination, time, maximal_price). realistic tod setups for both id and ve typically involve a relatively large number of labels (e.g., >100 different intent classes), commonly with a limited number of labelled instances per class. successfully addressing these tasks thus amounts to enabling sample-efficient learning by means of transferring knowledge from other tasks @xcite , languages @xcite , or domains @xcite . in recent years -in line with general nlp trends -most nlu models @xcite @xcite were obtained via standard, task-specific fine-tuning of pretrained transformerbased language models (plms) @xcite . standard fine-tuning comes with task-specific (discriminative) objectives -different from lm-ing as the pretraining objectivewhich in principle impedes both knowledge transfer (1) from pretraining to downstream tasks and (2) between different downstream tasks. prompting in contrast @xcite recasts downstream tasks into language modelling, making them more aligned with the models' pretraining. finally, instruction-tuning @xcite -supervised training in which prompts created from instances are prepended with natural language descriptions of the tasks -facilitate the transfer between arbitrary tasks, leveraging the generalisation over task descriptions for zero-shot inference (i.e., inference for tasks unseen in training). despite the impressive zero-shot and in-context fewshot inference abilities of the more recent large lms (llms) @xcite @xcite , supervised fine-tuning still brings substantial performance gains for dialog nlu @xcite and standard nlu tasks, even with low-resource supervision @xcite . as generalisation to new domains (with limited in-domain annotation effort) is one of the main desiderata of tod, some recent work on dialog nlu @xcite has recognised that id and ve can be cast as question answering (qa) tasks: this facilitates transfer from models trained on large qa datasets @xcite , allowing also to capitalise on other large datasets previously recast as qa @xcite . these efforts, however, amount to sequential transfer with standard fine-tuning for qa and thus (i) do not align their fine-tuning with the models' pretraining objective; and without an lm-based objective they (ii) cannot benefit from cross-task transfer via natural language task formulations. contributions. motivated by the above observations, we propose a new framework for dialogue nlu driven by qa-based instruction tuning. in sqatin (supervised question answering tuning on instructions for dialogue nlu), we reformulate id and ve into qa-based natural language instructions and, starting from a massively instruction-tuned plm @xcite , finetune it for our tasks relying on a small number of indomain examples. the rationale behind sqatin is two-pronged: (1) transfer with a model that was previously instruction-tuned at scale improves the efficiency of learning from task-specific samples -this is highly desirable in most tod domains, where one typically deals with only a handful of labelled utterances; (2) while small-scale id/ve instruction-tuning specialises the model for a particular tod domain (e.g., restaurant booking), the negligible size of in-domain training (compared to model's massive instruction-"pretraining") should prevent overfitting to the tod training domain and allow for effective cross-domain transfer. our results strongly support both of the above assumptions: sqatin yields state-of-the-art performance on two prominent dialogue nlu benchmarks both in in-domain and cross-domain evaluations. sqatin brings particularly large gains in transfer between close tod domains: classes in these domains have similar prompt descrip-tions, unlike the existing approaches based on standard fine-tuning. the code is openly available at https://github.com/cambridgeltl/sqatin/ . ## sqatin: methodology standard classification vs. instruction tuning for dialog nlu. id and ve are two tasks that comprise most dialogue nlu modules. both tasks are commonly cast as classification tasks: id as a sequence classification task (i.e., one or more intent labels assigned for the whole utterance) and ve as a span extraction task, i.e., token-level classification. in standard classification with pretrained lms, a task-specific classifier c t : @xmath0. this means that a classifier c t , trained for task t with classes c t , cannot be used to make predictions for any other classification task t ′ with a different set of classes c t ′ : thus, transfer between tasks can only occur indirectly through the parameters of the lm. this is particularly unfortunate for domain transfer in dialog nlu, where different domains often have semantically overlapping id and ve classes (e.g., intent confirm_order is essentially the same intent in flight booking and in food ordering). in contrast, instruction-tuning recasts classification as a language modelling (i.e., generation) task lm : @xmath1, with v t as the subset of the lm's vocabulary where each token v @xmath2. this removes the need for a task-specific classifier (on top of the lm) and facilitates transfer between tasks, especially those with semantically overlapping class tokens. ## experimental setup we rely on the flan-t5 instruction-pretrained models @xcite . unless stated otherwise, the main model is the base variant. training hyperparameters are described in detail in appendix d. ## main evaluation preliminary study: zero-shot id & ve. the key hypothesis behind sqatin is that instructiontuned models have stronger inductive bias for dialogue nlu than models fine-tuned in the standard manner, including those trained for qa @xcite . we thus preliminarily compare zero-shot id/ve performance of (1) the instruction-trained flan-t5 and (2) t5 finetuned for qa on squad2.0 (denoted qa-t5) on nlu++. the results in in-domain results. we next compare the supervised in-domain performance (i.e., training and test instances from the same domain) of sqatin against the cl-se and qa-ft baselines. tables 2 and 3 display the results on nlu++ and clinc-150, respectively. on nlu++, we additionally provide qa-ft results with two other base models, roberta @xcite and mdeberta @xcite , copied directly from @xcite and @xcite , respectively. sqatin consistently and considerably outperforms the baseline models, on both tasks and on both datasets. these results confirm that instructionbased models have stronger inductive biases than qa-fine-tuned models: these biases are propagated in task-specific instruction-based fine-tuning, resulting in sota performance. the gains seem more pronounced in setups with less training data (i.e., 20-fold in ## further analyses and discussion cross-task generalisation. we next hypothesise that sqatin facilitates transfer between the two dialogue nlu tasks, given that sqatin's qa formulation conceptually allows for such cross-task transfer and presents both tasks to the model in the same format. for ve. we observe substantial improvements in id after instruction-tuning for ve (around 5% in the banking domain and over 10% in the hotels domain), proving effective cross-task generalisation of sqatin in dialogue nlu. we then fine-tune the models jointly on id and ve. ## conclusion we have introduced a novel framework for dialogue nlu, sqatin, which combined (i) supervised instruction tuning and (ii) question-answering formulation of intent detection and value extraction. we evaluated sqatin on two established dialogue nlu benchmarks, demonstrating that sqatin brings substantial and consistent improvements over the existing sota approaches. the perfor6 we note for completeness that prior work proposed to utilise masked language modelling abilities of pretrained models on standard, non-dialogue nlu tasks via cloze-style phrases @xcite and demonstrated the potential of using few labelled examples for parameter updates @xcite . @xcite show the effectiveness of prompt-based fine-tuning of masked language models when several task demonstrations are included into the input context. this is in contrast to sqatin which a) focuses on dialogue nlu tasks; b) utilises instruction-tuned generative language models and relies on their instruction-following, rather than mask-filling capabilities. mance gains are especially pronounced in crossdomain transfer, as sqatin can leverage similarities between classes across domains via their descriptions. sqatin also performs well in crosstask transfer, enabling the two dialogue nlu tasks to benefit from one another. we also show that sqatin supports parameter-efficient fine-tuning and that it largely outperforms icl with much larger (and more expensive) language models.