G en SF : Simultaneous Adaptation of Generative Pre-trained Models and Slot Filling (2021)

Abstract: In transfer learning, it is imperative to achieve strong alignment between a pre-trained model and a downstream task. Prior work has done this by proposing task-specific pre-training objectives, which sacrifices the inherent scalability of the transfer learning paradigm. We instead achieve strong alignment by simultaneously modifying both the pre-trained model and the formulation of the downstream task, which is more efficient and preserves the scalability of transfer learning. We present GenSF (Generative Slot Filling), which leverages a generative pre-trained open-domain dialog model for slot filling. GenSF (1) adapts the pre-trained model by incorporating inductive biases about the task and (2) adapts the downstream task by reformulating slot filling to better leverage the pre-trained model’s capabilities. GenSF achieves state-of-the-art results on two slot filling datasets with strong gains in few-shot and zero-shot settings. We achieve a 9 F1 score improvement in zero-shot slot filling. This highlights the value of strong alignment between the pre-trained model and the downstream task.

## introduction the advent of pre-trained language models @xcite has transformed natural language processing. the dominant paradigm has shifted away from designing task-specific architectures towards transfer learning. fine-tuning pre-trained models on downstream datasets achieves strong performance on a variety of natural language understanding tasks @xcite . generally, prior to fine-tuning, the pre-trained models are adapted to the specifics of the downstream task through minor architectural modifications (e.g., adding a classification layer) @xcite . by avoiding major task-specific changes to the models, it figure 1 : to achieve a stronger alignment, both the downstream task and the pre-trained models must be adapted. the downstream task can be adapted with knowledge of the properties and capabilities of the pretrained models. likewise, the pre-trained model can be adapted with knowledge of the downstream task/data. recent work has validated the idea that stronger alignment between pre-training and the downstream task results in improved performance. rather than fine-tuning off-the-shelf models, it is more effective to first understand the downstream task and adapt the model's architecture, pre-training and inference algorithm accordingly. adapting pre-trained models in this manner is equivalent to incorporating inductive biases about the downstream task. for example, pretraining on open-domain dialog data results improves performance on downstream dialog tasks @xcite . designing task-specific pre-training objectives has yielded strong results in extractive question answering @xcite , paraphrase and translation @xcite and slot filling @xcite . this body of work attains stronger alignment by significantly modifying the pre-trained model through task-specific pre-training. this necessitates a new pre-trained model for every downstream task, and therefore relinquishes the inherent scalability of the transfer learning paradigm. instead, we achieve stronger alignment by simultaneously adapting both the pretrained model and the downstream task, such that both contain inductive biases about one another. the downstream task can be adapted to achieve stronger alignment with the capabilities of the pretrained model. to effectively leverage pre-trained models, it is important to first understand the properties and capabilities of the model derived from the model architecture, the pre-training data and task. then the downstream task can be adapted to be better aligned with the model. adapting the task to the model is equivalent to incorporating inductive biases about the pre-trained model into the downstream task. for example, given a pre-trained model that was trained with a ranking objective, it is likely to be more effective if the downstream fine-tuning and inference algorithms are modified to rank rather than to classify. by simultaneously adapting both the downstream task and the pretrained model, we intend to achieve stronger alignment without sacrificing the inherent scalability of the transfer learning paradigm (i.e., avoiding task-specific pre-trained models). we address the task of slot filling, a natural language understanding task with the goal of identifying values for pre-defined attributes (slots) in a natural language utterance. we leverage a di-alogpt @xcite , a generative language model, pre-trained on open-domain dialog data. to achieve strong alignment between the slot filling task and dialogpt, we (1) reformulate slot filling as a natural language response generation task, and (2) augment the dialogpt architecture with a copy-mechanism, constrained decoding and a post-processing heuristic. the resulting model, gensf (generative slot filling), is shown to achieve state-of-the-art results on two slot filling datasets. gensf achieves the strongest performance gains in few-shot and zero-shot settings, highlighting the importance of stronger alignment in the absence of abundant data. our code is open-sourced and can be found at https://github. com/shikib/generative_slot_filling . ## related work slot filling is the task of identifying values for predefined attributes, or slots, in a natural language utterance @xcite . slot filling is a vital natural language understanding component of task-oriented dialog systems @xcite . a variety of architectures have been explored for the task of slot filling, including cnns @xcite , deep lstms @xcite , rnns with external memory @xcite , encoder labeler lstms @xcite and joint pointer and attention seq2seq networks @xcite . with the introduction of large-scale pre-trained language models @xcite , strong slot filling results have been achieved with simple architectures @xcite . several approaches have been proposed for zeroshot slot filling. @xcite leverage slot names and descriptions to align slots across domains. @xcite leverage examples for zero-shot slot filling. @xcite achieve strong results in zero-shot slot filling with a coarseto-fine approach in combination with template regularization. we use the coach+tr model @xcite as a baseline in our zero-shot experiments. working on the hypothesis that pre-trained language models, such as bert @xcite , do not effectively capture the intricacies of dialog, recent work has attempted to mitigate this issue. coope et al. (2020) use convert @xcite , a lightweight model pre-trained on dialog data, in combination with cnn and conditional random field (crf) to outperform bert. @xcite achieves similar results with convbert, a model that further pre-trains bert on open-domain dialog data. recently, henderson and vulić (2020) introduces a 'pairwise cloze' pre-training objective that uses open-domain dialog data to specifically pre-train for the task of slot filling. the resulting convex model achieves significant improvements, particularly in few-shot settings. a common theme in recent work is achieving better alignment between the pre-trained models and the downstream task, either by pre-training on data that is closer to the domain of the downstream task (i.e., dialog data) @xcite or by designing custom pre-training objectives that better model the down-stream task @xcite . our proposed approach shares the goal of achieving better alignment, but we simultaneously adapt both the pre-trained model and the downstream task, with the goal of leveraging a generative pre-trained dialog model, dialogpt, for slot filling. ## methods in order to effectively leverage a pre-trained generative dialog model, dialogpt @xcite , for the task of slot-filling, we introduce the gensf model which achieves stronger alignment between the downstream task and the pre-trained model, by simultaneously adapting the task to the model and the model to the task. this paper first describes how the slot filling task is reformulated as a natural language response generation task to be better aligned with the dialogpt model. next, it describes several modifications to the dialogpt architecture and inference algorithm that act as inductive biases for the slot filling task. ## experiments experiments are performed to empirically validate the hypothesis that simultaneously adapting the downstream task and the pre-trained model results in stronger alignment and improved performance. we present experiments on two datasets and assess gensf in full-data, few-shot and zero-shot settings. an ablation study is performed to characterize the source of the performance gains and demonstrate the importance of simultaneous adaptation. ## conclusion this paper simultaneously adapts both the task and the pre-trained model in order to achieve strong alignment between a generative pre-trained dialog model and the downstream slot filling task. the resulting gensf model achieves state-of-the-art results on two slot filling datasets, with particularly strong gains in few-shot and zero-shot settings. the empirical results underlie the importance of incorporating inductive bias into both the task and the pre-trained model. while this paper demonstrates the value of simultaneous adaptation for the task of slot filling, a similar paradigm could potentially be extended to alternate tasks. future work should (1) explore improved mechanism for achieving stronger alignment between the task and the model, (2) extend the simultaneous adaptation strategy to other problems and (3) explore the use of pre-trained generative models for language understanding tasks.