ConVEx : Data-Efficient and Few-Shot Slot Labeling (2021)

Abstract: We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. Instead of relying on more general pretraining objectives from prior work (e.g., language modeling, response selection), ConVEx’s pretraining objective, a novel pairwise cloze task using Reddit data, is well aligned with its intended usage on sequence labeling tasks. This enables learning domain-specific slot labelers by simply fine-tuning decoding layers of the pretrained general-purpose sequence labeling model, while the majority of the pretrained model’s parameters are kept frozen. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling, with the largest gains in the most challenging, few-shot setups. We believe that ConVEx’s reduced pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its efficient fine-tuning and strong performance, promise wider portability and scalability for data-efficient sequence-labeling tasks in general.

## introduction slot labeling or slot filling is a critical natural language understanding (nlu) component of any task-oriented dialog system @xcite @xcite . its goal is to fill the correct values associated with predefined slots: e.g., a dialog system for restaurant bookings is expected to fill slots such as date, time, and the number of guests with the values extracted from a user utterance (e.g., next thursday, 7pm, 4 people). setting up task-oriented dialog systems, as well as slot labeling methods in particular, to support new tasks and domains is highly challenging due to inherent scarcity of expensive expertannotated data for a plethora of intended use scenarios @xcite @xcite . one plausible and promising solution is the creation of data-efficient models that learn from only a handful annotated examples in few-shot scenarios. this approach has been shown promising for learning intent detectors @xcite @xcite as well as for slot-filling methods @xcite . the dominant paradigm followed by the existing models of few-shot slot labeling is transfer learning @xcite : 1) they rely on representations from models pretrained on large data collections in a self-supervised manner on some general nlp tasks such as (masked) language modeling @xcite @xcite or response selection @xcite @xcite ; and then 2) add additional task-specific layers for modeling the input sequences. however, we detect several gaps with the existing setup, and set to address them in this work. first, recent work in nlp has validated that a stronger alignment between a pretraining task and an end task can yield performance gains for tasks such as extractive question answering @xcite and paraphrase and translation @xcite . we ask whether it is possible to design a pretraining task which is more suitable for slot labeling in conversational applications. second, is it possible to bypass learning sequence-level layers from scratch, and simply fine-tune them after pretraining instead? third, is it possible to build a generally applicable model which fine-tunes pretrained "general" sequence-level layers instead of requiring specialized slot labeling algorithms from prior work @xcite ? inspired by these challenges, we propose con-vex (conversational value extractor), a novel transformer-based neural model which can be pretrained on large quantities of natural language data (e.g., reddit) and then directly fine-tuned to a variety of slot-labeling tasks. similar to prior work @xcite , convex casts slot labeling as a span-based extraction task. for convex, we introduce a new pretraining objective, termed pairwise cloze. this objective aligns well with the target downstream task: slot labeling for dialog, and emulates slot labeling relying on unlabeled sentence pairs from natural language data which share a keyphrase (i.e., a "value" for a specific "slot"). instead of learning them from scratch as in prior work @xcite ), convex's pretrained conditional random fields (crf) layers for sequence modeling are fine-tuned using a small number of labeled in-domain examples. we evaluate convex on a range of diverse dialog slot labeling data sets spanning different domains: dstc8 data sets @xcite , restaurants-8k @xcite , and snips @xcite . convex yields stateof-the-art performance across all evaluation data sets, but its true usefulness and robustness come to the fore in the few-shot scenarios. for instance, it increases average f 1 scores on restaurants-8k over the previous state-of-the-art model @xcite ) from 40.5 to 71.7 with only 64 labeled examples. similar findings are observed with dstc8, and we also report state-of-the-art performance in the 5-shot slot labeling task on snips. in summary, our results validate the benefits of task-aligned pretraining from raw natural language data, with particular gains for data-efficient slot labeling given a limited number of annotated examples, which is a scenario typically met in production. they also clearly demonstrate that competitive performance can be achieved via quick fine-tuning, without heavily engineered specialized methods from prior work @xcite . further, we validate that learning sequence-level layers from scratch is inferior to fine-tuning from pretrained layers. from a broader perspective, we hope that this research will inspire further work on task-aligned pretraining objectives for other nlp tasks beyond slot labeling. from a more focused perspective, we hope that it will guide new approaches to data-efficient slot labeling for dialog. ## methodology before we delve deeper into the description of convex in §2.3, in §2.1 we first describe a novel sentence-pair value extraction pretraining task used by convex, called pairwise cloze, and then in §2.2 a procedure that converts "raw" unlabeled natural language data into training examples. ## experimental setup pretraining: technical details. the convex parameters at pretraining are randomly initialized, including the convert layers, and the model is pretrained on the pairwise cloze reddit data. pre-training proceeds in batches of 256 examples, 64 of which are randomly paired sentences where no value should be extracted, and the remaining being pairs from the training data. this teaches the model that sometimes no value should be predicted, a scenario frequently encountered with slot labeling. ## results and discussion intrinsic (reddit) evaluation. convex reaches a precision of 84.8% and a recall of 85.3% on the held-out reddit test set (see ## conclusion we have introduced convex (conversational value extractor), a light-weight pretraining and finetuning neural approach to slot-labeling dialog tasks. we have demonstrated that it is possible to learn domain-specific slot labelers even in low-data regimes by simply fine-tuning decoder layers of the pretrained general-purpose convex model. the convex framework has achieved a new leap in performance on standard dialog slot-labeling tasks, most notably in few-shot setups, by aligning the pretraining phase with the downstream fine-tuning phase for slot-labeling tasks. in future work, we plan to investigate the limits of data-efficient slot labeling, focusing on one-shot and zero-shot setups. we will also apply convex to related tasks such as named entity recognition and conversational question answering.