S pan- ConveRT : F ew-shot Span Extraction for Dialog with Pretrained Conversational Representations (2020)

Abstract: We introduce Span-ConveRT, a light-weight model for dialog slot-filling which frames the task as a turn-based span extraction task. This formulation allows for a simple integration of conversational knowledge coded in large pretrained conversational models such as ConveRT (Henderson et al., 2019). We show that leveraging such knowledge in Span-ConveRT is especially useful for few-shot learning scenarios: we report consistent gains over 1) a span extractor that trains representations from scratch in the target domain, and 2) a BERT-based span extractor. In order to inspire more work on span extraction for the slot-filling task, we also release RESTAURANTS-8K, a new challenging data set of 8,198 utterances, compiled from actual conversations in the restaurant booking domain.

## introduction conversational agents are finding success in a wide range of well-defined tasks such as customer support, restaurant, train or flight bookings @xcite @xcite , language learning @xcite , and also in domains such as healthcare @xcite or entertainment @xcite . scaling conversational agents to support new domains and tasks, and particular system behaviors is a highly challenging and resource-intensive task: it critically relies on expert knowledge and domain-specific labeled data @xcite @xcite . slot-filling is a crucial component of any taskoriented dialog system @xcite @xcite . for instance, a conversational agent for restaurant bookings must fill all the slots date, time and number of guests with correct values given by the user (e.g. tomorrow, 8pm, 3 people) in order to proceed with a booking. a particular challenge is to deploy slot-filling systems in low-data regimes (i.e., few-shot learning setups), which is needed to enable quick and wide portability of conversational agents. scarcity of in-domain data has typically been addressed using domain adaption from resource-rich domains, e.g. through multitask learning @xcite or ensembling @xcite . in this work, we approach slot-filling as a turnbased span extraction problem similar to @xcite : in our span-convert model we do not restrict values to fixed categories, and simultaneously allow the model to be entirely independent of other components in the dialog system. in order to facilitate slot-filling in resource-lean settings, our main proposal is the effective use of knowledge coded in representations transferred from large general-purpose conversational pretraining models, e.g., the convert model trained on a large reddit data set @xcite . to help guide other work on span extractionbased slot-filling, we also present a new data set of 8,198 user utterances from a commercial restaurant booking system: restaurants-8k. the data set spans 5 slots (date, time, people, first name, last name) and consists of actual user utterances collected "in the wild". this comes with a broad range of natural and colloquial expressions, foot_0 as illustrated in figure 1 , which makes it both a natural and challenging benchmark. each training example is a dialog turn annotated with the slots requested by the system and character-based span indexing for all occurring values. as our key findings show, conversational pretraining is instrumental to span extraction performance in few-shot setups. by using subword representations transferred from convert (henderson et al., 2019a), we demonstrate that: 1) our convert-backed span extraction model outperforms the model based on transferred bert representations, and 2) it also yields consistent gains over a span extraction model trained from scratch in the target domains, with large gains reported in few-shot scenarios. we verify both findings on the new restaurants-8k data set, as well as on four dstc8-based data sets (rastogi et al., 2019). all of the data sets used in this work are available online at: https://github. com/polyai-ldn/task-specific-datasets . ## methodology: span-convert before we delve into describing the core methodology, we note that in this work we are not concerned with the task of normalizing extracted spans to their actual values: this can be solved effectively with rule-based systems after the span extraction step for cases such as times, dates, and party sizes. there exist hierarchical rule-based parsing engines (e.g., duckling) that allow for parsing times and dates such as "the day after next tuesday". further, phrases such as "me and my wife and 2 kids" can be parsed using singular noun and number counts in the span with high precision. span extraction for dialog. we have recently witnessed increasing interest in intent-restricted approaches @xcite @xcite for slot-filling. in this line of work, slot-filling is treated as a span extraction problem where slots are defined to occur only with certain intents. this solves the issue of complex categorical modeling but makes slot-filling dependent on an intent detector. therefore, we propose a framework that treats slot-filling as a fully intentagnostic span extraction problem. instead of using rules to constrain the co-occurrence of slots and intents, we identify a slot as either a single span of text or entirely absent. this makes our approach more flexible than prior work; it is fully independent of other system components. regardless, we can explicitly capture turn-by-turn context by adding an input feature denoting whether a slot was requested for this dialog turn (see figure 1 ). pretrained representations. large-scale pretrained models have shown compelling benefits in a plethora of nlp applications @xcite : such models drastically lessen the amount of required task/domain-specific training data with in-domain fine-tuning. this is typically achieved by adding a task-specific output layer to a large pretrained encoder and then finetuning the entire model @xcite . however, this process requires a fine-tuned model for each slot or domain, rather than a single model shared across all slots and domains. this adds a large memory and computational overhead and makes the approach impractical in real-life applications. therefore, we propose to keep the pretrained encoder models fixed in order to emulate a production system where a single encoder model is used. 2 underlying representation model: convert. converrt @xcite ) is a lightweight sentence encoder implemented as a dualencoder network that models the interaction between inputs/contexts and relevant (follow-up) responses. in other words, it performs conversational pretraining based on response selection on the reddit corpus @xcite . it utilizes subword-level tokenization and is very compact and resource-efficient (i.e. it is 59mb in size and can be trained in less than 1 day on 12 gpus) while achieving state-of-the-art performance on conversational tasks @xcite . through pretrained convert representa-tions, we can leverage conversational cues from over 700m conversational turns for the few-shot span extraction task. 3 span convert: final model. we now describe our model architecture, illustrated in figure 2 . our approach builds on established sequence tagging models using conditional random fields (crfs) @xcite . we propose to replace the lstm part of the model with fixed convert embeddings. 4 we take contextualized subword embeddings from convert, giving a sequence of the same length as the subwordtokenized sentence. for sequence tagging, we train a cnn and crf on top of these fixed subword representations. we concatenate three binary features to the subword representations to emphasize important textual characteristics: (1) whether the token is alphanumeric, (2) numeric, or (3) the start of a new word. in addition, we concatenate the character length of the token as another integer feature. to incorporate the requested slots feature, we concatenate a binary feature representing if the slot is requested to each embedding in the sequence. to contextualize the modified embeddings, we apply a dropout layer followed by a series of 1d convolutions of increasing filter width. spans are represented using a sequence of tags, indicating which members of the subword token sequence are in the span. we use a tag representation similar to the iob format annotating the span with a sequence of before, begin, inside and after tags, see figure 2 for an example. the distribution of the tag sequence is modeled with a crf, whose parameters are predicted by a cnn that runs over the contextualized subword embeddings v. at each step t, the cnn outputs a 4 × 4 matrix of transition scores w t and a 4dimensional vector of unary potentials u t . the probability of a predcited tag sequence y is then modeled as: the loss is the negative log-likelihood, equal to minus the sum of the transition scores and unary 3 as we show later in §4, we can also leverage bert-based representations in the same span extraction framework, but our convert-based span extractors result in higher performance. 4 lstms are known to be computationally expensive and require large amounts of resources to obtain any notable success @xcite . by utilizing convert instead, we arrive at a much more lightweight and efficient model. potentials that correspond to the true tag labels, up to a normalization term. the top scoring tag sequences can be computed efficiently using the viterbi algorithm. ## experimental setup new evaluation data set: restaurants-8k. data sets for task-oriented dialog systems typically annotate slots with exclusively categorical labels @xcite . while some data sets such as snips @xcite or atis @xcite following prior work @xcite , we report the f 1 scores for extracting the correct span per user utterance. if the models extract part of the span or a longer span, this is treated as an incorrect span prediction. few-shot scenarios. for both data sets, we measure performance on smaller sets sampled from the full data. we gradually decrease training sets in size whilst maintaining the same test set: this provides insight on performance in low-data regimes. ## results and discussion the results across all slots are summarized in ## conclusion and future work we have introduced span-convert, a light-weight model for dialog slot-filling that approaches the problem as a turn-based span extraction task. the formulation allows the model to effectively leverage representations available from large-scale conversational pretraining. we have shown that, due to pretrained representations, span-convert is especially useful in few-shot learning setups on small data sets. we have also introduced restaurants-8k, a new challenging data set that will hopefully encourage further work on span extraction for dialogue. in future work, we plan to experiment with multi-domain span extraction architectures.