Augmented Natural Language for Generative Sequence Labeling (2020)

Abstract: We propose a generative framework for joint sequence labeling and sentence-level classification. Our model performs multiple sequence labeling tasks at once using a single, shared natural language output space. Unlike prior discriminative methods, our model naturally incorporates label semantics and shares knowledge across tasks. Our framework general purpose, performing well on few-shot learning, low resource, and high resource tasks. We demonstrate these advantages on popular named entity recognition, slot labeling, and intent classification benchmarks. We set a new state-of-the-art for few-shot slot labeling, improving substantially upon the previous 5-shot (75.0% to 90.9%) and 1-shot (70.4% to 81.0%) state-of-the-art results. Furthermore, our model generates large improvements (46.27% to 63.83%) in low resource slot labeling over a BERT baseline by incorporating label semantics. We also maintain competitive results on high resource tasks, performing within two points of the state-of-the-art on all tasks and setting a new state-of-the-art on the SNIPS dataset.

## introduction transfer learning has been the pinnacle of recent successes in natural language processing. large pre-trained language models are powerful backbones that can be fine-tuned for different tasks to achieve state-of-the-art performance in wide-raging applications @xcite @xcite @xcite . while these models can be adapted to perform many tasks, each task is often associated to its own output space, which limits the ability to perform multiple tasks at the same time. for instance, a sentiment analysis model is normally a binary classifier that decides between class labels "positive" and "negative", while a multi-class entailment system classifies each input as "entail", "contradict", or "neither". this approach presents difficulty in knowledge sharing among tasks. that is, to train the model for a new task, the top-layer classifier is replaced with a new one that corresponds to novel classes. the class types are specified implicitly through different indices in the new classifier, which contain no prior information about the label meanings. this discriminative approach does not incorporate label name semantics and often requires a non-trivial amount of examples to train @xcite . while this transfer learning approach has been immensely successful, a more efficient approach should incorporate prior knowledge when possible. conditional generative modeling is a natural way to incorporate prior information and encode the output of multiple tasks in a shared predictive space. recent work by @xcite built a model called t5 to perform multiple tasks at once using natural language as its output. the model differentiates tasks by using prefixes in its input such as "classify sentiment:", "summarize:", or "translate from english to german:" and classify each input by generating natural words such as "positive" for sentiment classification or "this article describes ..." for summarization. however, the appropriate output format for important sequence labeling applications in nlp, such as named entity recognition (ner) and slot labeling (sl) is not immediately clear. in this work, we propose an augmented natural language format for sequence labeling tasks. our format locally tags words within the sentence (figure 1 ) and is easily extensible to sentence-level classification tasks, such as intent classification (ic). our highlighted contributions and main findings are as follows: we propose a new formulation for this generation task such that, given the input sequence `, our method generates output o in augmented natural language. the augmented output o repeats the original input sequence `with additional markers that indicate the token-spans and their associated labels. more specifically, we use the format [ `j, . . . , `j+t í l ] to indicate that the token sequence `j, . . . , `j+t is labeled as l. fig. 1 depicts the proposed format and its equivalent canonical bio format for the same input sentence. the conversion between the bio format and our augmented natural language format is invertible without any information loss. this is crucial so that the generated output from model prediction can be converted back for comparison without uncertainty. there are other formats which can encapsulate all the tagging information but are not invertible. for instance, outputting only the token spans of interest with tagging patterns [ `j, . . . , `j+t í l ] without repeating the entire sentence results in the invertibility breaking down when there are duplicate token spans with different labels. we discuss this further in the appendix section a.3. ## related work sequence to sequence learning has various applications including machine translation @xcite , language modeling @xcite , abstractive summarization @xcite , generative question answering @xcite , to name a few. however, the sequence-to-sequence framework is often not a method of choice when it comes to sequence labeling. most models for sequence labeling use the token-level classification framework, where the model predicts a label for each element in the input sequence @xcite @xcite . while select prior work adopts the sequence-to-sequence method for sequence labeling @xcite , this approach is not widely in use due to the difficulty of fixing the output length, output space, and alignment with the original sequence. multi-task and multi-domain learning often benefit sequence labeling performance @xcite . the archetypal multi-task setup jointly trains on a target dataset and one or more auxiliary datasets. in the cross lingual setting, these auxiliary datasets typically represent high-resource languages @xcite . while in a monolingual scenario, the auxiliary datasets commonly represent similar, highresource tasks. examples of similar multi-task pairs include ner and slot labeling @xcite as well as dialogue state tracking and language understanding @xcite . a recent series of works frame natural language processing tasks, such as translation, question answering, and sentence classification, as conditional sequence generation problems @xcite @xcite . by unifying the model output space across tasks to consist of natural language symbols, these approaches reduce the gap between language model pre-training tasks and downstream tasks. moreover, this framework allows acquisition of new tasks without any architectural change. the gpt-3 model (brown @xcite 98.99 96.89 joint bert @xcite 98.60 97.50 97.00 96.10 elmo+bilstm @xcite 99.29 97.42 93.90 95.62 ner cloze-cnn @xcite 93.50 bert-mrc @xcite 93.04 91.11 bert-mrc + dsc @xcite 93.33 92.07 bert base @xcite 92 the conditional sequence generation framework makes it easy to incorporate label semantics, in the form of label names such as departure city, example values like san francisco, and descriptions like "the city from which the user would like to depart on the airline". label semantics provide contextual signals that can improve model performance in multi-task and low-resource scenarios. multiple works show that conditioning input representations on slot description embeddings improves multidomain slot labeling performance @xcite . embedding example slot values in addition to slot descriptions yields further improvements in zero-shot slot labeling @xcite . in contrast to our work, these approaches train slot description and slot value embedding matrices, whereas our framework can incorporate these signals as natural language input without changing the network architecture. ## discussion and future work our experiments consistently show that the generation framework is suitable for sequence labeling and sets a new record for few-shot learning. our model adapts to new tasks efficiently with limited samples, while incorporating the label semantics expressed in natural words. this is akin to how humans learn. for instance, we do not learn the concept of "person" from scratch in a new task, but have prior knowledge that "person" likely corresponds to names, and refine this concept through observations. the natural language output space allows us to retain the knowledge from previous tasks through shared embeddings, unlike the tokenlevel model which needs new classifiers for novel tasks, resulting in a broken chain of knowledge. our approach naturally lends itself to life-long learning. the unified input-output format allows the model to incorporate new data from any domain. moreover, it has the characteristics of a single, lifelong learning model that works well on many levels of data, unlike other approaches that only perform well on few-shot or high-resource tasks. our simple yet effective approach is also easily extensible to other applications such as multi-label classification, or structured prediction via nested tagging patterns.