An Enhanced Span-based Decomposition Method for Few-Shot Sequence Labeling (2022)

Abstract: Few-Shot Sequence Labeling (FSSL) is a canonical paradigm for the tagging models, e.g., named entity recognition and slot filling, to generalize on an emerging, resource-scarce domain. Recently, the metric-based meta-learning framework has been recognized as a promising approach for FSSL. However, most prior works assign a label to each token based on the token-level similarities, which ignores the integrality of named entities or slots. To this end, in this paper, we propose ESD, an Enhanced Span-based Decomposition method for FSSL. ESD formulates FSSL as a span-level matching problem between test query and supporting instances. Specifically, ESD decomposes the span matching problem into a series of span-level procedures, mainly including enhanced span representation, class prototype aggregation and span conflicts resolution. Extensive experiments show that ESD achieves the new state-of-the-art results on two popular FSSL benchmarks, FewNERD and SNIPS, and is proven to be more robust in the noisy and nested tagging scenarios.

## introduction many natural language understanding tasks can be formulated as sequence labeling tasks, such as named entity recognition (ner) and slot filling (sf) tasks. most prior works on sequence labeling follow the supervised learning paradigm, which requires large-scale annotated data and is limited to pre-defined classes. in order to generalize on the emerging, resource-scare domains, few-shot sequence labeling (fssl) has been proposed @xcite . in fssl, the model (typically trained on the source domain however, for the mbml methods, selecting the appropriate metric granularity would be fundamental to the success. we argue that prior works that focus on modeling the token-level similarity are sub-optimal in fssl: 1) they ignore the integrality of named entities or dialog slots that are composed of a text span instead of a single token. 2) in fssl, the conditional random fields (crf) is an important component for the token-level sequence labeling models @xcite , but the transition probability between classes in the target domain task can not be sufficiently learned with very few examples @xcite . the previous methods resort to estimate the values with the abundant source domain data, which may suffer from domain shift problem @xcite . to overcome these drawbacks of the token-level models, in this paper, we propose esd, an enhanced span-based decomposition model that formulates fssl as a span-level matching problem between test query and support set instances. specifically, esd decomposes the span matching problem into three main subsequent span-level procedures. 1) span representation. we find the span representation can be enhanced by information from other spans in the same sentence and the interactions between query and support set. thus we propose a span-enhancing module to reinforce the span representation by intra-span and interspan interactions. 2) span prototype aggregation. mbml methods usually aggregate the span vectors that belongs to the same class in the support set to form the class prototype representation. among all class prototypes, the o-type serves as negative examples and covers all the miscellaneous spans that are not entities, which poses new challenges to identify the entity boundaries. to this end, we propose a span-prototypical module to divide o-type spans into 3 sub-types to distinguish the miscellaneous semantics by their relative position with recognized entities, together with a dynamically aggregation mechanism to form the specific prototype representation for each query span. 3) span conflicts resolution. in the span-level matching paradigm, the predicted spans may conflict with each other. for example, a model may annotate both "albert einstein" and "albert" as "per". therefore, we propose a span refining module that incorporates the soft-nms @xcite algorithm into the beam search to alleviate this conflict problem. with beam soft-nms, esd can also handle nested tagging cases without any extra training, which previous methods are incapable of. we summarize our contribution as follows: (1) we propose esd, an enhanced span-based decomposition model, which formulates fssl as a spanlevel matching problem. (2) we decompose the span matching problem into 3 main subsequent procedures, which firstly produce enhanced span representation, then distinguish miscellaneous semantics of o-types, and achieve the specific prototype representation for each query, and finally resolve span conflicts . (3) extensive experiments show that esd achieves new state-of-the-art performance on both few-shot ner and slot filling benchmarks and that esd is more robust than other methods in nested and noisy scenarios. ## task formulation we define a sentence as x and its label as @xmath0, y i )} m @xmath1, where s i is a span of x (e.g., 'steve jobs'), y i is the label of s i ( e.g., 'per') and m is the number of spans in the x. following the previous fssl setting @xcite , we have data in source domain d source and target domain d target , and models are evaluated on tasks from d target . meta-learning based fssl has two stages, meta-training and meta-testing. in the meta-training stage, the model is trained on tasks sampled from d source to mimic the test situation, and in the meta-testing stage, the model is evaluated on test tasks. a task is defined as @xmath2, q}, consisting of a support set @xmath3, y i )} i @xmath4, and a query sentence @xmath5. s includes n types of entities or slots (n -way), and each type has k annotated examples (k-shot). for spans that do not belong to the n types, e.g., 'studied at' in figure 1 , we set their label to o. the types except o in each test task are guaranteed to not exist in the training tasks. given a task @xmath6, q}, the model needs to assign a label to each span in the query sentence q based on s. ## methodology our esd formulates fssl as a span-level matching problem and decomposes it into a series of span-related procedures for a better span matching. figure 2 illustrates the architecture of esd. ## analysis 6.1 ablation study to illustrate the effect of our proposed mechanisms, we conduct ablation studies by removing one component of esd at a time. ## conclusion in this paper, we propose esd, an enhanced spanbased decomposition model for few-shot sequence labeling (fssl). to overcome the drawbacks of previous token-level methods, esd formulates fssl as a span-level matching problem, and decomposes it into a series of span-related procedures, mainly including span representation, class prototype aggregation and span conflicts resolution for a better span matching. extensive experiments show that esd achieves the state-of-the-art performance on two popular few-shot sequence labeling benchmarks and that esd is more robust than previous models in the noisy and nested situation. partition the sentences in genia into two groups, g 1 and g 2 . g 1 consists of sentences with nested entities (4, 744 sentences in total), and g 2 consists of sentences without nested entities (11, 924 sentences in total). we utilize sentences in g 1 and g 2 to construct the query and support set of the genia task, respectively. our fewnerdnested contains 2000 5-way 5∼10-shot test tasks in total, where r nested percent tasks are from ge-nia, and the remained tasks are from fewnerd-intra. we use ace05 to construct the validation set. ace05 is a widely used named entity recognition dataset, which contains 7 coarse-grained entity types, 'fac', 'per', 'loc', 'veh', 'gpe', 'gpe', 'wea' and 'org'. the 'loc', 'per', 'org' and 'fac' are not included in the training set of fewnerd-intra, and therefore we use them (28 fine-grained entity types in total) and sample 1000 nested tasks as the validation dataset to tune the k, δ and u of bsnms. the search scope is included in the