Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning (2023)

Abstract: Unified Sequence Labeling that articulates different sequence labeling problems such as Named Entity Recognition, Relation Extraction, Semantic Role Labeling, etc. in a generalized sequence-to-sequence format opens up the opportunity to make the maximum utilization of large language model knowledge toward structured prediction. Unfortunately, this requires formatting them into specialized augmented format unknown to the base pretrained language model (PLMs) necessitating finetuning to the target format. This significantly bounds its usefulness in data-limited settings where finetuning large models cannot properly generalize to the target format. To address this challenge and leverage PLM knowledge effectively, we propose FISH-DIP, a sample-aware dynamic sparse finetuning strategy that selectively focuses on a fraction of parameters, informed by feedback from highly regressing examples, during the fine-tuning process. By leveraging the dynamism of sparsity, our approach mitigates the impact of well-learned samples and prioritizes underperforming instances for improvement in generalization. Across five tasks of sequence labeling, we demonstrate that FISH-DIP can smoothly optimize the model in low resource settings offering upto 40% performance improvements over full fine-tuning depending on target evaluation settings. Also, compared to in-context learning and other parameter-efficient fine-tuning approaches, FISH-DIP performs comparably or better, notably in extreme low-resource settings. The source code of FISH-DIP will be available at [this URL](https://github.com/psunlpgroup/FISH-DIP)

## introduction sequence labeling tasks such as named entity recognition, relation extraction, semantic role labeling, etc. aim to assign target labels to the components of a sequence from a set of categorical classes. accurate sequence labeling requires learning complex structural dependency, where even state-of-the-art pretrained llms like gpt-3.5turbo fail to show acceptable performance @xcite training samples plm sample loss top-n regressing samples k% parameter sparsity periodic update calculate parameter importance backprop sparse mask 1 ) are first channeled through plms (e.g. t5-large) to calculate the training loss. top-n most regressing samples are chosen for calculating parameter importance, for k% sparsity mask. in backpropagation, this sparsity mask is applied for sparse finetuning. parameter importance is updated dynamically using the feedback from underperforming samples. despite these advantages, fine-tuning plms to accommodate such an "augmented" format for sequence labeling in data-limited low-resource environments is significantly challenging, which has task t input x transformation f t that produces augmented nl target x ner over 900 million us dollars of foreign capital was actually utilized, increasing nearly 40% compared with the same period previous year. [ over 900 million us dollars | monetary ] of foreign capital was actually utilized , increasing [ nearly 40 % | percent ] compared with the same period [ previous year | date ] . relation extraction in the middle adriatic basin, there is evidence of permian volcanism observed on the vis island and as volcanic islands of jabuka and brusnik. relationship between [ jabuka ] and [ adriatic basin ] = located in or next to body of water joint er extraction in 1831 , the 20th president of the united states, james garfield, was born in orange , ohio. in 1831 , the 20th president of the [ united states | location ] , [ james garfield | person | lives @xmath0, ohio ] , was born in [ orange , ohio | location ] . dst [user]: hi, can you help me find a place to stay on the north side?; [agent]: i have 13 hotels on the north side of town, do you have any preferences [user]: yes, are there any expensive ones? i also would like free parking as well. [ belief ] hotel area north, hotel book day not given, hotel book people not given, hotel book stay not given, hotel internet not given, hotel name not given, hotel parking yes, hotel price range expensive, hotel stars not given, hotel type hotel [ belief ] srl the lawyers have renewed their arguments in texas and eight other states where the defense is [ permitted ] under state law. the lawyers have renewed their arguments in [ texas and eight other states | am-loc ] [ where | r-am-loc ] [ the defense | a1 ] is permitted [ under state law | am-loc ]. in the context of data-limited low-resource universal sequence labeling tasks, we identified fixed fisher information sparsity training @xcite as a promising direction. instead of introducing new parameters to train from scratch, it selects and finetunes the most influential parameters from the existing well-trained plms, yielding results on par with full fine-tuning (fine-tuning all the parameters of the model). however, our investigations reveal that a sample-aware dynamic selection of parameter importance can lead to a significant performance boost in few-sample settings. thus, we propose leveraging fisher information for sparse fine-tuning with dynamic importance of parameters (fish-dip), a method of sparse finetuning that dynamically chooses and adjusts a small fraction of the most relevant parameters based on the feedback from the available samples with the highest loss, prioritizing those that exhibit highest deviation from the desired outcome. as shown in figure 1 , augmented natural language training sam-ples are channeled through plms (e.g. t5-large), from which we can calculate losses for each of the samples. then under-performing samples are chosen for calculating parameter importance, which in turn is leveraged to determine a sparsity mask for sparse finetuning. as training progresses, the importance of parameters is updated to accommodate the evolving learning requirements of both the model parameters and training samples. consequently, fish-dip ensures that the fraction of the model parameters chosen is primarily influenced by the most pertinent samples, effectively helping in smoother optimization (figure 4 ) and generalization while sustaining the enhancement in few-shot performance. we conduct comprehensive testing of fish-dip across a range of tasks, including named entity recognition @xcite , relation extraction @xcite , joint entity-relation extraction @xcite , semantic role labeling @xcite , and dialogue state tracking @xcite . our findings reveal that across diverse few-shot settings, fish-dip significantly enhances the few-shot performance of universal sequence labeling tasks. besides, unlike other methods @xcite that require largescale data and pretraining, fish-dip only relies on prudent optimization strategy to augment performance. this unique feature makes it particularly valuable for future applications in low-resource language contexts, where external resources may be extremely scarce. furthermore, fish-dip consistently outperforms other popular parameter efficient finetuning (peft) methods, demonstrating the efficacy of dynamic parameter selection in low resource settings. our contributions are as below: ## fish-dip for unified low-resource sequence labeling in this section, we first discuss how different sequence labeling problems can be transformed into a unified "augmented" format (section 2.1). next, we discuss how fisher information is leveraged to calculate parameter importance (section 2.2). finally, we discuss how fish-dip leverages sampleaware dynamism to boost few-sample performance in sequence labeling (section 2.3). ## experiment setups we describe our tasks and datasets, evaluation settings, and different types of baselines. tasks and datsets to evaluate the efficacy of fish-dip, we cover five different tasks across seven datasets in named entity recognition @xcite , relation extraction @xcite , joint entity relation extraction @xcite , dialogue state tracking @xcite , and semantic role labeling @xcite tasks. for evaluation of ner and srl, we use entity f1-score, while in joint er we use entity and relation f1 scores. finally, in dst, algorithm 1: fish-dip require :low-resource training data x tr , train loss l tr , augmentation function f t , special tokens s, pretrained language model plm, sparsity k%, update steps m, regressing examples n, total training steps t initialize sparse mask @xmath1, if t mod @xmath2. 2 with top n most regressing examples sort all the parameters using fθ and select top k% parameters reset m and set the selected parameters to 1 end backprop and apply sparsity mask m update parameters end we use joint accuracy as evaluation metric. ## results and analysis our results and analysis aim to answer the following research questions: ## conclusion unified sequence labeling poses significant difficulties, particularly when dealing with limited resources necessitating the adoption of a nonconventional augmented format. fish-dip tackles this challenge through dynamic sparse finetuning. it prioritizes underperforming instances and reduces the impact of well-learned samples, optimizing only a fraction of the parameters. across five sequence labeling tasks, fish-dip demonstrated on-par or consistent improvement over all other baselines, particularly in more challenging extremely low-resource scenarios. for future research, exploring the potential of fish-dip in other low-resource/few-shot tasks (e.g. low-resource languages where external resources are extremely scarce) holds promise and warrants further investigation.