Span Fine-tuning for Pre-trained Language Models (2021)

Abstract: Pre-trained language models (PrLM) have to carefully manage input units when training on a very large text with a vocabulary consisting of millions of words. Previous works have shown that incorporating span-level information over consecutive words in pre-training could further improve the performance of PrLMs. However, given that span-level clues are introduced and fixed in pre-training, previous methods are time-consuming and lack of flexibility. To alleviate the inconvenience, this paper presents a novel span fine-tuning method for PrLMs, which facilitates the span setting to be adaptively determined by specific downstream tasks during the fine-tuning phase. In detail, any sentences processed by the PrLM will be segmented into multiple spans according to a pre-sampled dictionary. Then the segmentation information will be sent through a hierarchical CNN module together with the representation outputs of the PrLM and ultimately generate a span-enhanced representation. Experiments on GLUE benchmark show that the proposed span fine-tuning method significantly enhances the PrLM, and at the same time, offer more flexibility in an efficient way.

## introduction pre-trained language models (prlm), including electra @xcite , roberta @xcite , and bert @xcite , have demonstrated strong performance in downstream tasks @xcite . leveraging a self-supervised training on large text corpora, these * corresponding author. this paper was partially supported by national key research and development program of china (no. 2017yfb0304100), key projects of national natural science foundation of china (u1836222 and 61733011). this work was supported by huawei noah's ask lab models are able to provide contextualized representations in a more efficient way. for instance, bert uses masked language modeling and nest sentence prediction as pre-training objects and is trained on a corpus of 3.3 billion words. in order to be adaptive for a wider range of applications, prlms usually generate sub-token-level representations (words or subwords) as basic linguistic units. for downstream tasks such as natural language understanding (nlu), span-level representations, e.g. phrases and name entities, are also important. previous works manifest that by changing pre-training objectives, prlms' ability to capture span-level information can be strengthened to some extent. for example, base on bert, span-bert @xcite focuses on masking and predicting text spans, instead of sub-token-level information for pre-training. entity-level masking is used as a pre-training strategy by ernie models @xcite . the upper mentioned methods prove that the introduction of span-level information in pre-training to be effective for different nlu tasks. however, the requirements for span-level information of various nlu tasks differs a lot from case to case. the methods of introducing span-level information in pre-training phase, proposed by previous works, do not fit into the requirements and cannot improve the performance for all nlu tasks. for instance, ernie models @xcite perform remarkably well in relation classification, while underperforms bert in language inference tasks, such as mnli @xcite . therefore, it is imperative to develop a strategy to incorporate span-level information into prlms in a more flexible and universally adaptive way. this paper proposes a novel approach, span fine-tuning (sf), to leverage span-level information in fine-tuning phase and therefore formulate a task-specific strategy. compared to existing works, our approach requires less time and computing resources, and is more adaptive to various nlu tasks. in order to maximize the value and contribution of span-level information, in additional to the sub-token-level representation generated by bert, span fine-tuning also applies a computationally motivated segmentation to further improve the overall experience. although various techniques, such as dependency parsing @xcite or semantic role labeling (srl) @xcite , have been used as auxiliary tools for sentence segmentation, these methods demand extra parsing procedure, which increase complexities in actual practice. span fine-tuning first leverages a presampled n-gram dictionary to segment input sentences into spans. then, the sub-token-level representations within the same span are combined to generate a span-level representation. finally, the span-level representations are merged with subtoken-representations into a sentence-level representation. in this way, the sentence-level representation is able to contain and maximize the utilization of both sub-token-level and span-level information. experiments are conducted on the glue benchmark @xcite , which includes many nlu tasks, such as text classification, semantic similarity, and natural language inference. empirical results demonstrate that span fine-tuning is able to further improve the performance of different prlms, including bert @xcite , roberta @xcite and spanbert @xcite . the result of the experiments with spanbert indicates that span fine-tuning leverages span-level information differently compared to prlms pre-trained with span-level information, which shows the distinguishness of our approach. it is also verified in ablation studies and analysis that span fine-tuning is essential for further performance improvement for prlms. ## methodology figure 1 demonstrates the overall framework of span fine-tuning, which is essentially uses bert as a foundation and incorporates segmentation as an auxiliary tool. the figure does not exhaustively depict the details of bert, given the model is relatively popular and ubiquitous. further information on bert is available in @xcite . in span fine-tuning, an input sentence is divided into sub-word-level tokens and then sent to bert to generate sub-token-level representations. at the same time, the input is segmented into spans based on n-gram statistics. by combining the segmentation information with sub-token-level representations generated by bert, we divided the repre-sentation into several spans. then, the spans are sent through a hierarchical cnn module to obtain a span-level information enhanced representation. finally, the sub-token-level representation of [cls] token generated by bert and the span-level information enhanced representation are concatenated and form a final representation, which maximized the value of both sub-token-level and span-level information for nlu tasks. ## conclusion this paper proposes span fine-tuning that maximize the advantages of flexible span-level information in fine-tuning with sub-token-level representations generated by prlms. leveraging a reasonable segmentation provided by a pre-sampled n-gram dictionary, span fine-tuning can further enhance the performance of prlms on various downstream tasks. compared with previous span pre-training methods, our span fine-tuning remains competitive for the following reasons: task-adaptive for methods that incorporate span-level information in pre-training, the utilization of span-level information is unlikely easily adjusted for every downstream task as span pretraining has been fixed after tremendous computational cost. in our method, the extra module designed to incorporate span-level information is trained during the fine-tuning, resulting in a more dynamically adaptation to different downstream tasks. flexible to prlms our approach can be generally applied to various prlms including roberta and spanbert. novelty our approach can further improve the performance of prlms pre-trained with span-level information (e.g. spanbert). such result implies that we our method utilizes the span-level