Improving Pre-trained Language Model Sensitivity via Mask Specific losses: A case study on Biomedical NER (2024)

Abstract: Adapting language models (LMs) to novel domains is often achieved through fine-tuning a pre-trained LM (PLM) on domain-specific data. Fine-tuning introduces new knowledge into an LM, enabling it to comprehend and efficiently perform a target domain task. Fine-tuning can however be inadvertently insensitive if it ignores the wide array of disparities (e.g in word meaning) between source and target domains. For instance, words such as chronic and pressure may be treated lightly in social conversations, however, clinically, these words are usually an expression of concern. To address insensitive fine-tuning, we propose Mask Specific Language Modeling (MSLM), an approach that efficiently acquires target domain knowledge by appropriately weighting the importance of domain-specific terms (DS-terms) during fine-tuning. MSLM jointly masks DS-terms and generic words, then learns mask-specific losses by ensuring LMs incur larger penalties for inaccurately predicting DS-terms compared to generic words. Results of our analysis show that MSLM improves LMs sensitivity and detection of DS-terms. We empirically show that an optimal masking rate not only depends on the LM, but also on the dataset and the length of sequences. Our proposed masking strategy outperforms advanced masking strategies such as span- and PMI-based masking.

## introduction fine-tuning is the prevailing practice for adapting an lm to a new domain. a plethora of research works ranging from task-generalization @xcite @xcite , to few-shot learning @xcite to in-context tuning @xcite all unanimously credit fine-tuning for the state-ofthe-art results across a diverse set of nlp tasks. despite its remarkable strides, fine-tuning has been reasonably criticised for its instability and brittleness by a few pockets of nlp researchers (mos- ## mask-specific language modeling in designing our approach, we draw lessons from two prior tested and proven phenomena: (1) mlms are effective in learning representations for subtokens, words @xcite , phrases @xcite and spans @xcite ; and (2) high prediction rates (proportion of tokens to be predicted) substantively affect optimization, i.e. they increase training signals, which subsequently boost performance @xcite . we refer to these two phenomena respectively as the mlm-effect and the highprediction-effect in the remainder of this paper. ## comparisons with prior masking strategies besides our proposed masking strategy (i.e. joint elm-blm), there are various other advanced masking strategies such as pmi-masking (pmi) @xcite and random-span masking (span) @xcite . with pmi, spans of co-occurring words (2-4) (a.k.a collocations) are identified, ranked based on pmi scores computed using the pmi measure proposed by @xcite and stored in a vocabulary. the ranked spans discovered in an input sequence are masked. in the span approach, spans of varying lengths (2-4) are arbitrarily selected and masked. in both approaches, the total masking budget (number of tokens to mask) is maintained to avoid biasing the comparative analysis. extended details of how we implement span and pmi are in appendix e. in figure 5 , we directly replace joint elm-blm with either span or pmi and vary the total masking budget while maintaining the optimal budget for joint elm-blm. for instance, we keep @xmath0.75 and @xmath1.0 for joint elm-blm, when evaluating mslm_biobert on bc2gm dataset because they are the optimal rates. however, we vary the rates for both pmi and span across the values in the set of blm rates established in §4.1. we observe that joint elm-blm outperforms other strategies across all experiments. pmi produces majority of the second best results despite span masking being quite competitive. we attribute pmi's performance to the fact that the pmi's vocabulary from which spans to mask are drawn has a high concentration (> 50%) of ds-terms (details in appendix e.3)), which effectively makes it similar to elm that directly masks ds-terms. as noticed earlier in §4.1, masking ds-terms is highy effective even with no blm masking (i.e. @xmath2.0). we also observe the slight drop in performance as the masking rate increases across pmi and span, which further confirms the fact that lms are likely to struggle when decoding highly corrupted sequences @xcite . masking: the originally proposed masking approach that involved replacing a percentage of tokens at random (token masking) with [mask] tokens @xcite has been modified in recent works to improve mlm. @xcite and @xcite mask named entity spans (entity masking), @xcite mask random spans of tokens (span masking) and @xcite mask groups of co-occurring words (pmi masking). with the exception of pmi, our proposed joint elm-blm masking approach aligns well with all recent masking modifications. it simultaneously masks disjoint sets of random tokens and entity spans. targeting multiple units in a sentence makes it greedier than prior works, however, we emphasize mask rate tuning and upholding a masking budget to achieve optimal performance. ## conclusion we considered the problem of dsft aiming to improve an lm's sensitivity (i.e. awareness of) towards ds-terms. we proposed mslm, an approach that jointly masks ds-terms and random words, while conditioning the lm to larger penalties during optimisation for incorrect predictions of ds-terms. using the biomedical domain as a testbed, the performed experiments reveal improvements mslm makes over vanilla fine-tuning in exact ds-term match detection. mslm's efficiency is proven when models achieve higher ner f1 scores in a much shorter training time. we substantiate the recent narrative, dismissing 15% as a universally optimal rate in mlm @xcite , by proving that optimal performance is influenced by varying masking rates and length of sequences. the joint elm-blm masking strategy we propose outperforms advanced masking methods. although we focus on biomedical ner, our proposed mslm approach can be be adapted for dsft for other domains. the positive impact of our proposed masking method motivates us to investigate its effectiveness during pre-training of mlms in future work.