Simple Semantic-based Data Augmentation for Named Entity Recognition in Biomedical Texts (2022)

Abstract: Data augmentation is important in addressing data sparsity and low resources in NLP. Unlike data augmentation for other tasks such as sentence-level and sentence-pair ones, data augmentation for named entity recognition (NER) requires preserving the semantic of entities. To that end, in this paper we propose a simple semantic-based data augmentation method for biomedical NER. Our method leverages semantic information from pre-trained language models for both entity-level and sentence-level. Experimental results on two datasets: i2b2-2010 (English) and VietBioNER (Vietnamese) showed that the proposed method could improve NER performance.

## introduction in machine learning and especially deep learning approaches, performance of the trained models is often proportional to the size of the training data. consequently, for a model to achieve acceptable performance, we need a certain amount of labelled data. this would be an issue for low-resource domain and low-resource languages since annotating labelled data is time-consuming and expensive. to address the issue, data augmentation has been proposed to increase the variety of training data without directly collecting or annotating additional data @xcite . intuitively, data augmentation for named entity recognition (ner) task is more difficult to perform than for other sentence-level and sentencepair tasks. simple operations used to augment a sentence such as token swap, token deletion, and token insertion @xcite may not work well in the case of ner, especially in the biomedical domain. one of the reasons is that a named entity can be composed by multiple tokens and we have to preserve the semantic of entities after applying those operations. for example, consider the following sentence from the i2b2-2010 corpus @xcite with its entities: she can be given prn @xcite t reatment for [weight gain] p roblem or [shortness of breath] p roblem . if we randomly swap the 'lasix' token with 'weight', the sentence is not semantically correct. similarly, when the 'weight' token is deleted, the remaining 'gain' token is no longer suitable for an entity of problem. for the insertion operation, if we randomly insert a token into the sentence, the semantic of the sentence will be changed and we will not be able to assign a suitable entity label for it. as a result, it is necessary to have different augmentation methods specified for ner. there are several model-based data augmentation methods for ner. @xcite proposed local additivity-based data augmentation (lada) that can create virtual samples using interpolation technique. their exeperimental results showed that lada could help to produce state-ofthe-art (sota) on two ner benchmarks including conll 2013 @xcite and germeval 2014 @xcite . meanwhile, @xcite took advantages of the rich semantic information in pre-trained word embeddings to create a semantic augmentation module for ner models. they also reported sota performance on some social media corpora. obviously, model-based methods can help to improve ner performance, but they are often complicated and difficult to implement. in contrast, rulebased methods are simpler and more intepretable than model-based ones, but still effective. @xcite adjusted simple operations such as replacement and shuffle to preserve the semantic of both entities and sentences. specifically, they proposed synonym replacement (sr) and mention replacement (mr). sr replaces a word in a sentence with a word of the same semantics taken from wordnet. mr replaces the whole entity with another random entity in the same entity type based on the training data; the replacement action for each entity is decided based on the binomial distribution. as a result, they could improve the ner performance on both mascip and i2b2-2010 corpora. we find two limitations in dai and adel (2020)'s approach. firstly, although the sr operation takes into account the semantic aspect of tokens, it does not consider the semantic at the entity level. secondly, the mr operation is performed on the entity level randomly, which may cause semantically incorrect sentences. we hypothesise that if we somehow control the semantics in entity and sentence levels in augmentation operations, we could create a meaningful augmented data, hence improving the ner performance. to that end, we propose semantic neighbour replacement (snr), a simple data augmentation method for biomedical ner that considers the semantic aspects of both entity and sentence levels. specifically, at the entity level, unlike mr (dai and adel, 2020), we only replace a source entity with a target one if the target entity is in the same entity type and semantically related to the source one. at the sentence level, we only retain sentences that are semantically related to the original sentence. the semantically related entities and sentences are calculated by using pre-trained language models. we conducted experiments on two biomedical datasets: i2b2-2010 @xcite )an english corpus of clinical records and viet-bioner @xcite -a vietnamese corpus of biomedical texts. experimental results indicate that using snr, we can improve ner performance on low-resource settings as well as on full training data. in particular, the f1-scores were increased by 0.52% for i2b2-2010 and 1.3% for vietbioner. ## methodology the core idea of snr is to replace entities and to control augmented sentences based on semantic similarity. the method can be divided into three consecutive phases: semantic neighbour extraction, entity replacement, and sentence evaluation. semantic neighbour extraction: initially, we perform feature extraction for entities using pretrained language models. an entity embedding is calculated by taking an average of word embeddings in it. next, we generate sets of semantic neighbors based on cosine similarity. an entity is a semantic neighbor to another entity if both of them belong to the same entity type and have a cosine similarity greater than or equal to a threshold α. entity replacement: during this phase, we generate new sentences by replacing an entity with another random entity in its semantic neighbor set. for each entity type, we just randomly replace one entity of that type in a sentence. as a result, we obtain a set of augmented sentences from original ones. sentence evaluation: augmented sentences generated in the previous phase are probably semantically incorrect, which may affect the training process. to alleviate the issue, we perform an automatic evaluation to remove augmented sentences that are semantically different from their original sentences. to that end, we firstly represent both original and augmented sentences as vectors by using a pre-trained sentence-level language model. we then use cosine similarity to estimate the semantic similarity between two sentences. if the cosine similarity of an augmented sentence and its original sentence is less than a threshold θ, the augmented sentence will be discarded. in this paper, the two parameters α and θ will be in ranges of @xcite . the larger the α, the greater the semantic similarity between entities, but the smaller the number of neighbours. the θ parameter represents the degree of rigour in the automatic evaluation phase. when θ approximates to 1, only sentences that are very close to the meaning of the original sentence are retained. we therefore can keep only a few of the augmented sentences. in contrast, we can keep more sentences as θ approximates to 0. when θ is set to 0, the sentence evaluation phase will be disabled. at this point, we do not discard any augmented sentences from the second phase. we can fine-tune both α and θ to generate suitable augmented data. ## conclusion in this paper, we proposed a semantic-based data augmentation method for the named entity recognition task in the biomedical domain. our method, namely semantic neighbour replacement (snr), simply generates more training sentences based on semantics of entity and sentence. experiments on simulated low-resource settings show that using the proposed method, we can improve f1 score in both english (i2b2-2010) and vietnamese (viet-bioner) corpora, even on the full training setting. such results again confirm the importance of semantics in data augmentation. we believe that snr can be applied to other domains and other languages as long as we have corresponding pretrained language models. similar to previous work, our proposed method only augments in-domain data. therefore, a followup work would be to study cross-domain augmentation method @xcite , in which we can leverage rich-resource data to enrich lowresource ones.