CASSI : Contextual and Semantic Structure-based Interpolation Augmentation for Low-Resource NER (2023)

Abstract: While text augmentation methods have been successful in improving performance in the low-resource setting, they suffer from annotation corruption for a token-level task like NER. Moreover, existing methods cannot reliably add context diversity to the dataset, which has been shown to be crucial for low-resource NER. In this work, we propose Contextual and Semantic Structure-based Interpolation (CASSI), a novel augmentation scheme that generates high-quality contextually diverse augmentations while avoiding annotation corruption by structurally combining a pair of semantically similar sentences to generate a new sentence while maintaining semantic correctness and fluency. To accomplish this, we generate candidate augmentations by performing multiple dependency parsing-based exchanges in a pair of semantically similar sentences that are filtered via scoring with a pretrained Masked Language Model and a metric to promote specificity. Experiments show that CASSI consistently outperforms existing methods at multiple low resource levels, in multiple languages, and for noisy and clean text.

## introduction named entity recognition (ner) is the token-level task of classifying terms in text that belong to predefined entities like 'location', 'person', 'organization', etc. it is an important task in nlp that is used in recommendation systems, search and content classification systems for healthcare, academia, news organizations, etc., and as an integral part of other tasks like text summarization @xcite , question answering @xcite , topic modeling @xcite , etc. annotating data for a token-level task like ner can be expensive, often requiring domain expertise. therefore, ner datasets tend to be small, especially for narrow domains and low-resource languages. text augmentation has been a successful method for improving performance in the lowresource setting for various tasks like natural language inference, sentiment analysis, etc. it involves creating additional text using the existing training text while generally preserving the labels foot_1 . typical methods for text augmentation include random distortions such as random deletions, insertions, substitutions, @xcite , word replacement @xcite @xcite , backtranslation @xcite @xcite , paraphrasing @xcite , and label-conditioned text generation through language models @xcite , gans @xcite , vaes @xcite , etc. preserving the token-level entity tags in ner is a challenging task and text augmentation methods often cause annotation corruption, i.e., assignment of incorrect token-level tags. dai and adel 2020 avoid the annotation corruption problem by performing random replacements of the same entity type. however, this creates limited diversity in the dataset and generates noisy sentences since the entity replacements don't take sentence context into account. methods like backtranslation and paraphrasing show limited improvements and require an additional retagging step, which tends to be erroneous @xcite . more effective works attempt to solve the labelpreservation problem for low-resource ner by performing augmentations using a label-conditioned language model. the lm is conditioned on the tags by explicitly including the tags in the training sentences, in a process called label linearization. daga @xcite uses this process to generate new sentences including entity tags. however, these sentences tend to be ungrammatical and disfluent in nature when trained on a small number of examples. melm @xcite uses this process to perform entity replacements through masked generation using a pretrained masked language model, such that the replacements take sentence context into account, but does not introduce new context patterns into the dataset. both of these methods are also able to add new entities to the training set. these methods outperform simpler methods like random substitution, random deletion, etc. however, these methods involve training a language model on limited data and are therefore still prone to annotation corruption, generating noisy training examples. this also severely limits the number of augmentations that can be added. moreover, @xcite show that, in the lowresource setting, decreasing contextual diversity in the training set more significantly impacts ner performance than reducing entity diversity. therefore, contextual diversity is more important than entity regularity knowledge to generalize to unseen examples in the low-resource setting. while transformer language models are pretrained on large corpora, they are not able to sufficiently capture context patterns for ner when fine-tuned on a small number of examples. contextual augmentation has been applied to the token-level task of aspect term extraction by @xcite , who finetune mass @xcite to generate new partial contexts for english text by replacing 'o' tags. however, this requires additional in-domain data. moreover, mass is only pretrained on 4 languages: english, german, french, and romanian. additionally, shi et al. 2021 alter the content while maintaining sentence structure by performing random substitutions of same-label spans for few-shot structured prediction tasks where annotations contain structural information like part-of-speech tagging, dependency parsing, and constituency parsing. in this work, we propose contextual and semantic structure-based interpolation (cassi) as a text augmentation scheme for low-resource ner. with the goal of increasing contextual diversity and avoiding annotation corruption, we use dependency parse trees to identify subtrees containing the subject, object, or complement between pairs of semantically similar sentences. we then generate candidate augmentations by performing exchanges between all pairs of subject subtrees and all pairs of object/complement subtrees between the two sentences. to ensure semantic correctness and fluency, we filter the resulting candidates by scoring them with a language model, similar to rescoring methods used in automatic speech recognition @xcite and neural machine translation @xcite . we leverage an existing pretrained masked language model to avoid training a language model on limited data. furthermore, we take steps to reduce the bias from the language model against specific names and towards shorter sentences and high-frequency generic terms. specifically, we pick the top k scored sentences and further filter them using a simple metric based on jaccard scores between the candidate augmentations and the original sentences. our contributions can be summarized as follows: (1) we present a novel text augmentation scheme that improves context diversity in the dataset while avoiding annotation corruption, (2) we show that our method consistently and significantly outperforms existing methods in multiple languages for monolingual and cross-lingual ner, and (3) we show that our method outperforms existing methods on noisy social media text. ## proposed method our proposed method (shown in figure 1 ) generates augmentations by combining a pair of semantically similar sentences to generate a new sentence while ensuring a high degree of fluency, and contextual and semantic correctness. for every sentence in the training set, we first rank the remaining sentences by sentence similarity (figure 1a ; section 2.1). next, we generate multiple candidate augmentations by performing structure-based exchanges between the given sentence and a selected semantically similar sentence using their dependency parse trees (figure 1b ; section 2.2). we then filter out a subset of the candidate augmentations by scoring them with a pretrained masked language model. we select the final augmentation from this subset using a simple metric to promote specificity and diversity (figure 1c ; section 2.3). ## conclusion in this work, we propose cassi as a text augmentation scheme for low-resource ner. cassi produces contextually diverse sentences while preserving entity tags by structurally combining a pair of semantically similar sentences through the generation of candidate combinations that are filtered for semantic correctness and specificity. it leverages dependency parsers for structural information and a pretrained mlm for scoring. through extensive experiments, we show that it outperforms baseline methods for multiple languages in monolingual and cross-lingual settings. furthermore, we show that it is also able to outperform baseline methods under noisy text conditions where dependency parsing is likely to be unreliable.