ACLM : A Selective-Denoising based Generative Data Augmentation Approach for Low-Resource Complex NER (2023)

Abstract: Complex Named Entity Recognition (NER) is the task of detecting linguistically complex named entities in low-context text. In this paper, we present ACLM Attention-map aware keyword selection for Conditional Language Model fine-tuning), a novel data augmentation approach based on conditional generation, to address the data scarcity problem in low-resource complex NER. ACLM alleviates the context-entity mismatch issue, a problem existing NER data augmentation techniques suffer from and often generates incoherent augmentations by placing complex named entities in the wrong context. ACLM builds on BART and is optimized on a novel text reconstruction or denoising task - we use selective masking (aided by attention maps) to retain the named entities and certain keywords in the input sentence that provide contextually relevant additional knowledge or hints about the named entities. Compared with other data augmentation strategies, ACLM can generate more diverse and coherent augmentations preserving the true word sense of complex entities in the sentence. We demonstrate the effectiveness of ACLM both qualitatively and quantitatively on monolingual, cross-lingual, and multilingual complex NER across various low-resource settings. ACLM outperforms all our neural baselines by a significant margin (1%-36%). In addition, we demonstrate the application of ACLM to other domains that suffer from data scarcity (e.g., biomedical). In practice, ACLM generates more effective and factual augmentations for these domains than prior methods.

## introduction named entity recognition (ner) is a fundamental task in natural language processing (nlp) that aims to detect various types of named entities (nes) from text. recently, there has been considerable progress in ner using neural learning methods that achieve state-of-the-art (sota) performance @xcite ) on well-known benchmark datasets, including conll 2003 @xcite and ontonotes @xcite . however, these datasets are designed to evaluate the performance on detecting "relatively easy" nes like proper names (e.g., people such as "barack obama," locations such as "new york," or organizations such as "ibm") in well-formed, contextrich text that comes from news articles @xcite . on the other hand, complex ner benchmarks like multiconer @xcite present several contemporary challenges in ner, including short low-context texts with emerging and semantically ambiguous complex entities (e.g., movie names in online comments) that reduce the performance of sota methods previously evaluated only on the existing ner benchmark datasets. our experiments reveal that the performance of the current sota ner method @xcite ) (previously evaluated only on the conll 2003 dataset) drops by 23% when evaluated on multiconer and 31.8% when evaluated on a low-resource setting with just 500 training samples (more details in in the past, researchers have made several attempts at building supervised approaches to detect complex and compositional noun phrase entities in sentences @xcite @xcite . however, the scarcity of annotated training data for building effective systems has always been a challenge. data augmentation has been shown to be an effective solution for low-resource ner @xcite @xcite . in practice, though these systems perform well and generate coherent augmentations on common ner benchmark datasets with easy proper noun nes, they fail to be effective for complex ner, often generating incoherent augmentations. we first argue that certain types of complex nes follow specific linguistic patterns and appear only in specific contexts (examples in appendix 4), and augmentations that do not follow these patterns impede a ner model from learning such patterns effectively. this sometimes also leads to augmentations with context-entity mismatch, further hurting the learning process. for e.g., unlike proper names, substituting complex nes from other sentences in the corpus or replacing them with synonyms @xcite often leads to augmentations where the ne does not fit into the new context (e.g., swapping proper names across sentences might still keep the sentence coherent but swapping the name of a book with a movie (both creative work entity) or the name of a football team with a political party (both group entity) makes it incoherent). fine-tuning pretrained language models (plms), similar to priorwork @xcite @xcite , fail to generate new context around complex nes or completely new nes with the desired linguistic patterns due to low-context sentences and the lack of existing knowledge of such linguistically complex nes (examples in fig. 3 ). this leads to in-coherent augmentations and poses a severe problem in knowledge-intensive tasks like biomedical ner, where non-factual augmentations severely hurt learning. our experiments also reveal that introducing new context patterns around nes proves to be a more effective data augmentation technique for complex ner than diversifying nes (aclm vs. melm in ## background and related work complex ner background: complex ner is a relatively understudied task in the field of nlp. building on insights from @xcite , we discuss key reasons behind high performance on common ner benchmark datasets and try to understand why modern sota ner algorithms do not work well on complex ner benchmarks: complex ner: prior work has mostly focused on solving the entity complexity problem by learning to detect complex nominal entities in sentences @xcite @xcite . researchers have often explored integrating external knowledge in the form of gazetteers for this task. gazetteers have also proven to be effective for low-resource ner @xcite . gemnet @xcite , the current sota system for complex ner, conditionally combines the contextual and gazetteer features using a mixture-of-experts (moe) gating mechanism. however, gazetteers are difficult to build and maintain and prove to be ineffective for complex ner due to their limited entity coverage and the nature of unseen and emerging entities in complex ner. ## methodology in this section, we give an overview of our approach. fig. 1 represents the entire workflow of our aclm data augmentation framework. a sentence is first passed through a fine-tuned xlm-roberta fine-tuned on only gold data to generate the attention map for each token in the sentence. this attention map is then used to selectively mask the sentence and create a template. this template is then used as an input to optimize the model on the text reconstruction objective for fine-tuning aclm: the model is asked to reconstruct the entire original sentence from only the content in the template. while generating augmentations, aclm follows the same template generation process in addition to adding two templates through mixner, which we discuss in detail in section 3.3. training corpus mixner the enemy's infantry was attacked by royal artillerygrp guns. ## conclusion in this paper, we propose aclm, a novel data augmentation framework for low-resource complex ner. aclm is fine-tuned on a novel text reconstruction task and is able to generate diverse augmentations while preserving the nes in the sentence and their original word sense. aclm effectively alleviates the context-entity mismatch problem and generates diverse, coherent, and highquality augmentations that prove to be extremely effective for low-resource complex ner. additionally, we also show that aclm can be used as an effective data augmentation technique for low-resource ner in the domains of medicine and science due to its ability to generate extremely reliable augmentations.