When and how to paraphrase for named entity recognition? (2023)

Abstract: While paraphrasing is a promising approach for data augmentation in classification tasks, its effect on named entity recognition (NER) is not investigated systematically due to the difficulty of span-level label preservation. In this paper, we utilize simple strategies to annotate entity spans in generations and compare established and novel methods of paraphrasing in NLP such as back translation, specialized encoder-decoder models such as Pegasus, and GPT-3 variants for their effectiveness in improving downstream performance for NER across different levels of gold annotations and paraphrasing strength on 5 datasets. We thoroughly explore the influence of paraphrasers, and dynamics between paraphrasing strength and gold dataset size on the NER performance with visualizations and statistical testing. We find that the choice of the paraphraser greatly impacts NER performance, with one of the larger GPT-3 variants exceedingly capable of generating high quality paraphrases, yielding statistically significant improvements in NER performance with increasing paraphrasing strength, while other paraphrasers show more mixed results. Additionally, inline auto annotations generated by larger GPT-3 are strictly better than heuristic based annotations. We also find diminishing benefits of paraphrasing as gold annotations increase for most datasets. Furthermore, while most paraphrasers promote entity memorization in NER, the proposed GPT-3 configuration performs most favorably among the compared paraphrasers when tested on unseen entities, with memorization reducing further with paraphrasing strength. Finally, we explore mention replacement using GPT-3, which provides additional benefits over base paraphrasing for specific datasets.

## introduction named entity recognition (ner) seeks to extract entity mentions (e.g., shakespeare, warwickshire) from a text (shakespeare was born and raised in warwickshire) for predefined categories of interest (such as people and locations). it is a critical component underpinning many industrial pipelines for a variety of downstream natural language processing applications such as search, recommendation, and virtual assistant systems. however, in real-world applications, there is often a scarcity of labeled data for training advanced deep neural models because span-level ner annotations are costly, and domain expertise may be needed to annotate data from domains such as finance, biomedical sciences, etc. data augmentation is often used as an alternative to address the data scarcity issue in many nlp tasks (see an nlp data augmentation survey by @xcite ). however, data augmentation for ner imposes additional challenges because ner requires token/span level label preservation. therefore, most existing works on ner data augmentation primarily focus on local replacement for entity mentions @xcite @xcite as well as context words @xcite . the replacements can be other mentions with the same labels (dai and adel, 2020), synonyms from an external lexical resource such as wordnet @xcite , or tokens generated by the pretrained language models such as bert via masked token task @xcite @xcite . however, to enhance the reliability of masked token prediction, the language model usually needs to be fine-tuned on the ner training data and label information is often inserted close to the [mask]s @xcite , which requires a decent amount of labeled training data. a recent study by @xcite trained a sequence generator to synthesize sentences with inline ner annotations that can create novel ner training examples beyond local modifications but requires sufficient ner labeled examples for training the generator. this work primarily focuses on the less-studied data augmentation method for ner -paraphrasing -which has the potential to introduce structural and lexical replacement and does not assume many labeled examples. specifically, we compare established, and novel paraphrasing methods and propose simple ways to preserve span-level labels. unlike most existing studies that focus on the influence of the amount of gold data only, we systematically investigate the effect of different levels of paraphrasing on downstream performance, at different levels of gold annotations across 5 datasets. we investigate the quality of paraphrases from 6 different systems as augmentation data, as well as stand alone training data for ner. we further examine the entity memorization via the performance change on unseen mentions for each entity and address the issue with mention replacement. we find paraphrasing to be generally effective in low data regimes for most paraphrasers. however, the choice of paraphraser affects the magnitude, and direction of the change in performance across all levels of gold data. we find the use of llms to generate inline annotations 1 while paraphrasing to be superior to simpler heuristics, and gpt-3 davinci variant with inline annotations to be a generally superior choice across datasets for paraphrasing. in addition, our entity level analysis shows that entity classes with low support (number of mentions) or low number proportion benefit more from paraphrasing. we then investigate whether there is an indications of entity memorization with increasing paraphrasing strength, and find that gpt-3 davinci variant with inline annotations is more robust against entity memorization compared to other paraphrasers. we further reduce memorization in some datasets by introducing mention replacement based on gpt-3 davinci in the paraphrasing pipeline. twitter, biomedical research and search; while also having a diverse set of entity types (word phrases, alphanumeric, datetime, alphabetical etc.). we choose 5 datasets based on the above principles: ontonotes5 @xcite , tweebank @xcite , @xcite , mit restaurant ner dataset (mit-r) @xcite , biocreative v cdr (bc5cdr) @xcite . pre-formatted versions of all datasets are sourced from the tner project @xcite on huggingface datasets @xcite (see appendix a.16). datasets such as wnut also have rare entities by design, allowing us to probe robustness against entity memorization. ## future work while our work proposes a paraphrasing pipeline that performs consistently better than established paraphrasing pipelines in ner, we expect further benefits to come from more exhaustive tuning of prompts used to generate paraphrases. another potential direction to improve downstream performance is to explore better (than random) sampling strategy for paraphrases (based on entity density, entity recall, or other metrics). ## conclusion we study the effect of six paraphrasing systems on downstream ner performance across 5 datasets. we find that the choice of paraphraser system (model + prompt) strongly affects ner performance. gpt-3 dav-b performs the best at generating paraphrases capable of improving ner performance while other paraphrasers show mixed results. we further find that generating inline annotations using gpt-3 davinci works superior to strictly heuristic based annotations. while we find paraphrasing to be more effective at lower amount of training data, it helps at higher levels depending on dataset, and paraphraser. additionally, we find gpt-3 dav-b to be most immune against entity mention memorization, with the memorization re-ducing further with gpt-3 based mention replacement on certain datasets. our findings speak to the exceptional effectiveness of gpt-3 davinci based systems in generating paraphrases promoting generalization in ner applications, thereby making it the de facto choice for paraphrasing in ner.