MELM : Data Augmentation with Masked Entity Language Modeling for Low-Resource NER (2022)

Abstract: Data augmentation is an effective solution to data scarcity in low-resource scenarios. However, when applied to token-level tasks such as NER, data augmentation methods often suffer from token-label misalignment, which leads to unsatsifactory performance. In this work, we propose Masked Entity Language Modeling (MELM) as a novel data augmentation framework for low-resource NER. To alleviate the token-label misalignment issue, we explicitly inject NER labels into sentence context, and thus the fine-tuned MELM is able to predict masked entity tokens by explicitly conditioning on their labels. Thereby, MELM generates high-quality augmented data with novel entities, which provides rich entity regularity knowledge and boosts NER performance. When training data from multiple languages are available, we also integrate MELM with code-mixing for further improvement. We demonstrate the effectiveness of MELM on monolingual, cross-lingual and multilingual NER across various low-resource levels. Experimental results show that our MELM consistently outperforms the baseline methods.

## introduction named entity recognition (ner) is a fundamental nlp task which aims to locate named entity mentions and classify them into predefined categories. as a subtask of information extraction, it serves as a key building block for information retrieval @xcite ), question answering @xcite ) and text summarization systems @xcite etc. however, except a few high-resource languages / domains, the majority of languages / domains have limited amount of labeled data. since manually annotating sufficient labeled data for each language / domain is expensive, lowresource ner @xcite @xcite has received increasing attention in the research community over the past years. as an effective solution to data scarcity in low-resource scenarios, data augmentation enlarges the training set by applying label-preserving transformations. typical data augmentation methods for nlp include (1) wordlevel modification @xcite @xcite and (2) back-translation @xcite @xcite . despite the effectiveness on sentence-level tasks, they suffer from the token-label misalignment issue when applied to token-level tasks like ner. more specifically, word-level modification might replace an entity with alternatives that mismatch the original label. back-translation creates augmented texts that largely preserve the original content. however, it hinges on external word alignment tools for propagating the labels from the original input to the augmented text, which has proved to be errorprone. to apply data augmentation on token-level tasks, @xcite proposed to randomly substitute entity mentions with existing entities of the same class. they avoided the token-label misalignment issue but the entity diversity does not increase. besides, the substituted entity might not fit into the original context. @xcite avoided the token-label misalignment issue by only diversifying the context, where they replaced context (having 'o' label) tokens using mass @xcite and left the entities (i.e. aspect terms in their task) completely unchanged. however, according to the ner evaluations in @xcite , augmentation on context gave marginal improvement on pretrained-lm-based ner models. our preliminary results on low-resource ner (see figure 1 ) also demonstrate that diversifying entities in the training data is more effective than introducing more context patterns. inspired by the aforementioned observations, we propose masked entity language modeling (melm) as a data augmentation framework for low-resource ner, which generates augmented data with diverse entities while alleviating the challenge of token-label misalignment. melm is built upon pretrained masked language models (mlm), and it is further finetuned on corrupted training sentences with only entity tokens being randomly masked to facilitate entity-oriented token replacement. simply masking and replacing entity tokens using the finetuned mlm is still insufficient because the predicted entity might not align with the original label. taking the sentence shown in figure 2b as an example, after masking the named entity "european union" (organization), the finetuned mlm could predict it as "washington has". such prediction fits the context but it is not aligned with the original labels. to alleviate the misalignment, our melm additionally introduces a labeled sequence linearization strategy, which respectively inserts one label token before and after each entity token and regards the inserted label tokens as the normal context tokens during masked language modeling. therefore, the prediction of the masked token is conditioned on not only the context but the entity's label as well. after injecting label information and finetuning on the label-enhanced ner data, our melm can exploit rich knowledge from pre-training to increase entity diversity while greatly reducing token-label misalignment. code-mixing @xcite @xcite achieved promising results by creating additional code-mixed samples using the available multilingual training sets, which is particularly beneficial when the training data of each language is scarce. fortunately, in the scenarios of multilingual lowresource ner, our melm can also be applied on the code-mixed examples for further performance gains. we first apply code-mixing by replacing entities in a source language sentence with the same type entities of a foreign language. however, even though token-label alignment is guaranteed by replacing with entities of the same type, the candidate entity might not best fit into the original context (for example, replacing a government department with a football club). to solve this problem, we propose an entity similarity search algorithm based on bilingual embedding to retrieve the most semantically similar entity from the training entities in other languages. finally, after adding language markers to the code-mixed data, we use them to fine-tune melm for generating more code-mixed augmented data. to summarize, the main contributions of this paper are as follows: (1) we present a novel framework which jointly exploits sentence context and entity labels for entity-based data augmentation. it consistently achieves substantial improvement when evaluated on monolingual, cross-lingual, and multilingual low-resource ner; (2) the proposed labeled sequence linearization strategy effectively alleviates the problem of token-label misalignment during augmentation; (3) an entity similarity search algorithm is developed to better bridge entity-based data augmentation and code-mixing in multilingual scenarios. ## method fig. 2c presents the work flow of our proposed data augmentation framework. we first perform labeled sequence linearization to insert the entity label tokens into the ner training sentences (section 2.1). then, we fine-tune the proposed melm on linearized sequences (section 2.2) and create augmented data by generating diverse entities via masked entity prediction (section 2.3). the augmented data undergoes post-processing (section 2.4) and is combined with the original training set for training the ner model. algorithm 1 gives the pseudo-code for the overall framework. under multilingual scenarios, we propose an entity similarity search algorithm as a refined code-mixing strategy (section 2.5) and apply our melm on the union set of gold training data and code-mixed data for further performance improvement. ## experiments to comprehensively evaluate the effectiveness of the proposed melm on low-resource ner, we consider three evaluation scenarios: monolingual, zero-shot cross-lingual and multilingual lowresource ner. ## related work on sentence level tasks, one line of data augmentation methods are built upon word-level modifications, which can be based on synonym replacement @xcite , lstm language model @xcite , mlm @xcite , auto-regressive pretrained lm @xcite , or constituent-based tagging schemes @xcite . however, these methods suffer from token-label misalignment when applied to token-level tasks such as ner, which requires sophisticated post-processing to remove noisy samples in augmented data @xcite . existing works avoid token-label misalignment by replacing entities with existing entities of the same class @xcite , or only modifying context works and leaving entities / aspect terms unchanged @xcite . others attempt to produce augmented data by training / fine-tuning a generative language model on linearized labeled sequences @xcite . backtranslation @xcite @xcite translates source language sentences into a target language, and subsequently back to the source language, which preserve the overall semantics of the original sentences. on token-level tasks, however, they hinge on external word alignment tools for label propagation, which are often error-prone @xcite . ## conclusion we have proposed melm as a data augmentation framework for low-resource ner. through labeled sequence linearization, we enable melm to explicitly condition on label information when predicting masked entity tokens. thus, our melm effectively alleviates the token-label misalignment issue and generates augmented data with novel entities by exploiting pretrained knowledge. under multilingual settings, we integrate melm with code-mixing for further performance gains. extensive experiments show that the proposed framework demonstrates encouraging performance gains on monolingual, cross-lingual and multilingual ner across various low-resource levels.