Improving and Simplifying Template-Based Named Entity Recognition (2023)

Abstract: With the rise in larger language models, researchers started exploiting them by pivoting the downstream tasks as language modeling tasks using prompts. In this work, we convert the Named Entity Recognition task into a seq2seq task by generating the synthetic sentences using templates. Our main contribution is the conversion framework which provides faster inference. In addition, we test our method’s performance in resource-rich, low resource and domain transfer settings. Results show that our method achieves comparable results in the resource-rich setting and outperforms the current seq2seq paradigm state-of-the-art approach in few-shot settings. Through the experiments, we observed that the negative examples play an important role in model’s performance. We applied our approach over BART and T5-base models, and we notice that the T5 architecture aligns better with our task. The work is performed on the datasets in English language.

## introduction named entity recognition (ner) is traditionally approached as a sequence labeling task where a tag is predicted for each token. for a sentence with l tokens, the output would be l tags, usually, in the inside-outside-beginning (iob) tagging format @xcite . traditionally in probabilistic classification, language models (lms) "compute the probability of a label, conditioned on the text" @xcite , that is, they quantify the likelihood of a sentence to pertain to a language. contextualized word representations are used nowadays as pre-trained language models that have shown to be effective in natural language processing tasks as co-reference resolution, gender resolution, etc. current language modeling approaches can be broadly divided into two types: architectures like bidirectional encoder representations from transformers or bert models @xcite and t5 @xcite follow this approach. @xcite and @xcite suggested transferring the knowledge learned during pre-training to downstream tasks but with limited success. however, with the advances in neural networks, @xcite proved that it is possible by successfully fine-tuning a pretrained lstm @xcite ) language model on a downstream task. after pretraining, the model's last layer, which is used to predict the next word, is replaced with a new layer for a downstream task. this way, information from the remaining layers can help the model learn new tasks better. instead of introducing a new layer at the end of pre-trained models, schick and schütze (2020) converted the downstream task examples into a cloze-question format to leverage the knowledge acquired during the pre-training. later, the pretrained model is further fine-tuned in a method called pattern exploitative training (pet). downstream tasks like sentiment analysis would be much simpler to adapt since one can just append a template at the end of the review and expect the model to predict words related to the corresponding label. for instance, if we are predicting the polarity of a movie review, we append a template to the review as follows overall, the movie @xmath0. however, pet cannot be directly applied to named entity recognition (ner) since the outputs must be constrained to phrases within the sentence. it is difficult (if not impossible) to provide a single task description which allows the language model to assign a label to each token in the input text @xcite . @xcite handled this issue by appending a template, filled by each word and entity type, to the sentence where the model must predict if the word in the template is either beginning, inside or outside an entity type. the template is generated for all combinations of words in the sentence and entity types. @xcite approached the problem by converting the ner identification task into a seq2seq task by generating synthetic target sentences. figure 1 illustrates their data creation process. ngrams generated from the sentence are substituted in the template to generate positive and negative examples. the authors fine-tune bart language model @xcite to train a seq2seq model, which inputs the natural source sentence (sentence before the "→" symbol in figure 1 ) and outputs the synthetic target sentence (sentence after the "→" symbol). at inference, the fine-tuned model is used to score the synthetic sentences generated from all the n-grams. a major limitation of this approach is the n-grams part because, during inference, for a given sentence, all the synthetic target sentences generated from all the n-grams ranging from @xmath1. this situation exacerbates with a larger sentence and more number of entity types in a task. avoiding the problem of generating all possible ngrams, @xcite proposed a pointer-based seq2seq (bartner) framework, which converts ner sub-tasks to a unified sequence generation task and predicts entities from the input sentences and the corresponding type indexes. lightner @xcite introduced prompt-tuning to the attention mechanism of bartner and achieved promising improvement in low-resource scenarios. our contribution is to avoid the usage of ngrams by pivoting our task from a sequence labeling task into a seq2seq task that aligns with the pre-training objective of certain generative models like t5, bart, etc. specifically, we leverage templates to convert the target entities into sentences. figure 2 illustrates the conversion process. our approach performs better in low-resource settings and is comparable to existing works in resourcerich setting. in comparison to the recent works, our approach is faster than templatebart @xcite in terms of real-time inference. although bartner does inference in a single step, we observed that our approach yields better results at the cost of linear inference speed, proportional to the number of entity types. at the same time, our approach is simpler than lightner. we test our method's performance in resourcerich, low resource and domain transfer settings to prove its robustness. we approximate low resource setting by considering few-shot scenarios of the datasets. results show that our method achieves comparable results in the resource-rich setting and outperforms the current seq2seq paradigm state-ofthe-art approach in few-shot settings. ## related work to adapt the fine-tuning approach to the ner task, the standard norm has been to use architectures like lstms @xcite , cnns @xcite , and transformers @xcite to extract token-level features which are later passed on for classification into the corresponding entity classes. in the final layer, the softmax function @xcite @xcite or the conditional random fields algorithm (crf) @xcite @xcite are used for classification. with the rise of powerful lms, there has been a lot of interest in exploiting them for downstream tasks. ulmfit @xcite was the first approach to successfully fine-tune a pretrained lstm language model to a downstream task. however, this approach does not completely exploit the information lms have learned during pre-training. schick and schütze (2020) converted the downstream tasks into cloze questions to probe lms and leverage the knowledge learned during pre-training. simultaneously, brown et al. (2020) showed that, with large lms (170 billion parameters large), it is possible to probe information in a few-shot approach without having to train the full model. these methods gave rise to a new research field called pattern engineering @xcite . however, until recently, most prompt-based approaches were not designed for named entity recognition. @xcite were the first to adopt the template-based approach for sequence labeling. they convert the task into a seq2seq task and use language models as a scoring function for each span. as mentioned before, @xcite filter the n-grams by labels provided by a linguistic analyzer in a textual entailment task. ## methodology we consider ner as a language model generation task where the input is a text written in natural language, appended by synthetic sentence generated by a template and the output is also a synthetic sentence but mentioning the corresponding entities. (see section 3.1). we start this section by introducing the template creation process in section 3.1, and then explain the training and inference processes in sections 3.3 and 3.2, respectively. in later sections, we speak about the different assumptions made for different resource settings. ## conclusions and future work in this work, we project ner as a seq2seq task by generating synthetic sentences from templates and show that our approach is faster than the current state-of-the-art. we also show that our approach works better in few-shot and domain transfer scenarios especially when the input size is small, although the approach can be used in other scenarios. while performing the experiments, we found t5 to better align with our task and also observed the importance of negative examples. our system has been compared against seq2seq systems (templatener, lightner and bart-ner). but instruction-ner @xcite leverages auxiliary training and obtains better results than our system. for instance, in mit movies few shot scenario (@xmath2, our approach has 53 f1 score and instructionner 65 f1 score. in future, it would be interesting to explore automatic template generation along with auxiliary training.