PA Ph&Tech at S em E val-2022 Task 11: NER Task with Ensemble Embedding from Reinforcement Learning (2022)

Abstract: From pretrained contextual embedding to document-level embedding, the selection and construction of embedding have drawn more and more attention in the NER domain in recent research. This paper aims to discuss the performance of ensemble embeddings on complex NER tasks. Enlightened by Wang’s methodology, we try to replicate the dominating power of ensemble models with reinforcement learning optimizor on plain NER tasks to complex ones. Based on the composition of semeval dataset, the performance of the applied model is tested on lower-context, QA, and search query scenarios together with its zero-shot learning ability. Results show that with abundant training data, the model can achieve similar performance on lower-context cases compared to plain NER cases, but can barely transfer the performance to other scenarios in the test phase.

## introduction named entity recognition (ner) as a typical topic in the nlp field, has displayed numerous outstanding outcomes in recent years, especially benefiting from development of pretrained models. however, there are still challenging aspects that remain to be tackled, such as short texts (low-context), emerging entities, and complex entities. in this task, corpus from low-context, qa and search query scenarios are collected to represent different complex ner tasks @xcite . @xcite proposed an automated concatenation model on plain ner tasks with benchmark dataset like conll03, which automatically generates optimized concatenation of stack embeddings with reinforcement learning strategies for structure prediction tasks like ner. a similar methodology is applied on the given dataset to test if the challenges mentioned above can be properly addressed. ## related work the development of sequence tagging models can be concluded mainly into two separate parts: encoder and decoder. ever since the bilstm-crf model @xcite was brought up, it has been widely used as the decoder end of ner models, while researchers shifted increasing attention to the structure of encoder. various categories of embeddings have been raised, including character embeddings, non-contextual embedding like word2vec @xcite and pretrained contextualized embeddings like elmo @xcite and flair @xcite . with the emergence of transformers, the performance of large pretrained contextualized models @xcite have sweeped the leaderboard in lots of nlp tasks. in terms of language, multilingual bert(m-bert) @xcite demonstrates excellent representation of multilingual embeddings, which was later surpassed by xlm-r @xcite , a more powerful and comprehensive multilingual model. to better allocate these embedding methods, many recent researchers have tried different methods like ensemble, weighting and concatenation. automated concatenation @xcite is a superior method that automatically generates optimized concentration of stack embeddings with reinforcement learning strategies. it uses result accuracy as reward for the controller to decide which embeddings to drop. ## data as described by @xcite , the dataset @xcite of this semeval task (malmasi et al., 2022b) mainly consists of three sources: low-context wikipedia, ms-marco question @xcite and orcas search query @xcite . sentences from wikipedia are parsed and linked pages are resolved to their respective wikidata entities, to create a corpus of 1.4 million low-context sentences with annotated entities. part of them form the train and dev dataset. by templating questions in ms-marco qna dataset and 10 million bing user queries from the orcas dataset, and slotting with random entities based on frequency, 17, 868 questions and 471, 746 queries are generated to form the test set together with the rest of low-context sentences. several data processing and augmenting is conducted before training to cater for the applied model. since document-level embeddings have been proved to be effective for performance improvement in nlp tasks, we adopt yamada et al. (2020)'s method and extract features by sending the adjacent sentences in corpus together as a document to pretrained models. the number of sentences that each document contains is defined by grid search and eventually set to 30. although the adjacent sentences are not sentimental related as a document, the document level representation still enriches the context and greatly resolves the low-context situation. another novelty of the semeval dataset is the unbalance between train and test set, in terms of quantity and content. to solve the quantity unbalancing, we manage to augment the train set by slotting the entities in sentences and altering with another one from the same type. in case of overfitting, we eventually augment the train set to 3 times of its original size. ## methodology after dealing with data, a common approach for better results is to fine-tune the transformer-based embeddings first, where sentences are sent to the model which is connected to a linear layer for tag prediction. different language embeddings are selected for different tracks of the task. for english models, we fine-tune bert-base, bert-large, m-bert, xlnet @xcite , roberta @xcite , xlm-r separately and concatenate these embeddings with basic settings of other embeddings like elmo, flair and fasttext @xcite , for final training. for dutch, spanish and german models, we fine-tine m-bert, xlm-r and bert for each language respectively. the parameters for the fine-tuning process are 10 max epochs with batch size of 1 and learning rate of 5.0 × 10 -06 . the concatenated embeddings are used as inputs for a sequence tagging model with bilstm lay- ## results due to limitations in time and calculation resources, we only present the results on non-english language models. ## conclusion we focus on the performance of automatic concatenated embedding model on semeval complex ner task, and draw the conclusion that with proper data processing methods, the model can learn excellent sequence tagging ability from low-context corpus and achieve outstanding performance on corresponsive part in test set, but cannot transfer such ability to qa and search query domains in test set. meanwhile, document-level feature extraction and data augmenting by slotting and altering entities are