CNNB i F : CNN -based Bigram Features for Named Entity Recognition (2021)

Abstract: Transformer models fine-tuned with a sequence labeling objective have become the dominant choice for named entity recognition tasks. However, a self-attention mechanism with unconstrained length can fail to fully capture local dependencies, particularly when training data is limited. In this paper, we propose a novel joint training objective which better captures the semantics of words corresponding to the same entity. By augmenting the training objective with a group-consistency loss component we enhance our ability to capture local dependencies while still enjoying the advantages of the unconstrained self-attention mechanism. On the CoNLL2003 dataset, our method achieves a test F1 of 93.98 with a single transformer model. More importantly our fine-tuned CoNLL2003 model displays significant gains in generalization to out of domain datasets: on the OntoNotes subset we achieve an F1 of 72.67 which is 0.49 points absolute better than the baseline, and on the WNUT16 set an F1 of 68.22 which is a gain of 0.48 points. Furthermore, on the WNUT17 dataset we achieve an F1 of 55.85, yielding a 2.92 point absolute improvement.

## introduction named entity recognition (ner) is a fundamental task in knowledge extraction that detects named entities in text and assigns them to pre-defined categories such as persons, organizations, and locations. it plays a critical role in various applications including question answering, information retrieval, co-reference resolution, and topic modeling @xcite . pre-trained transformers fine-tuned with a sequence labeling objective have become the de facto standard for the ner task because these models have shown state-of-the-art performance without the human effort of feature engineering. despite these achievements, fine-tuning of pretrained transformer models has two potential weak-nesses: first, unconstrained self-attention implements a global receptive field for all interactions, with no inductive bias toward focusing on and composing local dependencies hierarchically @xcite , and second, with small amounts of labeled data, training such models end-to-end is susceptible to overfitting. to address these limitations we propose a novel joint sequence labeling objective, inspired by bert's next sentence prediction (nsp) objective @xcite . in contrast with the nsp objective, which evaluates sentence pairs, we design a word level objective specifically for the ner task. on top of the conventional sequence labeling objective, our novel objective enables modeling of the relationship of adjacent words based on a new tagging scheme, which helps the model to better capture local dependencies in a sequence. for the additional objective, we employ a simple convolutional architecture based on cnn bigram features (in short, cnnbif) to better capture the relationships between adjacent words. under the single loss objective of the conventional sequence labeling approach we have observed that the predictions output by pre-trained transformers quickly converge to the training target labels. our joint learning approach regularizes these models to encourage them to better capture the semantic and syntactic dependencies between nearby words. our key contributions in this paper are: recent applications of multi-task objectives with one of the objectives being named entity recognition has demonstrated improved performance on the ner task. zheng et al., 2017 applied a multitask objective learning to named entity recognition and relation extraction to show improvements over individual tasks. martins et al., 2019 performed joint learning of ner and entity linking tasks in order to leverage the information in two related tasks, using an lstm model architecture. similarly, eberts and ulges, 2019 presented a joint learning model based on a single transformer network to leverage interrelated signals between the ner and entity relationship tasks. prior to the advent of transformer-based networks, cnn networks were applied successfully to various nlp classification tasks. kim, 2014 reports on the effectiveness of these networks where a onelayer cnn is applied to pre-trained word vectors @xcite . ## proposed approach as illustrated in figure 1 our model leverages a pre-trained transformer network. this network is fine tuned with two sequence labeling objectives applied to the single ner task. the first sequence labeling objective is a standard ner objective with the iob2 tagging scheme as described in the following. given an input sequence of n words @xmath0, x 2 , ..., x n ], we perform a prediction on every word x i to obtain a corresponding ner-tag sequence y @xmath1, y 2 , ..., y n ], where y @xmath2,b-per,i-per,b-org,i-org,...} such that every new entity instance starts with a b tag and all subsequent words belonging to that entity instance are marked with an i tag. given the example sentence "obama graduated from columbia university .", the expected ner-tag sequence is "b-per o o b-org i-org o" as shown in figure 1 . the ner objective aims to learn the function f e (θ) : @xmath3. the second sequence labeling objective applies a group-consistency loss component with a new linkage or separation (shortly ls) tagging scheme. beling a word as l when it is internal to a mention (i.e., its ner tag has prefix i-), otherwise the word is labeled as s. given the example in figure 1 , we label 'university' as l because the word is in the same entity with the previous word, labeling other words as s as shown in figure 1 . furthermore, the feature vector for this word is computed by applying a convolutional network with a 2 × 1 kernel to the transformer output features for the current and preceding words. the group-consistency objective aims to learn the function f ls (θ) : @xmath4. for training, two loss functions are computed: l @xmath5. the total loss is given by an unweighted sum: @xmath6. the input sentence is tokenized by byte-pair encoded (bpe) tokens @xcite , and some individual words can be represented by multiple tokens. when a word consists of multiple bpe tokens, we select the first token as its feature vector. ## experiments we fine-tune the pre-trained transformer model on two popular annotated english ner datasets (conll2003 @xcite and ontonotes 5.0 1 ) along with inclusion of the cnn-based bigram features. the resulting models are tested with their repective test datasets as an in-domain evaluation. next, to assess generalization to out-of-domain data, we use the fine-tuned conll2003 model and evaluate its performance on out-of-domain benchmark datasets: ploner @xcite , which is a cross-domain generalization evaluation set with three entity types (person, location, organization), and wnut17 2 . the benchmark datasets are partitioned into a training, development and test set, with development set used for hyperparameter tuning and test set for evaluation. out-of-domain datasets. we employ the ontonotes and wnut16 datasets of ploner @xcite and wnut17 test data 6 to evaluate the proposed approach on unseen domains with a finetuned model on conll2003 training data. the out-of-domain evaluation datasets are summarized in we merge corporation and group into organization, and creative work and product into miscellaneous to align with the four conll2003 categories. following previous work, we measure the precision, recall, and f1 score for each entity category and report the micro-averaged values for each dataset. we use the roberta-large (roberta-l) transformer model @xcite with a simple linear classifier for sequence labeling as a baseline model. we include the cnnbif component on the baseline architecture and train the model with two sequence labeling objectives. we also employ the flert model proposed by @xcite to evaluate our approaches. the flert model leverages documentlevel features for state-of-the-art ner task results. to reproduce flert results, we stay with their proposed xlm-roberta-large (xlm-r-l) transformer model @xcite and finetuning configurations. we add the cnnbif component on top of that and train the model with two sequence labeling objectives. as the representation of each word given input sequence we use the last layer of the transformer and a common subword pooling strategy first @xcite . to fine-tune the transformers we use the adamw @xcite optimizer with the fixed same number of 20 epochs. for the roberta-l transformer we use a linear warmup and linear decay learning rate schedule with a learning rate of 1e-5 and for the flert (xlm-r-l) model we use a one-cycle training strategy with a learning rate of 5e-6 as suggested in their paper. we use the (roberta-l) transformer model from huggingface foot_1 and flert model from flairnlp foot_2 . cnn-based bigram feature component. on top of the two baseline models (roberta-l and flert) we add our proposed cnnbif along with the ner sequence labeling classifier. the input is the representations of individual words adding padding vectors to both sides. for each pair's representation we employ a simple cnn layer with a 2 × 1 kernel filter considering the previous word as the pair. after we truncate the last representation paired with the last padding vector, we produce the same length and same dimension of input representations. on top of the cnnbif layer we add a linear classifier to predict linkage or separation tags of individual pair representations. ## conclusion we propose a novel joint training objective for the ner task that, together with cnn-based bigram features (cnnbif), aims to better capture local dependencies in the transformer architecture. our results show that cnnbif achieves near state-ofthe-art f1 score with a single transformer model on the conll2003 and ontonotes 5.0 benchmark datasets. more importantly, we demonstrate that the proposed model achieves significant gains over the baseline in generalization to out-of-domain datasets. in the future we plan to investigate how the cnnbif component impacts smaller labeled data training sets and other sequence labeling problems such as part-of-speech tagging, word segmentation, and layout extraction from documents by joint modeling of language and document image.