Exploring Cross-sentence Contexts for Named Entity Recognition with BERT (2020)

Abstract: Named entity recognition (NER) is frequently addressed as a sequence classification task with each input consisting of one sentence of text. It is nevertheless clear that useful information for NER is often found also elsewhere in text. Recent self-attention models like BERT can both capture long-distance relationships in input and represent inputs consisting of several sentences. This creates opportunities for adding cross-sentence information in natural language processing tasks. This paper presents a systematic study exploring the use of cross-sentence information for NER using BERT models in five languages. We find that adding context as additional sentences to BERT input systematically increases NER performance. Multiple sentences in input samples allows us to study the predictions of the sentences in different contexts. We propose a straightforward method, Contextual Majority Voting (CMV), to combine these different predictions and demonstrate this to further increase NER performance. Evaluation on established datasets, including the CoNLL’02 and CoNLL’03 NER benchmarks, demonstrates that our proposed approach can improve on the state-of-the-art NER results on English, Dutch, and Finnish, achieves the best reported BERT-based results on German, and is on par with other BERT-based approaches in Spanish. We release all methods implemented in this work under open licenses.

## introduction named entity recognition (ner) approaches have evolved through various methodological phases, broadly including rule/knowledge-based, unsupervised, feature engineering and supervised learning, and feature inferring approaches @xcite . the use of cross-sentence information in some form has been a normal part of many ner methods in the former categories, but its role has diminished with the current feature inferring deep learning based approaches. rule/knowledgebased approaches such as that of @xcite typically match strings to lexicons and similar domain knowledge sources, possibly going through text multiple times with refinement based on entities found on earlier passes. later, manually engineered features were used to incorporate information from the surrounding text, whole documents, data sets and also from external sources. the number of different features and classifiers grew during the years and it was normal that the features also contained cross-sentence information in some form as for example in @xcite . dense representations of text such as word, character, string and subword embeddings first started to appear in ner methods as additional features given to classifiers @xcite . step by step, feature engineering has been demoted to a lesser role, as the most recent deep learning approaches learn to create meaningful and context-sensitive representations of text by pre-training with vast amounts of unlabelled data. these contextual representations are often used directly as features for existing ner architectures or fine-tuned with labelled data to match a certain task. in recent years, the development of natural language processing (nlp) in general and ner in particular have been greatly influenced by deep transfer learning methods capable of creating contextual representations of text, to the extent that many of the state-of-the-art ner systems mainly differ from one another on the basis of how these contextual representations are created @xcite @xcite . using such models, sequence tagging tasks are often approached one sentence at a time, essentially discarding any information available in the broader surrounding context, and there is only little recent study on the use of cross-sentence context -sentences around the sentence of interest -to improve sequence tagging performance. in this paper, we present a comprehensive exploration of the use of cross-sentence context for named entity recognition, focusing on the recent bert deep transfer learning models @xcite based on self-attention and the transformer architecture @xcite . bert uses a fixed-size window that limits the amount of text that can be input to the model at one time. the model maximum window size, or maximum sequence length, is fixed during pre-training, with 512 wordpieces a common choice. this window fits dozens of typical sentences of input at a time, allowing us to include extensive sentence context. here, we first study the effect of predicting tags for individual sentences when they are moved around the window, surrounded by their original document context from the source data. second, we utilize different predictions for the same sentences to potentially further improve performance, combining predictions using majority voting, adapting an approach that has been used already in early ner implementations @xcite @xcite . we evaluate these approaches on five languages, contrasting ner results using bert without cross-sentence information, sentences in context, and aggregation using contextual majority voting (cmv) on well-established benchmark datasets. we show that using sentences in context consistently improves ner results on all of the tested languages and cmv further improves the results in most cases. comparing performance to the current state-of-the-art ner results in the 5 languages, we find that our approach establishes new state-of-the-art results for english, dutch, and finnish, the best bert-based results on german, and effectively matches the performance of a bert-based method in spanish. ## related work the state-of-the-art in ner has recently moved from approaches using word/character representations and manually engineered features @xcite toward approaches directly utilizing deep learning-based contextual representations @xcite @xcite while adding few manually engineered features, if any. while successful in terms of ner performance, these approaches have tended to predict tags for one sentence at a time, discarding information from surrounding sentences. one recent method taking sentence context into account is that of @xcite , which addresses a weakness of an earlier contextual string embedding method @xcite , specifically the issue of rare word representations occurring in underspecified contexts. @xcite make the intuitive assumption that such occurrences happen when a named entity is expected to be known to the reader, i.e. the name is either introduced earlier in text or is of general in-domain knowledge. their approach is to maintain a memory of contextual representations of each unique word/string in text and pool together contextual embeddings of a string occurring in text with the contextual embeddings of the same string earlier in text. this pooled contextual embedding is then concatenated with the current contextual embedding to get the final embedding to use in classification. another recent approach taking broader context into account for ner was proposed by @xcite , where in addition to token representations, also sentence and document level representations are calculated and used for classification using a crf model. a sliding window is used by @xcite so that part of the input is preserved as context when the window is moved forward in text. @xcite state that they use longer paragraphs in pre-training their model, but it is not mentioned in the paper if such longer paragraphs are used also in fine-tuning the model or predicting tags for ner. some other approaches such as that of @xcite include explicit global information in the form of e.g. gazetteers. also, some approaches formulate ner as a span finding task instead of sequence labelling @xcite . these approaches would likely allow the use of longer sequences, but the incorporation of cross-sentence information is not explicitly proposed by the authors. in the paper introducing bert, @xcite write in the description of their ner evaluation "we include the maximal document context provided by the data." however, no detailed description of how this inclusion was implemented is provided, and some ner implementations using bert have struggled to reproduce the results of the paper. 1,2 the addition of document context to ner using bert is discussed also by @xcite , who fill each input sample with the following sentences and use the first sentence in each sample for predictions, and thus only introduce context appearing after the sentence of interest in the source text. of the related work discussed above, our approach most closely resembles that of @xcite , which in turn aims to directly follow @xcite . by contrast to other studies discussed above, we do not introduce extra features or embeddings representing cross-sentence information or incorporate extra information in addition to that captured by the bert model. instead, we directly utilize the bert architecture and rely on self-attention and voting to combine predictions for sentences in different contexts. ## data the data used in this study consists of pre-trained bert models and ner datasets for five different languages. we aimed to use monolingual bert models as numerous recent studies have suggested that well-constructed language-specific models outperform multilingual ones @xcite @xcite . we selected the following language-specific pre-trained bert models for our study, focusing on languages that also have established benchmark data for ner: for comparison purposes we also tested multilingual bert foot_8 with the spanish language. from the models introduced above all except german and multilingual bert have used the whole word masking variation of the masked language model objective in pre-training instead of the method introduced in the original paper @xcite . whole word masking was introduced by the developers of bert after the original paper was published. in this pre-training objective, all the tokens corresponding to one word in text are masked instead of completely random tokens, which often leaves some of the tokens in multi-token words unmasked. we aimed to apply sufficiently large, widely-used benchmark datasets for evaluating ner results, assessing our methods primarily on the conll'02 and conll'03 shared task named entity recognition datasets @xcite , which cover four of our five target languages. for the fifth language, finnish, we use two recently published named entity recognition corpora (ruokolainen et al., 2019; luoma et al., 2020) 9,10 . these two finnish datasets are annotated in a compatible way, and for this study they are combined into a single corpus by simple concatenation, following @xcite . all of the ner datasets define separate training, development and test sets, and we follow the given subdivision for each. the training sets for each language are used for fine-tuning the corresponding bert model for ner, development sets are used for evaluation in hyperparameter selection, and the test sets are only used in final experiments for evaluating models trained with the selected hyperparameters. as previous studies vary in whether to combine development data to training data for training a final model, we report also results where models are trained with a combined training and development set for final test experiments. the datasets for the conll shared task languages contain four different classes of named entities: person (per), organization (org), location (loc) and miscellaneous (misc). the finnish ner datasets also use the per, org, and loc types along with three others, product (prod), event (event), and date (date). for implementation purposes we converted all the datasets to the same format prior to experiments: the character encoding of each file was converted to utf-8, and the ner labelling scheme was converted to iob2 @xcite also for corpora that were originally in the iob scheme @xcite . by contrast to the older iob scheme, in the iob2 scheme the label for the first token of a named entity is always marked with a b-prefix (e.g. b-per), even if the previous token is not part of a named entity. the key statistics for the ner datasets are presented in ## methods as the starting point for exploring the cross-sentence information for ner using bert, we use a ner pipeline implementation introduced by @xcite that closely follows the straightforward approach presented by @xcite . here, the last layer of the pre-trained bert model is followed by a single time-distributed dense layer which is fine-tuned together with the pre-trained bert model weights to produce the softmax probabilities of ner tags for input tokens. no modelling of tag transition probabilities or any additional processing to validate tag sequences is used. in our implementation, exactly one example is constructed for each sentence of the corpus unless the sentence is so long that it does not fit to the maximum sequence length foot_11 . the sentence is placed at the beginning of the bert window and following sentences from the corpus are used to fill the window (up to the maximum sequence length), with special separator ([sep]) tokens separating the sentences. partial sentences are used to fill up the bert examples. as a special case, the sentences used for filling the window for the last sentences in input data are picked by wrapping back to the beginning of the corpus. this approach creates situations where some input samples contain sentences from different original documents, if the documents were next to one another in the corpus. for this reason, we also implemented documentwise wrapping of sentences if the input data had document boundaries marked with -docstart-tokens. we used this information to build input samples by filling the sentences at the end of one document with the sentences from beginning of that same document instead of the next sentences in the original data. in this case only full sentences are added to each input sample, and padding ([pad]) tokens are used to fill empty space if the next sentence in the input data does not fit into the window as demonstrated in (figure 1b ). constructing inputs in this way implies that the same sentences from the original data occur in different positions and with varying (sizes of) left and right contexts in different samples. we wanted to examine the predictions in different contexts more closely to see if there are consistent effects on tag prediction quality depending on the starting position of a sentence inside a context. one challenge here was how to consistently measure performance with different contexts: sentences are of different lengths, and as they are added to input samples, the beginning of the window was only place where the starting locations of sentences would align. also, the number of sentences that fit into the window vary substantially. for this reason, it is not possible e.g. to always pick the nth sentence to study as there are no guarantees one will exist in all examples. to address this issue and build input samples for testing predictions at different locations, we placed the sentence of interest to start at a specified location inside the window, and filled the window in both directions with sentences before / after the sentence of interest in the original data. we tested the starting positions of the sentence of interest from 1 (0 being the [cls] token) up to the maximum sequence length (512 wordpieces) with intervals of 32 wordpieces. if the sentence of interest was longer than the space between a starting position and the maximum sequence length, the starting position for that particular sentence was moved backwards to fit the sentence in the window. ensembles of classifiers are commonly used to improve classification performance at various tasks, and it seems reasonable to assume that predictions for the same input sentences in different positions and contexts create an ensemble-like construct. this is not an ensemble in the conventional sense, as the number of predictions we get for each sentence varies. we evaluate two different variations combining the results from multiple predictions in different contexts. the first approach is to assign labels to sentences in each location first, and then take a majority vote of the assigned labels. the second approach is to add together the softmax probabilities of predictions in different contexts, and then take the argmax of the sum. for simplicity, we here term both contextual majority voting (cmv) as they are variations of the same underlying idea. the implementation uses only predictions of tokens in whole sentences, not ones in partial sentences that may appear in input examples. for fine-tuning the pre-trained bert models, we largely follow the process introduced in @xcite . we use the maximum sequence length of 512 in all experiments to include maximal cross- we select hyperparameters with an exhaustive search of the grid proposed by @xcite , modified to skip batch size 32 and add batch sizes 2 and 4 instead as our initial experiments indicated better performance with smaller batch sizes. that is, the grid search is done over the following parameter ranges: we repeated each experiment 5 times with every hyperparameter combination. the best hyperparameters were selected based on the mean of exact mention-level f1 scores, as evaluated against the development set using a python implementation of the standard conlleval evaluation script. as a reference we use a bert model which is fine-tuned using only single sentences from the input data. for this baseline, predictions are also made on the basis of single sentences (see figure 1a ). ## results based on initial development set results, we decided to focus only on cmv using examples constructed document-wise of the variations of this method (see section 4). the exception here is the spanish conll dataset, for which document boundary information was not available. further, as the differences between cmv variations were found not to be large, we decided to only consider the variant that first assigns labels and then votes between the labels. the effect of the sentence of interest starting location and the effect of cmv method on development data is illustrated in figure 2 . our initial expectation was that placing the sentence of interest near the middle of the sequence would generally yield the best performance. however, while this effect can be observed e.g. for english (figure 2a ), the pattern does not hold in all cases, although in most cases performance does improve when moving the starting position away from either end of the context window. the problem was that the performance in the middle of the context did not appear to be stable enough to pick a reliable starting position to look at prediction time. this can be seen in the figure 2 where the results for different starting locations tend to vary without a clear central optimum. the results for dutch (figure 2b ) deviated the most from our expectations, and a possible reason for this was later found from the source data: the sentence order of the documents inside the original dutch language data set has been randomized for copyright reasons. to test if randomizing the sentence order of documents has an effect on results, we tested this with other languages. however, in our initial experiments randomizing sentences inside each document did not result in significant performance drop on any of the tested languages. the final test set results for models trained with the best hyperparameter combinations found using the development sets are summarized in these results show that bert ner predictions systematically benefit from access to cross-sentence context. for all tested languages except spanish, models that are fine-tuned and tested with samples containing context outperform models which do not use any context and are relying only on single sentences. what is not directly seen from to further evaluate the performance of cmv method, we checked the results of each fine-tuned model on the development set during hyperparameter search. there were 48 hyperparameter combinations to evaluate for each model. for english, german, spanish and finnish, the cmv method outperformed the method first for every hyperparameter combination when calculating the results as the mean of mention-level f1 scores from 5 repetitions. for spanish this includes both the experiments with the spanish monolingual model as well as the experiments with the multilingual model. the only exception to this were the results on dutch, for which cmv outperformed the method first in 41 cases out of 48. the fact that sentences in dutch data are in randomized order may contribute to this. in total, the cmv method improved the results over method first in 281 cases out of 288. in the same fashion, we evaluated the difference in performance between the method single and the method first evaluated against the development set. the method first outperformed the method single for every hyperparameter combination for every tested language. in ## discussion the results presented here are, as far as we know, the first systematic study on how cross-sentence information can be utilized with bert for ner, and the methods presented here form a good starting point for discussion and further research into the subject. contextual majority voting is straightforward to implement in existing bert-based systems as the actual model and associated infrastructure is not modified. it is quite probable that similar ways of including cross-sentence information or majority voting structures may be beneficial with other attention-based models as well. the computational overhead for the required pre-and postprocessing of the samples is very modest, but increasing the maximum sequence length in fine-tuning e.g. from 128 to 512 to fit more sentences in one sample does come with a tradeoff of increased computational cost. one aspect deserving more study is how prediction performance is affected if sentences are not repeated, or repeated fewer times, in examples during prediction. reducing or entirely avoiding repetition would allow for more efficient use of the model while still providing context for sentences, which might be a reasonable compromise between performance and computational efficiency for large-scale practical applications. a further possibility for future research would be to explore weighted majority voting. our results lend some support to the idea that predictions made for tokens around in the center of the window are generally more reliable than predictions for tokens near its edges, where context is limited on one side of the token. providing higher weight to predictions in the middle of the sequence could potentially help further improve the performance of the aggregation approach. another aspect for future work would be to study the effect of the context and sentence order. our preliminary tests with randomized sentence order from same documents showed minimal effect on performance. is it enough to have context from the same document? would the situation change drastically if random sentences from the whole input data were used instead? finally, the incorporation of transition probabilities or other processing to check tag sequences for illegal transitions would likely improve performance further. ## conclusions we have presented a comprehensive evaluation of the effect of including cross-sentence context for named entity recognition with bert and introduced a simple and easy-to-implement approach for the task using majority voting. the proposed method established new state-of-the-art results in named entity recognition for three languages and is near the state-of-the-art for two other languages, demonstrating how simple ideas may boost the performance of even very strong models. we release all methods implemented in this work under open licenses from https://github . com/jouniluoma/bert-ner-cmv .