Identifying Named Entities as they are Typed (2021)

Abstract: Identifying named entities in written text is an essential component of the text processing pipeline used in applications such as text editors to gain a better understanding of the semantics of the text. However, the typical experimental setup for evaluating Named Entity Recognition (NER) systems is not directly applicable to systems that process text in real time as the text is being typed. Evaluation is performed on a sentence level assuming the end-user is willing to wait until the entire sentence is typed for entities to be identified and further linked to identifiers or co-referenced. We introduce a novel experimental setup for NER systems for applications where decisions about named entity boundaries need to be performed in an online fashion. We study how state-of-the-art methods perform under this setup in multiple languages and propose adaptations to these models to suit this new experimental setup. Experimental results show that the best systems that are evaluated on each token after its typed, reach performance within 1–5 F1 points of systems that are evaluated at the end of the sentence. These show that entity recognition can be performed in this setup and open up the development of other NLP tools in a similar setup.

## introduction automatically identifying named entities such as organizations, people and locations is a key component in processing written text as it aids with understanding the semantics of the text. named entity recognition is used as a pre-processing step to subsequent tasks such as linking named entities to concepts in a knowledge graph, identifying the salience of an entity to the text, identifying coreferential mentions, computing sentiment towards an entity, in question answering or for extracting relations. figure 1 : an example of the proposed task and evaluation setup. after the word 'foreign' is typed, the model immediately predicts an ner label for this word, only using left context ('a spokesman for') and the word itself. the prediction is then compared against the gold label to compute token-level f 1 score. this token's prediction will not be changed, even if the model's internal prediction for it can be revised later as more tokens are typed. identifying named entities as they are typed benefits any system that processes text on the fly. examples of such applications include: a) news editorswhere named entities can be highlighted, suggested (auto-completion), co-referenced or linked as the editor is typing; b) auto-correct -where named entities that are just typed are less likely to need correction as they may come from a different language or be out-of-vocabulary (oov); c) simultaneous machine translation -where translation of oov named entities requires different approaches; d) live speech-to-text (e.g., tv shows) -where named entities are more likely to be oov, hence the transcription should focus more on the phonetic transcription rather than on n-gram language modelling. this paper introduces a novel experimental setup of named entity recognition systems illustrated in figure 1 . in this setup, inference about the span and type of named entities is performed for each token, immediately after it was typed. the sentence level tag sequence is composed through appending all individual token predictions as they were made. the current named entity recognition systems that are trained and evaluated to predict full sentences are likely to under-perform in this experimental setup as they: expect that right context is available, are faced with unseen types of inputs in the form of truncated sentences and can not reconcile the final sentence-level tag sequence across the entire sentence as the result may not be a valid sequence. the goal of this study is to present a comprehensive analysis of the task of ner in the as-you-type scenario, with the following contributions: a) a novel experimental setup for conducting named entity recognition experiments, denoted as the as-you-type scenario; b) experiments with state-of-the-art sentence-level approaches to named entity recognition in the as-you-type setup across three languages, which indicate a 1-5 f 1 points decrease compared to sentence-level inference; c) tailored methods for as-you-type entity recognition models which reduce the gap to entire sentence-level inference by 9-23% compared to regular approaches; d) an extensive analysis of existing data sets in the context of this task and model error analysis, which highlight future modelling opportunities. ## related work named entity recognition is most commonly treated as a sequence labelling problem, where a prediction of whether a token is an entity and its type is done jointly for all tokens in the sentence. over the past recent years, the dominant approach is based on recurrent neural networks, such as lstms @xcite . these architectures use a stacked bi-directional lstm units to transform the word-level features into distributions over named entity tags @xcite . usually, an additional conditional random field (crf) @xcite is used on the bilstm output in order to take into better model neighbouring tags. the tokens inputs are represented using one or a concatenation of pretrained static word embeddings such as glove @xcite , contextual word embeddings @xcite @xcite , pooled contextual word embeddings @xcite or character embeddings trained using bilstms @xcite or cnns @xcite . in addition to research on improving the performance of the ner model, other experimental setups have been proposed for this task. these include domain adaptation, where a model trained on data from a source domain is used to tag data from a different target domain @xcite @xcite , temporal drift, where a model is tested on data from future time intervals @xcite , cross-lingual modelling where models trained in one language are adapted to other languages @xcite @xcite , identifying nested entities @xcite or high-precision ner models @xcite . however, all these experimental setups assume that training is done over full length sentences. perhaps the most related experimental setup to the one we propose for the task of entity recognition is the task of simultaneous machine translation. in this setup, the task is to generate an automatic translation in a target language as the text is being processed in the source language. the goal of the task is to produce a translation that is as accurate as possible while limiting the delay as compared to the input. initial approaches involved identifying translatable segments and translating these independently @xcite @xcite or by learning where to segment in order to optimize the system's performance @xcite . more recent approaches involve learning training an agent, usually using reinforcement learning, that makes a set of decisions of whether to should wait for another word from the input or write a token to the output @xcite . other operations are shown to help, including predicting the verb @xcite or the next word @xcite , better decoding with partial information @xcite , and connecting the machine translation system to the agent's decisions @xcite . our experimental setup is different as we do not want to wait for another input token before we make a prediction about the named entity. we analyze the impact a delay has, albeit our experimental setup does not aim to combine quality and delay. the challenges are related, as the input may contain important cues for the translation or named entity decision after the current token or towards the end of the sentence, such as the verb in verb-final (sov) languages such as german @xcite . the proposed as-you-type ner model can be useful to improve simultaneous machine translation. ## experimental setup we propose a new experimental setup for the standard task of named entity recognition that would best suit real-time applications that need to process text in an online fashion. in the regular ner experimental setup, a model is presented with a sequence of inputs @xmath0, x 2 , ..., x n } and it outputs a sequence of labels @xmath1, y 2 , ..., y n } where y @xmath2, where e are the set of entity types and t is the entity tag representation. throughout the rest of the paper, we use the bio tagging scheme (@xmath3, i}), as this is arguably the most popular and differences in results between this tagging scheme and others, such as the bilou scheme, are very small in practice @xcite . the types of entities we consider are @xmath4, per, loc, misc}. the as-you-type named entity recognition setup assumes that the editor writing the text @xmath5, x 2 , ..., x n } needs each label prediction y i right after the corresponding token x i was typed. in this case, the information available for predicting y i is only the sub-sequence x 1,@xmath6, x 2 , ..., x i }. the sequence @xmath7, y 2 , ..., y n } is obtained by concatenating the individual y i predictions made for each token. token-level micro f 1 score is used as the metric in our experiments. the evaluation process is illustrated in figure 1 . this setup presents the model with several challenges. first, the model has no access to right context when making the prediction for each tag. however, this information is available in training. secondly, the output sequence may contain invalid sequences of tags. for example, in the output sequence, b-org could be followed by i-loc if the model decided to revise its predictions based on new information, but the evaluation setup prevents the model from revising the previous wrongly predicted tag (i.e. b-org). lastly, sequences and sentences of the same length are likely to be qualitatively different and the model might need to adapt in training in order to account for these differences. we note that this experimental setup can further be extended to account for delays in prediction, to trade-off between delays and quality or to predict entities before they are typed. ## data we test our methods on three different data sets covering three different languages. we use the data sets released as part of conll shared tasks in 2002 for spanish @xcite 1 and in 2003 for english and german @xcite . 2 the data sets contain four types of named entities: persons, locations, organizations and names of miscellaneous entities that do not belong to the previous three types. we use the standard train, dev and test splits defined for these data sets. we chose these data sets as they are arguably the most popular data sets for performing named entity recognition and are regularly used as benchmarks for this task. we use the data sets in different languages in order to compare the impact of the language on the experimental results, identify if the commonalities and peculiarities for performing asyou-type entity recognition in different languages and draw more robust conclusions regarding our task setup. ## methods this section describes the methods used to perform named entity recognition in the as-you-type scenario. we use a base neural architecture that achieves state-of-the-art performance on the standard sentence-level ner task. we study its performance and observe the impact of different variants of the architecture in the as-you-type scenario. following, we propose changes to the model to adapt to the as-you-type setup. we use the flair package to conduct our experiments @xcite . 3 implementation details and hyperparameter choices for all models are listed in the appendix. ## results we present the experimental results of the various ner models in the as-you-type setup, contrasting them with the regular sentence-level setup. all models are trained using the standard splits for the conll data sets. the evaluation metric is tokenlevel micro f 1 , as this is reflective of our evaluation setup where each token is evaluated separately. the top section of adding the crf hurts performance in all except one case when holding everything else constant, sometimes by wide margins (e.g. 5.3 f1 drop on spanish with flair forward embeddings and unidirectional lstm). we attribute this to the mismatch between the structure of the sequences in the training data containing only full sentences, when compared to truncated sentences which can be observed by comparing figures 3 to figures 4. the bottom section of the changes to the last token representation and weighted loss improves on the regular bidirectional lstm model by a substantial margin, adding between 0.45 for spanish up to 4.25 f1 on english on the as-you type setup performance. we also notice that the sentence-level evaluation is near to the regular model performance (-0.39 for german to +0.05 for spanish), showing that the weighted loss is able to achieve a good compromise between the representations. adding the feature weighting based on position marginally improves performance, between 0.02 on german to 0.21 on english. however, the weighting through attention is more effective at the embedding level improving on the previous model by between 0.19 f1 on english to 1.01 f1 on german. overall, our as-you-type adaptation methods improve on the best variation of the base architecture on english (+0.45 f1) and german (+0.83 f1). the model is competitive on spanish (-0.12 f1) to the flair forward unidirectional lstm with no crf, albeit this is likely driven by the very long average sentence length in the spanish data set (see figure 2 ). overall, the improvements represent between 9.3% -23% error reduction when comparing to the best as-you-type and sentence level setups. we highlight that an additional benefit of the proposed adaptation methods is that the model retains high performance on the sentence-level setup, in contrast with the flair forward uni-directional lstm, which performs between 1.89 (spanish) and 4.34 (english) worse on the sentence-level. finally, the results of our proposed model are actually marginally better than the state-of-the-art approach of bilstm+crf using flair embeddings on german (+0.45 f1) and spanish (+0.73 f1), albeit this was not the original goal of our additions. this highlights that the proposed modelling ideas are more generally beneficial as they push the model to learn more robust representations. ## error analysis finally, we perform error analysis to identify the limitations of our approaches. ## untyped entity detection named entity recognition combines two different components: entity detection -identifying the named entity spans -and typing -identifying entity types. we study the impact of typing in the asyou-type setup by removing the typing information from the output labels (@xmath8, thus reducing the output space to @xmath9, i-ent, o}. results using the best as-you-type models with and without as-you-type adaptations are shown in ## conclusions this paper introduced and motivated the as-youtype experimental setup for the popular ner task. we presented results across three different languages, which show the extent to which sentencelevel state-of-the-art models degrade in this setup. through insights gained from data analysis, we proposed modelling improvements to further reduce the gap to the regular sentence-level performance. our error analysis highlights the cases that pose challenges to the as-you-type scenario and uncovers insights into way to further improve the modelling of this task. this setup is tailored for end-applications such as text editors, speech-to-text, machine translation, auto-completion, or auto-correct. for text editors, the editor would be able to receive suggestions for entities inline, right after they type the entity, which can further be coupled with a linking algorithm. this would increase the user experience and efficiency of the editor, as they can make selections about entities inline (similar to a phone's autocorrect), rather than having to go back over the entire sentence after it was completed. another avenue of future work would be to couple the ner as-you-type with asr data and using methods that adapt ner to noisy asr input @xcite for building an end-to-end live speech to entities system.