Identifying named entities in written text is an essential component of the text processing pipeline used in applications such as text editors to gain a better understanding of the semantics of the text . However , the typical experimental setup for evaluating Named Entity Recognition ( NER ) systems is not directly applicable to systems that process text in real time as the text is being typed . Evaluation is performed on a sentence level assuming the end-user is willing to wait until the entire sentence is typed for entities to be identified and further linked to identifiers or coreferenced . We introduce a novel experimental setup for NER systems for applications where decisions about named entity boundaries need to be performed in an online fashion . We study how state-of-the-art methods perform under this setup in multiple languages and propose adaptations to these models to suit this new experimental setup . Experimental results show that the best systems that are evaluated on each token after its typed , reach performance within 1 - 5 F 1 points of systems that are evaluated at the end of the sentence . These show that entity recognition can be performed in this setup and open up the development of other NLP tools in a similar setup . Automatically identifying named entities such as organizations , people and locations is a key component in processing written text as it aids with understanding the semantics of the text . Named entity recognition is used as a pre-processing step to subsequent tasks such as linking named entities to concepts in a knowledge graph , identifying the salience of an entity to the text , identifying coreferential mentions , computing sentiment towards an entity , in question answering or for extracting relations . Figure 1 : An example of the proposed task and evaluation setup . After the word ' Foreign ' is typed , the model immediately predicts an NER label for this word , only using left context ( ' A spokesman for ' ) and the word itself . The prediction is then compared against the gold label to compute token-level F 1 score . This token 's prediction will not be changed , even if the model 's internal prediction for it can be revised later as more tokens are typed . Identifying named entities as they are typed benefits any system that processes text on the fly . Examples of such applications include : a ) News editorswhere named entities can be highlighted , suggested ( auto-completion ) , co-referenced or linked as the editor is typing ; b ) auto-correct -where named entities that are just typed are less likely to need correction as they may come from a different language or be out-of-vocabulary ( OOV ) ; c ) simultaneous machine translation -where translation of OOV named entities requires different approaches ; d ) live speech-to-text ( e.g. , TV shows ) -where named entities are more likely to be OOV , hence the transcription should focus more on the phonetic transcription rather than on n-gram language modelling . This paper introduces a novel experimental setup of Named Entity Recognition systems illustrated in Figure 1 . In this setup , inference about the span and type of named entities is performed for each token , immediately after it was typed . The sentence level tag sequence is composed through appending all individual token predictions as they were made . The current named entity recognition systems that are trained and evaluated to predict full sentences are likely to under-perform in this experimental setup as they : expect that right context is available , are faced with unseen types of inputs in the form of truncated sentences and can not reconcile the final sentence-level tag sequence across the entire sentence as the result may not be a valid sequence . The goal of this study is to present a comprehensive analysis of the task of NER in the as-you-type scenario , with the following contributions : a ) A novel experimental setup for conducting named entity recognition experiments , denoted as the as-you-type scenario ; b ) Experiments with state-of-the-art sentence-level approaches to named entity recognition in the as-you-type setup across three languages , which indicate a 1 - 5 F 1 points decrease compared to sentence-level inference ; c ) Tailored methods for as-you-type entity recognition models which reduce the gap to entire sentence-level inference by 9 - 23 % compared to regular approaches ; d ) An extensive analysis of existing data sets in the context of this task and model error analysis , which highlight future modelling opportunities . This paper introduced and motivated the as-youtype experimental setup for the popular NER task . We presented results across three different languages , which show the extent to which sentencelevel state-of-the-art models degrade in this setup . Through insights gained from data analysis , we proposed modelling improvements to further reduce the gap to the regular sentence-level performance . Our error analysis highlights the cases that pose challenges to the as-you-type scenario and uncovers insights into way to further improve the modelling of this task . This setup is tailored for end-applications such as text editors , speech-to-text , machine translation , auto-completion , or auto-correct . For text editors , the editor would be able to receive suggestions for entities inline , right after they type the entity , which can further be coupled with a linking algorithm . This would increase the user experience and efficiency of the editor , as they can make selections about entities inline ( similar to a phone 's autocorrect ) , rather than having to go back over the entire sentence after it was completed . Another avenue of future work would be to couple the NER as-you-type with ASR data and using methods that adapt NER to noisy ASR input ( Benton and Dredze , 2015 ) for building an end-to-end live speech to entities system .

Challenge: The current experimental setup and evaluation for named entity recognition systems run on complete sentences limiting the real-time applications that process texts as being typed.
Approach: They propose an experimental setup where systems are evaluated in an online fashion and perform analysis of state-of-the-art methods in multiple languages.
Outcome: They find that models degrade performance by 1 to 5 F1 points in their setup and the proposed tailored methods can reduce the gaps.