Neural Modeling for Named Entities and Morphology ( NEMO 2) (2021)

Abstract: Named Entity Recognition (NER) is a fundamental NLP task, commonly formulated as classification over a sequence of tokens. Morphologically rich languages (MRLs) pose a challenge to this basic formulation, as the boundaries of named entities do not necessarily coincide with token boundaries, rather, they respect morphological boundaries. To address NER in MRLs we then need to answer two fundamental questions, namely, what are the basic units to be labeled, and how can these units be detected and classified in realistic settings (i.e., where no gold morphology is available). We empirically investigate these questions on a novel NER benchmark, with parallel token- level and morpheme-level NER annotations, which we develop for Modern Hebrew, a morphologically rich-and-ambiguous language. Our results show that explicitly modeling morphological boundaries leads to improved NER performance, and that a novel hybrid architecture, in which NER precedes and prunes morphological decomposition, greatly outperforms the standard pipeline, where morphological decomposition strictly precedes NER, setting a new performance bar for both Hebrew NER and Hebrew morphological decomposition tasks.

## introduction named entity recognition (ner) is a fundamental task in the area of information extraction (ie), in which mentions of named entities (ne) are extracted and classified in naturally occurring texts. this task is most commonly formulated as a sequence labeling task, where extraction takes the form of assigning each input token with a label that marks the boundaries of the ne (e.g., b, i, o), and classification takes the form of assigning labels to indicate entity type @xcite . despite a common initial impression from latest ner performance, brought about by neural models on the main english ner benchmarks-conll 2003 @xcite and onto-notes @xcite -the ner task in real-world settings is far from solved. specifically, ner performance is shown to greatly diminish when moving to other domains @xcite , when addressing the long tail of rare, unseen, and new user-generated entities @xcite , and when handling languages with fundamentally different structure than english. in particular, there is no readily available and empirically verified neural modeling strategy for neural ner in those languages with complex word-internal structure, also known as morphologically rich languages. morphologically rich languages (mrl) @xcite @xcite are languages in which substantial information concerning the arrangement of words into phrases and the relations between them is expressed at the word level, rather than in a fixed word-order or a rigid structure. the extended amount of information expressed at word-level and the morpho-phonological processes creating these words result in high token-internal complexity, which poses serious challenges to the basic formulation of ner as classification of raw, space-delimited, tokens. specifically, while ner in english is formulated as the sequence labeling of space-delimited tokens, in mrls a single token may include multiple meaning-bearing units, henceforth morphemes, only some of which are relevant for the entity mention at hand. in this paper we formulate two questions concerning neural modeling strategies for ner in mrls, namely: (i) what should be the granularity of the units to be labeled? space-delimited tokens or finer-grain morphological segments? and, (ii) how can we effectively encode, and accurately detect, the morphological segments that are relevant to ner, and specifically in realistic settings, when gold morphological boundaries are not available? to empirically investigate these questions we develop a novel parallel benchmark, containing parallel token-level and morpheme-level ner annotations for texts in modern hebrew-a morphologically rich and morphologically ambiguous language, which is known to be notoriously hard to parse @xcite . our results show that morpheme-based ner is superior to token-based ner, which encourages a segmentation-first pipeline. at the same time, we demonstrate that token-based ner improves morphological segmentation in realistic scenarios, encouraging a ner-first pipeline. while these two findings may appear contradictory, we aim here to offer a resolution; a hybrid architecture where the token-based ner predictions precede and prune the space of morphological decomposition options, while the actual morpheme-based ner takes place only after the morphological decomposition. we empirically show that the hybrid architecture we propose outperforms all token-based and morpheme-based model variants of hebrew ner on our benchmark, and it further outperforms all previously reported results on hebrew ner and morphological decomposition. our error analysis further demonstrates that morpheme-based models better generalize, that is, they contribute to recognizing the long tail of entities unseen during training (out-of-vocabulary, oov), in particular those unseen entities that turn out to be composed of previously seen morphemes. the contribution of this paper is thus manifold. first, we define key architectural questions for neural ner modeling in mrls and chart the space of modeling options. second, we deliver a new and novel parallel benchmark that allows one to empirically compare and contrast the morpheme vs. token modeling strategies. third, we show consistent advantages for morpheme-based ner, demonstrating the importance of morphologically aware modeling. next we present a novel hybrid architecture which demonstrates an even further improved performance on both ner and morphological decomposition tasks. our results for hebrew present a new bar on these tasks, outperforming the reported state-of-the-art results on various benchmarks. 1 2 research questions: ner for mrls in mrls, words are internally complex, and word boundaries do not generally coincide with the 1 data & code: https://github.com/onlplab /nemo . it is clear that ‫/תאילנד‬thailand (thailand) and ‫/סי‬sin (china) are nes, and in english, each ne is its own token. in the hebrew phrase though, neither ne constitutes a single token. in either case, the ne occupies only one of two morphemes in the token, the other being a case-assigning preposition. this simple example demonstrates an extremely frequent phenomenon in mrls such as hebrew, arabic, or turkish, that the adequate boundaries for nes do not coincide with token boundaries, and tokens must be segmented in order to obtain accurate ne boundaries. 3 the segmentation of tokens and the identification of adequate ne boundaries is, however, far from trivial, due to complex morpho-phonological and orthographic processes in some mrls @xcite . this means that the morphemes that compose nes are not necessarily transparent in the character sequence of the raw tokens. consider for example phrase (2): here, the full form of the ne ‫הלב‬ ‫הבית‬ / habayit halavan (the white house), is not present in the utterances, only the sub-string ‫הלב‬ ‫בית‬ / bayit halavan ((the) white house) is present in (2)due to phonetic and orthographic processes suppressing the definite article ‫/ה‬ha in certain environments. in this and many other cases, it is not only that ne boundaries do not coincide with token boundaries, they do not coincide with characters or sub-strings of the token either. this calls for accessing the more basic meaning-bearing units of the token, that is, to decompose the tokens into morphemes. unfortunately though, the morphological decomposition of surface tokens may be very challenging due to extreme morphological ambiguity. the sequence of morphemes composing a token is not always directly recoverable from its character sequence, and is not known in advance. 4 this means that for every raw space-delimited token, there are many conceivable readings which impose different segmentations, yielding different sets of potential ne boundaries. consider for example the token ‫לבני‬ (lbny) in different contexts: (3) (a) ‫לבני‬ ‫השרה‬ hasara livni the-minister [livni] p er 'minister [livni] p er ' (b) ‫גנ‬ ‫לבני‬ le-beny gantz for-[benny gantz] p er 'for [benny gantz] p er ' (c) ‫היקר‬ ‫לבני‬ li-bni hayakar for-son.poss.1sg the-dear 'for my dear son' (d) ‫חימר‬ ‫לבני‬ livney kheymar brick.cs clay 'clay bricks' in (3a) the token ‫לבני‬ is completely consumed as a labeled ne. in (3b) ‫לבני‬ is only partly consumed by an ne, and in (3c) and ( 3d ) the token is entirely out of an ne context. in (3c) the token is composed of several morphemes, and in (3d) it consists of a single morpheme. these are only some of the possible decompositions of this surface token, other alternatives may still be available. as shown by @xcite , @xcite , @xcite , @xcite , @xcite , and others, the correct morphological decomposition becomes apparent only in the larger (syntactic or semantic) context. the challenge, in a nutshell, is as follows: in order to detect accurately ne boundaries, we need to segment the raw token first, however, in order to segment tokens correctly, we need to know the greater semantic content, including, for example, the participating entities. how can we break out of this apparent loop? finally, mrls are often characterized by an extremely sparse lexicon, consisting of a long-tail of oov entities unseen during training @xcite . even in cases where all morphemes are present in the training data, morphological compositions of seen morphemes may yield tokens and entities which were unseen during training. take for example the utterance in (4), which the reader may inspect as familiar: given the aforementioned complexities, in order to solve ner for mrls we ought to answer the following fundamental modeling questions: q1. units: what are the discrete units upon which we need to set ne boundaries in mrls? are they tokens? characters? morphemes? a representation containing multiple levels of granularity? q2. architecture: when employing morphemes in ner, the classical approach is ''segmentationfirst''. however, segmentation errors are detrimental and downstream ner cannot recover from them. how is it best to set up the pipeline so that segmentation and ner could interact? q3. generalization: how do the different modeling choices affect ner generalization in mrls? how can we address the long tail of oov nes in mrls? which modeling strategy best handles pseudo-oov entities that result from a previously unseen composition of already seen morphemes? ## the data: a novel ner corpus this work empirically investigates ner modeling strategies in hebrew, a semitic language known for its complex and highly ambiguous morphology. @xcite , the only previous work on hebrew ner to date, annotated space-delimited tokens, basing their guidelines on the conll 2003 shared task @xcite . popular arabic ner corpora also label spacedelimited tokens (anercorp @xcite , aqmar @xcite , tweets @xcite ), with the exception of the arabic portion of ontonotes @xcite and ace @xcite which annotate ner labels on gold morphologically pre-segmented texts. however, these works do not provide a comprehensive analysis on the performance gaps between morpheme-based and token-based scenarios. in agglutinative languages as turkish, token segmentation is always performed before ner @xcite , re-enforcing the need to contrast the token-based scenario, widely adopted for semitic languages, with the morpheme-based scenarios in other mrls. our first contribution is thus a parallel corpus for hebrew ner; one version consists of gold-labeled tokens and the other consists of gold-labeled morphemes, for the same text. for this, we performed gold ne annotation of the hebrew treebank @xcite , based on the 6,143 morpho-syntactically analyzed sentences of the haaretz corpus, to create both token-level and morpheme-level variants, as illustrated at the topmost and lowest rows of annotation scheme we started off with the guidelines of @xcite , from which we deviate in three main ways. first, we label ne boundaries and their types on sequences of morphemes, in addition to the space-delimited token annotations. 6 secondly, we use the finer-grained entity categories list of ace (ldc, 2008). 7 finally, we allow nested entity mentions, as in finkel and @xcite and @xcite . 8 annotation cycle as @xcite put it, examples and rules would never cover all possible cases because of the specificity of natural language and the ambiguity of formulation. to address this we employed the cyclic approach of agile annotation as offered by @xcite . every cycle consisted of: annotation, evaluation and curation, clarification and refinements. we used webanno @xcite as our annotation interface. the initial annotation cycle was a two-stage pilot with 12 participants, divided into 2 teams of 6. the teams received the same guidelines, with the exception of the specifications of entity boundaries. one team was guided to annotate the minimal string that designates the entity. the other was guided to tag the maximal string which can still be considered as the entity. our agreement analysis showed that the minimal guideline generally led to more consistent annotations. based on this result (as well as low-level refinements) from the pilot, we devised the full version of the guidelines. 9 annotation, evaluation, and curation: every annotation cycle was performed by two annotators (a, b) and an annotation manager/curator (c). we annotated the full corpus in 7 cycles. we evaluated the annotation in two ways, manual curation and automatic evaluation. after each annotation step, the curator manually reviewed every sentence in which disagreements arose, as well as specific points of difficulty pointed out by the annotators. the inter-annotator agreement metric described below was also used to quantitatively gauge the progress and quality of the annotation. ## conclusion this work addresses the modeling challenges of neural ner in mrls. we deliver a parallel tokenvs-morpheme ner corpus for modern hebrew, that allows one to assess ner modeling strategies in morphologically rich-and-ambiguous environments. our experiments show that while ner benefits from morphological decomposition, downstream results are sensitive to segmentation errors. we thus propose a hybrid architecture in which ner precedes and prunes the morphological decomposition. this approach greatly outperforms a standard pipeline in realistic (non-gold) scenarios. our analysis further shows that morpheme-based models better recognize oovs that result from morphological composition. all in all we deliver new state-of-the-art results for hebrew ner and md, along with a novel benchmark, to encourage further investigation into the interaction between ner and morphology. webanno: a flexible, web-based and visually supported system for distributed annotations. in proceedings of the 51st annual meeting of the association for computational linguistics: system demonstrations, pages 1-6, sofia, bulgaria. association for computational linguistics. ziqi zhang. 2013. named entity recognition: challenges in document annotation, gazetteer construction and disambiguation. ph.d. thesis, university of sheffield.