From SPMRL to NMRL : What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages ( MRL s)? (2020)

Abstract: It has been exactly a decade since the first establishment of SPMRL, a research initiative unifying multiple research efforts to address the peculiar challenges of Statistical Parsing for Morphologically-Rich Languages (MRLs). Here we reflect on parsing MRLs in that decade, highlight the solutions and lessons learned for the architectural, modeling and lexical challenges in the pre-neural era, and argue that similar challenges re-emerge in neural architectures for MRLs. We then aim to offer a climax, suggesting that incorporating symbolic ideas proposed in SPMRL terms into nowadays neural architectures has the potential to push NLP for MRLs to a new level. We sketch a strategies for designing Neural Models for MRLs (NMRL), and showcase preliminary support for these strategies via investigating the task of multi-tagging in Hebrew, a morphologically-rich, high-fusion, language.

## introduction the ability to process natural language data and to automatically extract structured meanings out of them has always been the hallmark of artificial intelligence (ai), and today it is also of immense practical value in downstream technological applications for information extraction, text analytics, and diverse data science applications. the introduction of deep learning models @xcite into natural language processing (nlp) has led to an explosion in the neural models and pretraining techniques applied to nlp tasks -from classical tasks as tagging and parsing to end-toend tasks as machine translation and question answering -raising the performance bar on these tasks to an all-times peak. so far though, these advances have been reported mostly for english. can these advances carry over to languages that are typologically vastly different from english, such as morphologically-rich languages? the term morphologically-rich languages (mrls) refers to languages such as arabic, hebrew, turkish or maltese, in which significant information is expressed morphologically, e.g., via word-level variation, rather than syntactically, e.g., via fixed word-order and periphrastic constructions, as in english. these properties lead to diverse and ambiguous structures, accompanied with huge lexica, which in turn make mrls notoriously hard to parse @xcite . a decade ago, @xcite put forth three overarching challenges for the mrls research community: for nlp in the pre-neural era, effective solutions have been proposed and successfully applied to address each of these challenges for mrls, using data from mrls treebanks and designated shared tasks @xcite @xcite . the solutions proposed to the above challenges included: (i) parsing morphemes rather than words, (ii) joint modeling of local morphology and global structures, and (iii) exploiting external knowledge to analyze the long tail of unattested word-forms. upon the introduction of neural network models into nlp @xcite , it was hoped that we could dispense with the need to model different languages differently. curiously though, this has not been the case. languages with rich morphology typically require careful treatment, and often the design of additional resources (cf. @xcite ). moreover, current modeling strategies for neural nlp appear to stand in contrast with the pre-neural proposals for processing mrls. first, unsupervised pre-training techniques employing language modeling objectives (lm, mlm) are applied nowadays to raw words rather than morphemes, and deliver word-embeddings agnostic to internal structure. while some morphological structure may be implicitly encoded in these vectors, the morphemes themselves remain un-accessible @xcite . second, pre-neural models for parsing mrls call for joint inference over local and global structures, tasking multiple, ambiguous, morphological analyses (a.k.a. lattices) as input, and disambiguating these morphological structure jointly with the parsing task @xcite @xcite @xcite . in contrast, pre-trained embeddings select a single vector for each input token -prior to any further analysis. finally, pre-trained embeddings trained on words cannot assign vectors to unseen words. the use of unsupervised char-based or sub-word units @xcite to remedy this situation shows mixed results; while these models learn orthographic similarities between seen and unseen words, they fail to learn the functions of sub-word units @xcite ; @xcite and references therein). this paper aims to underscore the challenges of processing mrls, reiterate the lessons learned in the pre-neural era, and establish their relevance to mrl processing in neural terms. on the one hand, technical proposals as pre-trained embeddings, finetuning, and end-to-end modeling, have advanced nlp greatly. on the other hand, neural advances often overlook mrl complexities, and disregard strategies that were proven useful for mrls in the past. we argue that breakthroughs in neural models for mrls (nmrl) can be obtained by incorporating symbolic knowledge and pre-neural strategies into the end-to-end neural architectures. the remainder of this paper is organized as follows. in section 2 we survey the methodological changes that neural modeling brought into nlp. in section 3 we characterize mrls and qualify the challenges that they pose to neural nlp. in section 4 we assess the compatibility of pre-neural modeling and current neural modeling practices for mrls, and in section 5 we suggest to re-frame pre-neural solution strategies in neural terms. in section 6 we present preliminary empirical support for these strategies, and in section 7 we conclude. ## the backdrop: from classical natural language processing to end-to-end deep learning classical nlp research has been traditionally devoted to the development of computer programs called parsers, that accept an utterance in a human language as input and deliver its underlying linguistic structure as output. the output may be of various sorts: morphological parsing analyzes the internal structure of words. syntactic parsing analyses the structure of sentences. semantic parsing assigns a formal representation to the utterance, one that reflects its meaning. discourse parsing identifies the discourse units, discourse relations, as well as rhetoric and pragmatic structure associated with complete narratives. since natural language exhibits ambiguity at all levels of analysis, statistical parsers aim to learn how to pick the best analysis from multiple suitable candidates @xcite . the introduction of deep learning has revolutionized all areas of artificial intelligence, and nlp research is no exception @xcite . neuralnetwork models now demonstrate an all-times peak in the performance of various nlp tasks, from conventional tasks in the nlp pipeline like tagging and parsing @xcite @xcite to diverse downstream applications, such as machine translation @xcite , question answering @xcite ), text-to-code generation @xcite and natural language navigation @xcite . in addition to revolutionizing empirical nlp, neural models have also altered the methodology of conducting nlp research, in various ways, which we review here in turn. first, while state-of-the-art models for structure prediction in nlp used to rely heavily on intricate formal structures and carefully designed features (or feature-templates) @xcite , current neural models provide a form of representation learning and may be viewed as automatic feature-extractors (kiperwasser and @xcite . that is, as long as the input object can be represented as a vector, the neural model will learn how to map it to the appropriate set of structural decisions, without having to write features or feature-templates by hand. second, most neural models for nlp rely on pre-training, the process of acquiring word-level vector representations termed word-embeddings. these vectors are used as input, instead of actual words. initially, word embeddings were noncontextualized @xcite , i.e., they assigned the same vector to the occurrences of a word in different contexts. later models present contextualized embeddings @xcite @xcite , they assign different vectors to the occurrences of the same word in different contexts. embeddings in general, and contextualized ones in particular, dramatically increased the performance of any nlp task they were applied to. third, working with contextualized embeddings has been so successful, that it shifted the focus of nlp practitioners from training models from scratch to fine-tuning @xcite pre-trained embeddings. that is, instead of tailoring hugely complex models for specific tasks and training them from scratch, a huge effort is invested in learning a general language model (lm) that can assign contextualized embeddings to words. these vectors are often argued to capture, or encode, various aspects of structure and meaning @xcite , and then, a relatively small amount of task-specific data may be used to fine-tune the pretrained embeddings, so that the model can solve a particular task at hand. finally, traditional nlp tasks, such as the parsing layers mentioned earlier, were typically organized into a pipeline turning unstructured texts gradually into more complex structures by gradually increasing the complexity of analysis. eventually, complex semantic structures formed the basis for the design of dialogue systems, question answering systems, etc. nowadays, nn models for complex semantic tasks are often designed and trained end-to-end (e2e) on examples of inputoutput pairs. there is an implicit assumption that all relavnt linguistic features are already encoded in the pre-trained representations, and that they will be automatically extracted in the learning process. this methodology of pre-training, automatic feature extraction and fine-tuning has been applied to a wide variety of tasks and saw immense success for english -and also for similar languages. notwithstanding, the majority of achievements and results for complex natural language understanding (nlu) does not yet carry over to all languages, and in particular, for languages known as morphologically-rich languages. ## the challenge: nlp for morphologically-rich languages the term morphologically-rich languages entered the nlp research community about a decade ago @xcite bringing to the forefront of the research a set of languages which are typologically different from english and share a host of similar processing challenges. subsequent spmrl events and shared tasks @xcite @xcite illustrated how methodologies and modeling assumptions for english nlp often break down in the face of such typologically diversity. that is, while most nlp models can in principle be trained on data in any given language, foot_0 such models are often developed with english in mind, and the bias injected into such models is not optimal for languages that exhibit flexible word order, and rich word-internal structure, as is the case in mrls. let us briefly survey the properties of mrls and the challenges associated with them, and observe how pre-neural studies proposed to address them. the essence of mrls. the term morphologically rich languages (mrls) refers to languages in which significant information regarding the units in the sentence and the relations between them is expressed morphologically, i.e., via word structure, rather than syntactically, e.g., using word order and rigid structures. morphologically-marked information may be of various sorts. for example, consider the following hebrew sentence: 2 (1) hild hpil at hspr fl hildh. there are several lessons to be learned from ( 1 ). first note that the 6 tokens in hebrew correspond to 9 tokens in the english translation -we can observe three types of morphological phenomena that has led to this. first, elements such as prepositions, relativizers and the definite markers h (the) in hebrew always attach as clitics to lexical hosts, and do not stand on their own. second, features as gender, number, person, tense etc. are marked by in-flectional morphemes. in particular, the final h distinguishes ildh kid.fem from its ild kid.masc counterpart. interestingly, an initial h marks definiteness in hild, hspr and hildh, so there is no 1:1 relation between surface elements (chars) and what they can mark. finally, the hebrew verb, hpil, which also begins with an h, corresponds to the construction ("binyan", pattern) 'cause-to-fall' via a derivational morphological process that combines the pattern h i (causative) and the lexical root n.p.l (to fall). note that the h i causative morpheme is non-concatenative. moreover, when combining h i + n.p.l into hpil the n drops, leaving only a part of the root explicit. this word-level complexity then requires decomposition of raw surface tokens into constituent morphemes in order to transfer them to the syntactic, semantic, or downstream tasks that require this information. however, rich morphology may lead to extreme ambiguity in the decomposition of tokens into morphemes. take for example the two occurrences of the word form hpil in (2): two different morphological processes lead to two different decompositions of hpil, one is concatenative: "the" + "elephant" (h+pil) and one is not: "cause-to" + "fall" (h i + n.p.l). moreover, neither interpretation is a-priory more likely than the other. we need the global context in order to select the single human-perceived analysis for each form. the typology of mrls. the extent to which morphological phenomena is reflected in different languages varies, and linguistic typology describes morphological diversity along two dimensions. one is the synthesis dimension, which captures the ratio of morphemes per word. isolating languages on one end present one-morpheme-perword, like most words in english. at the other end we have polysynthetic languages, where multiple morphemes can form a single word, as it is in turkish. the other dimension is fusion, and it refers to how easy it is to decompose the word into morphemes. in turkish, which is agglutinative, the segmentation into morphemes is rather straightforward. this stands in contrast with fusional languages, such as hebrew, where the decomposition of a word like hpil is less trivial due to the intricate 'fusion' processes that went into creation. key challenges in nlp for mrls the linguistic characteristics of mrls are known to pose challenges to the development of nlp models, shared across languages and tasks. the overarching challenges are summerized in tsarfaty et al. (2010): (i) the architectural challenge: what are the units that should enter as input into the nlp pipeline for mrls? are they words? morphemes? how are these units identified and propagated down the pipeline? (ii) the modeling challenge: what are the modeling assumptions that are appropriate for models for mrls? what kind of structure representations and features (or featuretemplates) are appropriate? (iii) the lexical challenge: how can pre-neural solutions in nlp for mrls. let us now survey the solutions proposed for these three overarching challenges in the pre-neural era. in response to the architectural challenge, several input alternatives have been proposed. the input to processing an mrl can be composed of raw tokens, segmented morphemes, or complete morphological lattices that capture the multiple possible analyses for each input tokens @xcite . morphological lattices seem particularly advantageous, since on the one hand they represent the explicit decomposition of words into morphemes, and on the other hand retain the morphological ambiguity of the input stream, to be disambiguated downstream, when information from later phases, syntactic or semantic, becomes available. lattice-based processing has led to re-thinking the modeling architectures for mrls, and to propose joint models, where multiple levels of information are represented during training, and are jointly predicted at inference time. such joint models have been developed for mrls in the context of phrase-structure parsing @xcite @xcite and dependency parsing @xcite @xcite . in all cases, it has been shown that joint models obtain better results than their morphological or syntactic standalone counterparts. 3 finally, the lexical challenge refers to the problem of out-of-vocabulary items. supervised training successfully analyzes attested forms, but fails to analyze the long tail of morphological forms in the language, not yet attested during training. pre-neural models for mrls thus benefit from additional symbolic information beyond the supervised data. it can be in the form of online dictionaries, wide-coverage lexica, or a-priori knowledge of the structure of morphological paradigms in the language @xcite . where we're at upon the introduction of neural models into nlp the hope was that we could dispense with the need to develop language-specific modeling strategies, and that models will seamlessly carry over from any one language (type) to another. curiously, this was not yet shown to be the case. nlp advances in mrls still lag behind those for english, with lower empirical results on classical tasks @xcite , and very scarce results for applications as question answering and natural language inference @xcite . more fundamentally, nlp researchers nowadays successfully predict linguistic properties of english via neural models as in @xcite ; gulordava et al. ( 2018 ), but they are less successful in doing so for languages that differ from english, as in @xcite . it is high time for the mrl community to shed light on the methodological and empirical gaps between neural models for english and for mrls, and to bridge this gap. ## the research objective: nlp for mrls in the deep learning era the point of departure of this paper is the claim that neural modeling practices employed in nlp nowadays are suboptimal in the face of properties of mrls. in what follows we illuminate this claim for the four neural methodological constructs that we termed pre-training, fine-tuning, feature-extraction and end-to-end modeling. pre-training of word embeddings presupposes that the input to an nlp architecture consists of raw words. however, word-level embeddings may not be useful for tasks that require access to the actual morphemes. for example, for semantic tasks in mrls, it is often better to use morphological embeddings of lemmas rather than words @xcite . also, dependency parsing for mrls requires access to morphological segments, according to the ud scheme @xcite . a reasonable solution might be to morphologically analyze and segment all input words prior to pre-training. unfortunately, this solution does not fit the bill for mrls either. first, current neural segmentors and taggers for mrls are not accurate enough, and errors in the analyses propagate through the pre-training to contaminate the trained embeddings and later tasks. in the universal segmentation work of @xcite , for instance, neural segmentation for languages which are high on both the synthesis and the fusion index, such as arabic and hebrew, lags far behind. beyond that, there is the technical matter of resources. pre-training models as @xcite al. (2018); liu et al. (2019b); yang et al. (2019) requires massive amounts of data and computing resources, and such training often takes place outside of academia. training morphological embeddings rather than word embeddings was not taken up by any commercial partner. 4 next, let us turn to the notion of fine-tuning, widely used today in all sorts of nlp tasks, typically in conjunction with contextualized embeddings as @xcite @xcite ). an argument may be advanced that contextualized embeddings actually encode accurate disambiguated morphological analyses in their context-based representations, and all we have to do is to probe these vectors and make these morphological analyses explicit. this argument is appealing, but it was never seriously tested empirically, and it is an open question whether we can successfully probe the fine-grained morphological functions from these vectors. a possible caveat for this line of research has to do with the inner-working of contextualized representations. most contextualized embeddings operate not on words but on word-pieces. a wordpieces algorithm breaks down words into subwords, and the model assigns vectors to them. the word-pieces representations are later concatenated or pooled together to represent complete words. it is an open question whether these word-pieces capture relevant aspects of morphology. in particular, it is unclear that the strategy of relying on chars or char-strings is adequate for encoding nonconcatenative phenomena that go beyond simple character sequencing, such as templatic morphology, substraction, reduplication, and more (ackerman and @xcite . the notion of word-pieces leads us to consider the lexical challenge. the suggestion to use sub-word units (chars or char n-grams) rather than words could naturally help in generalizing from seen to unseen word tokens. there is a range of subword units that are currently employed (chars, chargrams, bpes @xcite ), nicely compared and contrasted by @xcite . @xcite ; @xcite show that for the type of sub-word units that are currently used, standard pre-training leads to clustering words that are similar orthographically, and do not necessarily share their linguistic functions. when a downstream task requires the morphological signature (e.g., dependency parsing in @xcite ) this information is not recoverable from models based on sub-word units alone. on the whole, it seems that end-to-end modeling for mrls cannot completely rely on automatic feature extraction and dispense with the need to explicitly model morphology. it is rather the contrary. explicit morphological analyses provide an excellent basis for successful feature extraction and accurate downstream tasks. when such analysis is missing, results for mrls deteriorate. so, we should aim to recover morphological structures rather than ignore them, or jointly infer such information together with the downstream tasks. 5 a different, however related, note concerning automatic feature extraction in mrls has to do with the flexible or free word-order patterns that are exhibited by many mrls. many neural models rely on rnns @xcite for feature extraction. these models assume complete linear ordering of the words and heavily rely on positions in the process of representation learning. even pre-training based on attention and self-attention @xcite assign weights to positional embeddings. in this sense, the bias of current neural models to encode positions stands in contrast with the properties of mrls, that often show discrepancies between the linear position of words and their linguistic functions. it is an open question whether there are more adequate architectures for training (or pre-training) for more flexible or free word-order languages. ## research questions and strategies the overarching goal the purpose of the proposed research theme, which we henceforth refer to as neural models for mrls (nmrl), is to devise modeling strategies for mrls, for classical nlp tasks (tagging, parsing) and for downstream language understanding tasks (question answering, information extraction, nl inference, and more). this research diverges from the standard methodology of applying dl for nlp in three ways. first, current end-to-end neural models for complex language understanding are developed mostly for english @xcite . here we aim to situate neural modeling of natural language understanding in cross-linguistic settings (e.g., @xcite ). second, while current neural models for nlp assume pre-training with massive amounts of unsupervised data @xcite @xcite , research on mrls might be realistically faced with resource-scarce settings, and will require models that are more "green" @xcite . finally, while many neural-based models developed for english presuppose that linguistic information relevant for the downstream task is implicitly encoded in word vectors, and may be successfully predicted by neural models @xcite , we question the assumption that ready-made pre-trained embeddings, will indeed encode all relevant information required for end-to-end models in mrls. the key strategies we propose in order to address nmrl include transitionining to (i) morphologicalembeddings, (ii) joint lattice-based modeling, and (iii) paradigm cell-filling @xcite , as we detail shortly. ## preliminary empirical evidence goal. in this section we aim to empirically assess the ability of neural models to recover the wordinternal structure of morphologically complex and highly ambiguous surface tokens in modern hebrew. hebrew is a semitic language which lies high on both the synthesis and fusion typological indices, and thus provides an interesting case study. specifically, we devised a multi-tagging task where each raw input token is tagged with the sequence of part-of-speech tags that represent the functions of its constituent morphemes. for example, the token hpil in section 3 can assume two different multi-tag analyses: verb (made-fall) or det+noun (the elephant). the number of distinct tags in the multi-tagging analyses of hebrew tokens can be up to seven different tags, that represent distinct functions contained in the word token. ## discussion and conclusion this paper proposes nmrl, a new (or rather, redefined) research theme aiming to develop neural models, benchmarks, and modeling strategies for mrls. we surveyed current research practices in neural nlp, characterized the particular challenges associated with mrls, and demonstrated that some of the neural modeling practices are incompatible with the accumulated wisdom concerning mrls in the spmrl literature. we proceeded to define the three deep counterparts to the challenges proposed in @xcite , namely, the deep architectural challenge, deep modeling challenge and deep lexical challenge, and sketched plausible research avenues that the nmrl community might wish to explore towards their resolution. our preliminary experiments on hebrew multitagging confirmed that relying on lessons learned for mrls in the pre-neural era and incorporating similar theoretical constructs into the neural architecture indeed improves the empirical results on multi-tagging of hebrew, on the very basic form of analysis of modern hebrew -a morphologically rich and highly-fusional language. this type of research needs to be extended to the investigation of multiple tasks, multiple languages, and multiple possible pre-training regimes (words, chars, morphemes, lattices) in order to investigate whether this trend extends to other languages and tasks. whether adopting solution strategies for mrls proposed herein or devising new ones, it is high time to bring the linguistic and moprhological complexity of mrls back to the forefront of nlp research, both for the purpose of getting a better grasp of the abilities, as well as limitations, of neural models for nlp, and towards serving the exciting nlp/ai advances to the understudied, less-privileged, languages.