A Truly Joint Neural Architecture for Segmentation and Parsing (2024)

Abstract: Contemporary multilingual dependency parsers can parse a diverse set of languages, but for Morphologically Rich Languages (MRLs), performance is attested to be lower than other languages. The key challenge is that, due to high morphological complexity and ambiguity of the space-delimited input tokens, the linguistic units that act as nodes in the tree are not known in advance. Pre-neural dependency parsers for MRLs subscribed to the joint morpho-syntactic hypothesis , stating that morphological segmentation and syntactic parsing should be solved jointly, rather than as a pipeline where segmentation precedes parsing. However, neural state-of-the-art parsers to date use a strict pipeline. In this paper we introduce a joint neural architecture where a lattice-based representation preserving all morphological ambiguity of the input is provided to an arc-factored model, which then solves the morphological segmentation and syntactic parsing tasks at once. Our experiments on Hebrew, a rich and highly ambiguous MRL, demonstrate state-of-the-art performance on parsing, tagging and segmentation of the Hebrew section of UD, using a single model. This proposed architecture is LLM-based and language agnostic, providing a solid foundation for MRLs to obtain further performance improvements and bridge the gap with other languages.

## introduction dependency parsing is the task of automatically analyzing the syntactic structure of a sentence and exposing the functional relationships between its words. in the past, dependency parsing was shown to be extremely useful for many language processing tasks, as machine translation @xcite , question answering @xcite and information extraction @xcite , to name a few. while nowadays many english nlp tasks are solved end-to-end using large language models (llms) and without accessing any symbolic structure, for low-and mediumresource languages, parsers are still indispensable, enabling a host of downstream applications. most neural state-of-the-art dependency parsers to date presuppose a pipeline architecture @xcite @xcite that includes several analysis stages -tokenization, word segmentation, partof-speech (pos) tagging, morphological feature tagging, dependency parsing, and sometimes also named entity recognition -and the linguistic features from each stage are provided as input to the tasks that follow it, and contribute to the overall efficacy. in morphologically-rich languages (mrls), many raw space-delimited tokens consist of multiple units, each of which serves a distinct role in the overall syntactic representation @xcite . consequently, segmentation is essential for accurate mrl parsing. however, due to high morphological ambiguity, when segmentation is performed prior to (and independently of) the parsing phase, segmentation errors may propagate to undermine the syntactic predictions, and subsequently lead to an overall incorrect parse. according to the joint hypothesis, that was heavily populated in parsing studies for mrls in the preneural era @xcite @xcite @xcite , morphological segmentation and syntactic predictions are mutually dependent, and hence, these two tasks should be solved together. following these lines, @xcite developed a joint morpho-syntactic transition-based parser that achieved state-of-the-art (sota) results on hebrew parsing. this system employs a morphological lattice as input for a transition system, with both syntactic and morphological transitions, for picking the right arcs and segments in tan-dem. another influential work is that of @xcite , that displays all potential segments that could be involved in any token's analysis, and allows an mst graph-based parser pick the highest scoring subset of arcs and nodes as the output dependency tree, showing parsing improvements for both hebrew and turkish. similarly, @xcite contributed a nonneural graph-based parser for sanskrit which employ energy-based modeling to determine the optimal path on a graph which jointly represents a valid segmentation and syntactic analysis. however, this architecture is non-neural and sanskrit specific, relying on labor-intensive hand-crafted feature engineering. in neural settings, and still for sanskrit, @xcite introduced a pretraining approach which focuses on amalgamating word representations generated by encoders trained on auxiliary tasks, such as morphological and syntactic tags. unlike the present work, this neural architecture does not make any segmentation decisions nor does it leverage the lattice structure for joint segmentation and parsing. all in all, in the case of neural multilingual dependency parsers, the pipeline approach of segment-then-parse is fully maintained @xcite , and no language-agnostic architecture for morphological segmentation and syntactic parsing is yet to be found. in this paper we revisit the joint hypothesis as a viable way to improve neural dependency parsing for mrls. the idea, in a nutshell, is as follows. we start off with an arc-factored model @xcite that accepts a sequence of words as input, and generates a dependency tree by picking the highest scoring arcs connecting all words. in our approach, the arc-factored model takes as input a linearized lattice containing all possible morphological segments that may potentially act as nodes, and learns to assign a head and label to each such node. during training, incorrect segments are mapped to an auxiliary node, of which subtree is excluded from the final dependency tree. at inference time then, the model maps relevant segments to the main branch and unused segments to the auxiliary branch, building a complete dependency tree. in this process, morphological segmentation decisions get informed by the syntactic arcs, and vice versa. we further extend the architecture with a multi-task learning (mtl) component to predict the features of each node, e.g., pos, gender, num-ber and person. our experiments on the hebrew section of ud foot_0 show that in cases where the input morphological analyses are complete, our model provides new and improved state of the art results for segmentation and parsing for hebrew, in a single, jointly trained, model. in the more realistic case, where some of the word lattices may lack possible analyses (the case of out of vocabulary (oov) tokens), the model still delivers competitive results for segmentation, tagging and parsing, outperforming the state-ofthe-art results of the de-facto standard pipelines, stanza and trankit. ## challenge and research objectives the goal of dependency parsing is to automatically analyze the syntactic structure of a sentence by indicating the functional relationship between its words. the input is assumed to be a sequence of space-delimited tokens that represent words, and the output is a directed tree where each input word serves as a node, and each arc represents a relation between two such words. an arc can be labeled to indicate the relation type between the two words. deep neural networks have recently achieved unprecedented results in many areas of natural language processing, including the dependency parsing task. the architecture of @xcite (that followed up on @xcite ) is currently accepted as the standard architecture for dependency parsing. dozat and manning present a simple neural architecture where an arc factored model selects the best set of dependency arcs and labels. this approach is the foundation of several sota dependency parsers, including stanza @xcite and trankit (minh nguyen and nguyen, 2021), which have been trained and successfully used across different languages. crucially, these parsers and others @xcite @xcite all subscribe to a pipeline approach, where the input tokens are pre-segmented, and these segments uniquely determine the nodes in the tree. this pipeline approach has been applied across many language types, including morphologically rich languages (mrl). however, mrls pose a significant challenge to such architectures. in a 2019 ). the upper tree illustrates the syntactic structure corresponding to "in their pleasant shadow", while the lower tree corresponds to "their onion was pleasant". this highlights the existence of multiple morphological decompositions and various potential dependency trees. pre-neural models addressed this challenge by jointly modeling morphological segmentation and dependency parsing, and have shown that it yields superior results for both tasks. the pressing research question at hand is whether this hypothesis can also be validated within the context of neural parsing architectures. in other words, can neural parsing models benefit from a joint approach to segmentation and parsing, similar to what has been observed in non-neural models? this paper addresses two primary objectives. firstly, we aim to introduce a unified neural architecture that jointly solves segmentation, tagging and parsing, with the aim of empirically validating the joint hypothesis within the realm of neural architectures. secondly, we seek to attain stateof-the-art (sota) results for hebrew, a language renowned for its formidable parsing challenges attributed to its substantial morphological ambiguity. ## the proposal: a model for joint morphological segmentation and syntactic dependency parsing task definition formally, our proposed model is defined as a structure prediction function f : @xmath0, where @xmath1, and @xmath2, which we refer to here as morphological segments. crucially, we retain morphological ambiguity, and deliver all possible analyses of s to the parser. hence, we assume a morphological analyzer (ma), that given an input sentence @xmath3, . . . , s k yields a token-lattice termed l @xmath4. the complete lattice of the input sentence l @xmath5 our structure prediction function becomes f : @xmath6, with l @xmath7. input linearization upon receiving an input lattice l s , we aim to linearize it in order to be able to encode it as an input vector for the neural architecture. as shall be seen shortly, the linearization is a critical phase for obtaining a neural encoding of the non-linear, morphologically ambiguous, input. we illustrate the linearization process using the hebrew sentence 'bkrti bbit hlbn' (lit: "i-visited in-the-house the-white", trans: "i visited the white house"). initially, the m a provides a list of all potential analyses for each token: bkrti: [('bkrti')], bbit: [('b', 'bit'), ('b', 'h', 'bit')], hlbn: [('h', 'lbn'), ('hlbn')]. subsequently, each token is linearized independently: bkrti: ['bkrti'], bbit: @xcite , hlbn: ['h', 'lbn', 'hlbn']. finally, all linearized analyses are concatenated according to the initial order: @xcite . formally, the linearize function takes an input lattice l s and returns a sequence of m analyses while maintaining the partial order of the tokens. within the input lattice l @xmath8, each m a(s j ) encompasses a comprehensive set of potential analyses -segmentation options -for the token s j . let k j be the number of analyses of the j th token; then, it holds that n @xmath9. also, let a i j be the i th analysis of the j th token. the linearization function works as follow: the number of morphemes in an analysis a j i is r (i,j) , denoted as consequently, the total number of morphemes in the linearized lattice is given by n @xmath10,j) . thus, the linearized lattice can be expressed as a sequence of morphemes: ## experimental setup goal we set out to evaluate the performance of the proposed joint architecture on segmentation, tagging and parsing. in all experiments, we show the segmentation (seg) and dependency parsing (dep) f 1 score. additionally, for experiments with the pos mtl, we present the pos f 1 score. data all our experiments were trained and tested on the standard split of the hebrew section of the ud treebank collection @xcite . the training set, dev set, and test set consist of 5,168 sentences, 484 sentences, and 491 sentences, respectively. the morphological analysis for the input of our model is provided by the morphological analyzer (ma) of @xcite . 3 the ma provides the segmentation, part-of-speech (pos) tags and morphological features for each segment in each one of the possible analyses. 4 embeddings the way we generate embedding for the input lattice complements the architectural design and significantly impacts the parser performance. alongside our proposed sentencecontextualized embedding (section 4), we assessed two alternative techniques. we tested this structure with alephbert's @xcite static (static) enbeddings and contextualized (contextualized) embeddings generated directly for the linearized lattices. evaluation scenarios as part of our model we use a morphological analyzer (ma) component for generating the lattices. however, any realistic ma is not exhaustive, as it may lack some analyses, with certain tokens entirely missing (out of vocabulary, oov). hence, we aim to gauge the effect of the ma coverage on the parser performance. in the infused scenario, we experiment in a setup where for all sentences the correct analysis is guaranteed to be incorporated as one of the lattice's internal paths. to establish the infused scenario, we examine all tokens in the dataset and integrate the gold analysis for each token back into the ma. in the uninfused scenario, which represents a realistic scenario, we use the ma as is, and there may be missing analyses in the constructed lattices at inference time. models current sota results in hebrew dependency parsing are presented by trankit @xcite and stanza @xcite , both of which are multilingual neural parsers. since our proposed architecture essentially extends that of @xcite , we also evaluate their architecture in a pure pipeline setting. in this paper, we introduce three variations of hebrew parsing employing distinct segmentation techniques, as described below. for each baseline we present seg, pos and dependency parsing dep scores. ## results and analysis furthermore, finally, since the llm may be seamlessly replaced, further improvement may come from a better llm encoder. error analysis we performed a manual error analysis on a subset of 50 sentences from the hebrew ud htb dev set. in these, there are merely 8 segmentation errors, with 5 of being a missing definite article ('he hydia'h') and the remaining 3 involving incorrect segmentation of fused suffixes. in addition, a total of 108 dependency errors were identified, classified into four categories: prediction errors, wrong gold, truly ambiguous, and others ( ## related and future work previous research has delved into lattice-based dependency parsing for mrls such as hebrew @xcite , turkish @xcite , and sanskrit @xcite . however, these prior contributions predominantly utilized graph-based and transition-based systems grounded in feature functions that are handengineered. in contrast, our current work takes a different perspective, presenting a purely neural architecture. a distinct challenge lies in generating embeddings for the lattice arcs, which represent a non-linear structure -an atypical input signal for language models. the aforementioned lattice-based parsing architectures do not attend to this complexity thereby missing out on the advantages offered by contemporary large language models (llms). this paper bypasses this divide, proposing an approach that effectively handles the intricate context and creates robust representations for lattices using neural encoders. while neural studies in mrl parsing, such as the work by @xcite , also leverage the biaffine architecture of @xcite , they typically focus on architectures that handles segmented and unambiguous inputs. consequently, these models do not cope well with the challenges posed by the vast number of ambiguous words prevalent in mrls such as hebrew. in contrast, our proposed architecture accommodates ambiguous input, offering a unified solution that addresses both segmentation, morphological disambiguation, and parsing, in a single model. in future research, we aim to assess our proposed framework on other mrls, evaluate its performance across various language types and assess its generalization capabilities for lower-resourced languages. additionally, we aim to explore further enhancements of mtl in parsing, by adding joint semantic predictions such as ner and srl. ## conclusions in this paper, we present a novel neural framework for jointly segmenting and parsing morphosyntactic structures in morphologically rich languages (mrls). we address the intricate and complex nature of words in these languages and propose a method for incorporating linguistic information structured in a lattice into a neural parsing architecture. the contribution of this paper is manifold. first, we provide a language-agnostic neural joint architecture that can be used to confirm or disprove the joint hypothesis juxtaposed in the pre-neural era for mrls. second, we provide a thorough empirical investigation of hebrew, providing sota results using a single joint model. finally, as the proposed architecture relies on an llm encoder, advances are expected to be achieved as llms further improve for low-and mediumresources mrls, potentially closing the gap with non-mrls.