A Pointer Network Architecture for Joint Morphological Segmentation and Tagging (2020)

Abstract: Morphologically Rich Languages (MRLs) such as Arabic, Hebrew and Turkish often require Morphological Disambiguation (MD), i.e., the prediction of morphological decomposition of tokens into morphemes, early in the pipeline. Neural MD may be addressed as a simple pipeline, where segmentation is followed by sequence tagging, or as an end-to-end model, predicting morphemes from raw tokens. Both approaches are sub-optimal; the former is heavily prone to error propagation, and the latter does not enjoy explicit access to the basic processing units called morphemes. This paper offers MD architecture that combines the symbolic knowledge of morphemes with the learning capacity of neural end-to-end modeling. We propose a new, general and easy-to-implement Pointer Network model where the input is a morphological lattice and the output is a sequence of indices pointing at a single disambiguated path of morphemes. We demonstrate the efficacy of the model on segmentation and tagging, for Hebrew and Turkish texts, based on their respective Universal Dependencies (UD) treebanks. Our experiments show that with complete lattices, our model outperforms all shared-task results on segmenting and tagging these languages. On the SPMRL treebank, our model outperforms all previously reported results for Hebrew MD in realistic scenarios.

## introduction in morphologically rich languages (mrls) @xcite , raw tokens are morphologically ambiguous, complex, and consist of sub-token units referred to as morphemes. 1 morphological disambiguation (md) is the task of decomposing the tokens into their constituent morphemes, to be used as the basic processing units for nlp tasks down the pipeline @xcite . as opposed to the commonly known scenario of morphological tagging @xcite , where every input token is assigned a single morphological signature (containing its lemma, part-of-speech tag, and morphological features such as gender, number, person, tense, etc.), in the md scenario internally-complex input tokens may consist of multiple distinct units, each of which gets assigned its own morphological signature. pre-neural statistical approaches for md @xcite @xcite typically used weighted finite-state machines to unravel the possible morphological decompositions, and classic machine learning models to select the most likely decomposition. current neural models, however, take radically different paths. one neural approach to md employs pipeline, where a predicted segmentation of words into morphemes is passed on to sequence labeling component that performs tagging of each segment in context. this segmentation-first scenario employs sequence tagging to assign a single morphological tag to each segment similar to pos tagging in english, where each token in the input sequence is assigned a single label by the tagger. this method might be expected to work for mrls just as well as standard nlp models do for english tagging, however, in actuality, such pipeline architectures are prone to error propagation, which undermines the accuracy of almost any task down the nlp pipeline (tagging, parsing, named entity recognition, relation extraction, etc.) @xcite @xcite . a second conceivable approach is an end-to-end sequence-to-sequence model that consumes a sequence of tokens (or characters) and produces a sequence of morphological signatures. notably, the number of morphological signatures may vastly exceed the number of input tokens, (e.g., with an average of 1.4 tags per word in hebrew). the drawback of this approach is that the model has no access to morphological information in the input, and is expected to extract all morphological information directly from the raw text. tokens in mrls are lexically and syntactically ambiguous, and carry many possible interpretations, so it is unclear if the surface signal is in fact sufficient. this fact is exacerbated by the fact that some mrls are low resourced and even with pre-trained word embeddings, many forms are lacking when operating on internally-complex tokens. in this paper we propose an alternative approach, that enjoys the power of end-to-end neural modeling while maintaining access to morphemes. we frame the problem as a morphological analysis and disambiguation (ma&d) task, in which every raw token in the input sequence first goes through morphological analysis (ma) that exposes all of its possible morphological decompositions as a lattice (see figure 1 ). this morphological lattice is then passed to the md component, based on a pointer network, which selects a sequence of most likely arcs in the context of the sentence being processed. since every lattice arc contains rich information that is made available by the ma -namely, segmentation boundaries, lemma, part-of-speech tag and a set of morphological features -this ma&d framework can jointly predict rich morphological layers while avoiding the pipeline pitfall. based on this architecture, we design a neural model for joint segmentation and tagging and apply it to two mrls, hebrew and turkish. in realistic circumstances, the lexical coverage of the lattice may be partial, and we report md results in both ideal and realistic scenarios. our results on the hebrew and turkish ud treebanks show state-ofthe-art performance for complete morphological morphological disambiguation (md) is the task of selecting a single most-likely analysis for each token in the context of the sentence. the resulting morpheme sequence may then serve as the input processing units for downstream tasks (similarly to space-delimited words in english). our above example, bbit hlbn is likely to be disambiguated as: the ambiguous ma output is stored in a lattice data structure. a lattice is directed acyclic graph (dag) often used to encode ambiguity in nlp. in a morphological lattice, every node represents a segment boundary, and every edge represents a morpheme. every path through the lattice represents a single possible analysis of the entire sentence. notably, not all segmental forms in the lattice are overt in the input stream. some are implicit, due to intricate morpho-phonological and orthographic processes. for example, the analysis of the token bbit contains three morphological segments b, h, bit in the chosen path, yet the h segment is not visible in the input token bbit (figure 1 ). ## proposed method the task the input to our ma&d framework is a sequence of tokens and the output is a sequence of disambiguated morphological analyses, one per token. we assume a symbolic ma that generates ambiguous lattices containing all possible morphological analyses per token, based on a broad-coverage lexicon and/or symbolic rules of the language. given an input lattice, we frame md as a lattice disambiguation task. @xcite approached this task by constructing a specific architecture that captures the lattice representation. we, in contrast, choose to modify the lattice representation and feed it to an existing network architecture. the key idea, in a nutshell, is to linearize the lattice into a sequence of partially-ordered analyses, and feed this partial order to a pointer network. for each token, the network will then learn to point to (select) the most likely analysis, preserving the linear constraints captured in the lattice structure. pointer network (ptrnet) pointer networks @xcite are designed as a special case of sequence-to-sequence (seq2seq) networks. seq2seq models take an input sequence and produce an output sequence which may differ in length and vocabulary. ptrnet in addition can handle output vocabulary depending on the input sequence which can be variable in length. seq2seq is composed of an encoder and a decoder. the encoder consumes and encodes the entire (embedded) input sequence. then, the decoder is fed the entire encoded input representation and step by step produces discrete outputs which are fed back as input to the next decoding step. ptrnets have an additional copy attention layer. the attention layer focuses on specific elements of the encoded input sequence at each decoding step @xcite . copy attention is a special case where the attention weights determine which input element the decoder's state is most aligned with, which can then be copied to the output. pointer networks for md (ptrnetmd) the ptrnet architecture is designed to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence @xcite . our goal is then to encode the morphological lattice as a sequence, and then feed it to the ptrnet so that the individual analyses in the lattice can be pointed, selected and copied into the output sequence, while respecting the lattice ordering constraints. given a lattice we serialize it by going over each token and listing all of its analyses. the linearization function maps a sequence of n tokens into a sequence of m analyses while preserving the partial order of the tokens, and where m is the sum of all token analyses. that is, for input tokens t 1 , ...t n , let a j i denote the i'th analysis of the j'th token. then the following holds, such that n @xmath0. an analysis a i j is expressed as a list of morphemes where each morpheme is represented as a tuple of morphological properties. both the spmrl and ud scheme specify four properties form, lemma, pos tag, morphological features. for example, (3) is an analysis composed of three morphemes: we design a morphological embedding layer which acts as an interface between the symbolic ma and the neural md. figure 2 describes the encoding of a single morphological analysis into an embedded vector: each property is embedded and averaged across all the morphemes in a single analysis, and all of the averaged embedded properties are concatenated to form a single embedded vector of a fixed size. the entire ma&d process is depicted in figure 3 . ## experimental setup the data the ptrnetmd architecture we propose does not depend on any specific definition of morphological signature. to showcase this, we experiment with data from two different languages and two different annotation schemes. we use the universal dependencies v2.2 dataset @xcite , 2016) from the conll18 ud shared task. 3 in addition we download the corresponding lattice files of each treebank from the conll-ul project. 4 since our approach is sensitive to the lexical coverage of the ma lattices, we focus on the hebrew (he htb) and turkish (tr imst) treebanks. unlike the other languages in the shared task, hebrew and turkish provided lattice files generated by broadcoverage analyzers (heblex and trmorph2). 5 for comparability with previous work on modern hebrew, we also train and test our model on the hebrew spmrl treebank standard split. 6 lattice embedding we use pre-trained fasttext models to embed the forms and lemmas. fast-text models generate vectors for any word using character ngrams, thus handling out-of-vocabulary forms and lemmas @xcite . for pos tags and features we instantiate and train from scratch two embedding modules. together, these 4 embedded properties are combined to produce a single morphological analysis vector. lattice encoding the above-mentioned morphological embedding layer turns the input analysis sequence into an embedded sequence. the partially ordered sequence of embedded analyses is fed to an encoder layer thus encoding the entire lattice. next a step-by-step decoding process begins in which a decoder is using an attention mechanism in order to score the alignment between each of the relevant encoded analyses and the token currently being decoded. our copy attention module is the global dot-product of @xcite using masking mechanism to make sure each decoding step is focused only on the corresponding input token analyses (in figure 3 the masks are represented by the grouped arrows pointing from the decoder back to the encoded sequence). the decoder chooses the highest scoring analysis. the full output sequence contains a list of indices, one per token, pointing to the selected analyses from the input lattice (fig. 2 ). ## conclusions and future work in this work we addressed the challenge of morphological disambiguation for mrls. we design a general framework that consumes lattice files and output a sequence of disambiguated morphemes, each containing the segmentation boundary, lemma, part-of-speech tag and morphological features. our solution is language agnostic and we apply it on two different languages and two different annotation schemes. we show that access to symbolic morphological information aids the neural disam-biguation model, compared to end-to-end strong baselines that only have access to the raw tokens. we empirically evaluate our model using two evaluation methods. the conll18 ud shared task evaluation, and a multi-set intersection-based evaluation, which is a more informative metric for downstream tasks that operate directly on morpheme sequences. in an ideal scenario, where full lexical coverage is assumed, our model outperformed the shared task leaders in the word segmentation task as well as the joint segmentation-andtagging task, in both turkish and hebrew. furthermore, we match the leading joint segmentation and tagging scores in realistic scenario with only partial lexical coverage on hebrew. we further show superior performance of our model compared to previous models on the hebrew spmrl treebank. this work motivates two future research directions. our infused-vs-uninfused analysis suggests that most errors on uninfused lattices are due to partial ma coverage. our disambiguation model proves to be very reliable in selecting the correct analysis, when available. it follows that a broadcoverage ma component may improve the overall quality of the disambiguation in realistic (uninfused) scenarios. this motivates learning to induce universal, high-recall, ma which is free to generate large lattices, and rather than focusing on precision, reward high recall. a second research path towards improving realistic partial-coverage (uninfused) lattices is by combining our morphologically-aware pointer network with an end-to-end model that operates on the raw token sequence. finally, we intend to extend this lattice-based architecture for complete joint morpho-syntactic and morpho-semantic tasks. that is, in addition to morphological segmentation and tagging, the pointer network can be trained to predict span labels (as in ner), headedness relations (as in dependency parsing) and possibly more properties for the lattice arcs, so that these multiple layers of information may be jointly predicted as part of the lattice-disambiguation task.