Getting the ##life out of living: How Adequate Are Word-Pieces for Modelling Complex Morphology? (2020)

Abstract: This work investigates the most basic units that underlie contextualized word embeddings, such as BERT — the so-called word pieces. In Morphologically-Rich Languages (MRLs) which exhibit morphological fusion and non-concatenative morphology, the different units of meaning within a word may be fused, intertwined, and cannot be separated linearly. Therefore, when using word-pieces in MRLs, we must consider that: (1) a linear segmentation into sub-word units might not capture the full morphological complexity of words; and (2) representations that leave morphological knowledge on sub-word units inaccessible might negatively affect performance. Here we empirically examine the capacity of word-pieces to capture morphology by investigating the task of multi-tagging in Modern Hebrew, as a proxy to evaluate the underlying segmentation. Our results show that, while models trained to predict multi-tags for complete words outperform models tuned to predict the distinct tags of WPs, we can improve the WPs tag prediction by purposefully constraining the word-pieces to reflect their internal functions. We suggest that linguistically-informed word-pieces schemes, that make the morphological structure explicit, might boost performance for MRLs.

## introduction contextualized word-embedding models, such as bert @xcite and xlnet @xcite , rely on sub-word units called wordpieces @xcite , that enable these models to generalize over frequent charactersequences and elegantly handle out-of-vocabulary items (with minimal resort to character-based models). this word-pieces architecture helps the models make better predictions for complete words without the need to keep a large dictionary for all the possible word-forms in a language. effectively analyzing the internal content of words is important for morphologically-rich languages (mrls) @xcite , that express multiple units of meaning at word level. due to morphological ambiguity, the interpretation of the many functions of a complete word has to be determined in the context of the utterance, making explicit the contribution of each linguistic sub-word unit (a.k.a., morpheme) to the global meaning. in this study we aim to investigate how well morphological information is captured by contextualized embedding models, or, more specifically, by their underlying word-pieces. we hypothesize that the word-pieces tokenization scheme in these models, which is not reflective of the actual morphology, will decrease the models ability to predict morphological functions on sub-word units. in order to test this hypothesis we use multilingual bert @xcite on the task of multi-tagging raw words in a morphologically rich and ambiguous language, modern hebrew. pre-neural studies on hebrew found that explicitly modeling sub-word morphological information, substantially improves results on tagging and parsing down the nlp pipeline @xcite . here our results show a significant drop in multi-tagging accuracy in word-level settings compared to settings where we aim to tag the distinct wps. nevertheless, when we purposefully incorporate morphological knowledge that reflect the internal functions of wps, the tagging of wps substantially improves. we conjecture that current word-pieces architectures might be sub-optimal for capturing complex (e.g., fusional) morphology, and that more morphologically-informed schemes may yield better models, at least for mrls. ## the data: all analytic languages are alike, each mrl is rich in its own way morphologically-rich languages (mrls) @xcite are languages that express syntactic relations by inflection or agglutination at word level. in nlp, mrls often require segmentation into sub-word units called morphemes as part of the pre-processing in the nlp pipelines. the term morphological fusion, or simply fusion, refers to the degree to which morphemes are connected to a word host or stem @xcite . there are three values for the degree of fusion: isolating (low), concatenative (mild) and non-concatenative (high). mrls thus belong to the mild-and high-fusion language groups. in concatenative mrls like turkish @xcite and russian @xcite morphemes are linearly connected to the stem, and so a concatenated word-form can easily be segmented back into its composing morphemes. segmenting highly fusional mrls (henceforth fmrl), like @xcite , is not as simple, since words can be affixed in such a way that makes the stem and/or affix undergo morpho-phonological changes resulting in ambiguous, syncretic word-forms. these changes cannot be restored without morphological disambiguation of the word in context of the whole sentence. furthermore, word-forms may involve a combination of a root and a template which are intertwined via a non-concatenative process, and both contribute meaning to the word-form. let us consider two examples for high fusion morphological phenomena in modern hebrew. first, consider the word-form ‫.בצל‬ it can either mean 'in their shadow' ( ‫-ב‬preposition ∧ ‫-צל‬ noun ∧ ‫-שלה‬possessive), 'their onion' ( ‫-בצל‬ noun ∧ ‫-שלה‬possessive)), 'in the photographer' ( ‫-ב‬preposition ∧ ‫-ה‬definite ∧ ‫-צל‬noun) or 'betselem' ( ‫-בצל‬proper noun, a known organization). the differences between the actual word-form ‫בצל‬ and the segments representing the composing morphemes in the different analyses, illustrate how complex morphological processes in hebrew dictate the final word form -that is, the final form is no longer re-constructable by (simply concatenating) the morphological segments. among the different analyses, no interpretation is a-priori more likely than others. only in context the correct analysis can be determined. next, let us consider the following two verbs: ‫שומר‬ ('/somer', keep.pres.masc.sg, 'keeps') and ‫שמור‬ ‫נ‬ ('ni-/smor', 1st.pl.fut-keep.fut, 'we will keep'). here, although the affixes ‫,ו‬ ‫נ‬ can be separated from the root letters ‫,שמר‬ the analysis of the verb cannot be constructed by analyzing the mere character sequences, it must be understood from the unified form of the morphemes. so, from the first example, we observe that morphological disambiguation is crucial, and that contextualized models may actually be good candidates for morphological disambiguation where the external context is crucial. but from the second example, we learn that the linear order and strict separation of words into word-pieces, as is done in current contextualized embeddings, may be too arbitrary and too strict, which may in turn undermine the performance of tasks down the nlp pipeline, particularly for fmrls. ## the question: how adequate are the goal this paper aims to investigate whether word pieces capture sufficient morphological information about whole words. that is, we ask whether the information contained in such representations would allow to predict the multiple functions of an input, i.e. a space-delimited word-form. in particular, we empirically examine this capacity via the task of multi-tag assignment in hebrew -where each multi-tag reflects the analyses of a single word-form bearing multiple pos tags -as illustrated in our hebrew example in section 2. we conduct a series of experiments on multi pos-tag assignment to raw word forms in hebrew texts, changing the granularity of the input and the output to reflect word-internal functions that are potentially captured by individual word-pieces. the task we define a multitag as a single label that consists of the multiple pos tags reflecting the categories of the (morphological) segments of a word-form. for example, we assign the wordform ‫,בבית‬ which means 'in the house', the multitag in ∧ def ∧ nn. in all of our experiments, the model receives as input a sentence that underwent a tokenization into word pieces by the built-in tokenizer of mbert @xcite . we then output a multitag for each word as whole. our models vary in how much (and what kind of) information is predicted for each of the word-pieces. experimental setup we use the hebrew section of the spmrls treebank, which consists of 6500 sentences from the daily newspaper ha'aretz @xcite . this corpus was manually annotated for pos tags at morpheme-level by trained experts, and it is the accepted benchmark for all morphological processing tasks in hebrew. we fine-tune the models using the pytorch implementation of transformers by @xcite . we use its standard berttokenizer and bertforto-kenclassification, with multilingual bert (cased) as our model for fine-tuning. we use the standard train set as input for finetuning, and evaluate and report results on the dev set. we report on two measures. the first is exact match (em), that is, the percentage of correct multitag assignments from all multitag assignments to word-forms in the evaluation set. the second is existence f1: precision and recall on the existence of correct pos tags in a (possible incorrect) multitag assignment. we compute existence f1 based on the precision and recall that follow. for calculating the precision and recall the predicted multitag is split into its composing simple pos tags. note that f1 gives partial credit on correctly identified pos in the case of partial identification or wrong order, while em doesn't. we begin with an oracle scenario that emulates an english-like pos tagging scenario, where the input is a sequence of strings, in our case gold presegmented morphemes, and the output is a single pos tag per segment. for fine-tuning, we use presegmented words along with their corresponding pos tags, as it is gold-annotated in our training data. it should be noted that these segments undergo additional tokenization into word pieces by mbert's tokenizer, based on its internal wordpieces lexicon, prior to fine-tuning. for comparability with the other models, the evaluation is done on raw words i.e., we combine before tokenization: after tokenization: nn the predicted simple tags into a multitag and compare it to the original multitag per word. this scenario is of course not realistic, in the sense that gold segmented data at morpheme level are slow and costly to deliver. however, this setting provides an empirical upper-bound for the performance of bert on a simple pos tagging in hebrew. we hypothesize that, had bert's tokenization into word pieces been morphologically informed, the model's accuracy in word-level settings could rise up to the level of performance on this pre-segmented oracle scenario. ## results the input, output and training regimes for our models are illustrated in as expected, the oracle scenario assigning single tags to gold segments outperformed all other models that aim to multi-tag complete words. for word-level multi-tagging, the word-level model performed at the same level as the prefix/host model -narrowing down the labels' space in this fashion does not seem to improve results or provide any further generalization capacity. purposefully fine-tuning our model to assign a single pos tag per wp (trained on our gold morphological data) did not help, in fact it dramatically hurts performance. this indicates that wps in and of themselves do not coincide with the notion of morphemes. curiously though, informing bert's wps as to their own internal function prior to fine-tuning significantly improves the results compared to the model trained to assign a pos-per-wp based on gold morphology. this last result suggests that, while current wps do not reflect morphological structure and lose morphological distinctions in their sub-word units, informing these word-units as to their own internal functions can provide a major performance boost. so far, we only incorporated such morphological information during fine-tuning. we conjecture that informing the wp algorithm earlier on, prior to pre-training, with a linguistically-informed decomposition into wps, may greatly advance the performance of contextualized models for fmrls. ## related work although the term 'word pieces' was only coined in 2017, by @xcite , the idea that sub-word segmentation might be useful for downstream tasks was already well-known and studied, especially in the field on neural machine translation. in 2010 @xcite explicitly showed that incorporating morphological knowledge in the translation process significantly improves translation. in @xcite found that for learning morphology it is better to use character based representation rather than word-based ones. they also found that neural networks encode morphology in the lower layers of the network, which might explain why mere finetuning is insufficient to capture morphological complexity. later, @xcite achieved sota on pos tagging on 54 languages, including heberew, but was using bert embeddings along with character level embeddings and fasttext @xcite word embeddings on gold morphology, which strengthen our claim that word pieces by themselves don't capture morphology well. this was also supported by @xcite , that explicitly mentioned the non-concatenativity of hebrew and arabic as the major drawback of sub word tokenization systems. ## conclusion in this work we examined the adequacy of bert's word-pieces as sub-word units for representing complex morphology. we chose to investigate multi-tagging in a high fusional language, as a proxy for assessing the underlying segmentation into distinct morphemes. we expected that if distinct word-pieces indeed reflect units of meaning, then tagging them would be as accurate as it is for languages that assign a single tag per word. our results show that the current word pieces do not reflect actual morphology, resulting in decreased performance for tagging complex hebrew words. nonetheless, we found that imposing morphological knowledge during fine-tuning (an informed setup) is indeed helpful, albeit a bit late. we conjecture that pre-training with a morphologicallyinformed word-pieces scheme that reflects a complex morphological reality, has the potential to improve multi-tagging, as well as other tasks down the pipeline, in hebrew and other fmrls.