Quantifying Synthesis and Fusion and their Impact on Machine Translation (2022)

Abstract: Theoretical work in morphological typology offers the possibility of measuring morphological diversity on a continuous scale. However, literature in Natural Language Processing (NLP) typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the rigidity of such claims, by quantifying morphological typology at the word and segment level. We consider Payne (2017)’s approach to classify morphology using two indices: synthesis (e.g. analytic to polysynthetic) and fusion (agglutinative to fusional). For computing synthesis, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study. Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish, and verbs in English-Spanish) and segment level (previous language pairs plus English-German in both directions). We complement the word-level analysis with human evaluation, and overall, we observe a consistent impact of both indexes on machine translation quality.

## introduction one of the first barriers to develop language technologies is morphology, i.e., how systematically diverse their word formation processes are. for instance, agglutination and fusion are two morphological kind of processes that concatenate morphemes to a root with explicit or non-explicit boundaries, respectively. processing morphologically-diverse languages and evaluating morphological competence in nlp models is relevant for language generation and understanding tasks, such as machine translation (mt). it is unfeasible to develop models * work started when the first author was doing a research internship with jb at aalborg university, campus copenhagen with capacity large enough to encode the full vocabulary of every language, and it is a must to rely on subword segmentation approaches that help to constrain the capacity when generating rare, or even new words @xcite . hence, understanding morphology is essential to develop robust subword-based models and evaluate the quality of their outputs @xcite . nevertheless, there is a potential gap between the probing of whether an nlp model can handle "morphological richness", and what is a proper measure of "morphological richness" from linguistic typology. in most of the recent nlp literature, different types of languages (e.g. agglutinative, polysynthetic) are chosen to test a more diverse handling of morphological richness @xcite . there is, however, a debate as to whether languages can indeed be classified into discrete morphological categories. payne (2017) provided a morphological typology measurement in a continuous spectrum using the indices of synthesis and fusion. synthesis measures if a segment is highly analytic or synthetic (from 1 to more), whereas fusion measures whether it is highly agglutinative or fusional (from 0 to 1). and surprisingly, with respect to nlp publications, it is possible to identify english segments with a very low fusion index, meaning that they are highly agglutinative 1 . from a more applied perspective, if the references of an evaluation set (in any language generation task) are labelled with the indices, we could perform a stratified analysis (e.g. low fusion and high fusion) to determine how well an nlp model handles morphology for multiple languages. for example, we could assess whether a machine translation model is failing in generating more fusional than agglutinative segments for a specific target language. knowing and quantifying that problem concerning morphology is the first step towards proposing a solution. our contributions then are listed as follows: ## word-level analysis of synthesis and fusion in machine translation in this analysis, we ask the following question: how difficult is translating a word concerning its index of synthesis or fusion? for evaluating synthesis, we work with turkish foot_10 nouns and verbs, and for fusion, we keep working on spanish verbs. for both cases, english is the source language in the translation task. ## segment-level analysis of synthesis and fusion in machine translation following up the word level analysis, we study the relationship between machine translation difficulty and the degree of synthesis or fusion at the segment level. for this purpose, we process a set of translation systems for the language pair we want to evaluate. the general steps are as follow: 1. for each system output, we compute automatic evaluation metrics (bleu @xcite , chrf @xcite and/or comet @xcite ) with respect to the reference set, per sentence. 12 2. for each sentence of the evaluation set, we compute potential predictor variables for the automatic metric, such as the degree of synthesis or fusion. we complement the predictor variable list with other heuristics, such as the length of the sentence in characters (char.count) or words (word.count). the full list of all the predictors per language pair is in the appendix. 3. with the previous inputs, we generate generalized linear models per system output and evaluation metric, in which each model's output is set to the predictor variables. the goal is to identify which predictors affect each method's performance. 4. following model creation, we extract the significant predictors of each model. this provides an indication of which variables can be used to predict the outcome of the model's dependent variable -in our case the degree of synthesis or fusion, or any heuristic. 13 ## discussion it is important to note the limitations of this study. overall results do not suggest that translating into more analytic languages (e.g. chinese) or more agglutinative ones (e.g. turkish) is easier than their counterparts. highly analytic ones present the significant issue of word coverage and vocabulary size of the model. besides, we cannot isolate the fusional degree from synthesis entirely. for instance, turkish is a highly agglutinative language, but also highly synthetic, and there are languages that present both agglutinative and fusional traits, like navajo. moreover, the language scope is another limitation: is it possible to extend it to further languages in a practical way? synthesis can be calculated directly only if the morphological analyser splits the word into morphemes, and fusion poses several issues as mentioned before. furthermore, payne (2017) also indicated that the discourse can impact the computed degrees due to the diversity of the vocabulary. this study focuses on news data only, and it will be relevant to extend it to different domains. to address the limitations, we consider that our word level analysis, that targets specific pos, has been fundamental to enable the study of the indexes, and to partially isolate them from each other. the selection of our study cases was also relevant. spanish verbs do not present more than three morphemes, keeping a low synthesis value across all the analysis, whereas turkish is more agglutinative than fusional. moreover, to rapidly extend the evaluation for new languages and domains, we could follow a less fine-grained analysis in each index. for instance, we can compare @xmath0. @xmath1, or @xmath2. @xmath3, as we did in this work. ## conclusion and future work in conclusion, we proposed methods to quantify the indices of synthesis and fusion in automatic and semi-automatic ways, respectively. besides, for the chosen language pairs, we observed that the studied degrees have an impact in machine translation performance at both word and segment level, where we included a human evaluation of the former case. our analysis opens the possibility for further fine-grain evaluation approaches for mt and other nlp generation tasks. for instance, as future work, we can ask: are we improving the automatic translation of highly fusional words or segments? following our methodology, we could stratify evaluation sets to measure how our models performs in different parts of the spectrum. besides, the indices could also be helpful for evaluation approaches in morphological segmentation. furthermore, another potential research avenue is to aid model training in mt: e.g. knowing which segments are more or less synthetic and/or fusional could be beneficial for sampling strategies.