Linguistically-Motivated Y orùbá- E nglish Machine Translation (2022)

Abstract: Translating between languages where certain features are marked morphologically in one but absent or marked contextually in the other is an important test case for machine translation. When translating into English which marks (in)definiteness morphologically, from Yorùbá which uses bare nouns but marks these features contextually, ambiguities arise. In this work, we perform fine-grained analysis on how an SMT system compares with two NMT systems (BiLSTM and Transformer) when translating bare nouns in Yorùbá into English. We investigate how the systems what extent they identify BNs, correctly translate them, and compare with human translation patterns. We also analyze the type of errors each model makes and provide a linguistic description of these errors. We glean insights for evaluating model performance in low-resource settings. In translating bare nouns, our results show the transformer model outperforms the SMT and BiLSTM models for 4 categories, the BiLSTM outperforms the SMT model for 3 categories while the SMT outperforms the NMT models for 1 category.

## introduction languages differ with regard to how grammatical information such as "case" and "number" are expressed. in some languages, this information is overtly marked using morphological or syntactic means, whereas in others it has to be inferred from context. this asymmetry of information representation poses an important problem for machine translation @xcite . several phenomena where asymmetry arises have been identified as challenging problems for machine translation. these include: pronoun translation and coreference @xcite , politeness @xcite , lexical cohesion @xcite , and lexical disambiguation @xcite . asymmetry of information representation presents an interesting test case for mt systems because it can shed light on their true linguistic ability @xcite . it is, therefore, important to use evaluation measures which can capture this aspect of the translation task. however, the most popular evaluation metric for machine translation, the bleu score @xcite , is a coarse metric which can often hide these fine-grained morphological and semantic distinctions. in fact, a high bleu score is no guarantee of improved translation quality and bleu, being based on precision on short ngrams, may be poorly suited for measuring the coherence and grammaticality of a sentence. in this paper we investigate the performance of yorùbá foot_0 to english machine translation. we specifically evaluate performance on translating bare nouns (bns). bns @xcite @xcite @xcite are nouns without an overt determiner or quantifier. for instance "houses" in houses are expensive in new york is a bn. whereas english accounts for only plural bns, bns in yorùbá are number neutral and can also be definite or indefinite depending on the context. consider the following example: our study provides a fine grained analysis that sheds light on issues in mt that are not often discussed in main stream research. we turn away from the current research trend in massively multilingual translation systems, to this largely underexplored fine-grained aspect of the mt problem. we investigate how smt and nmt systems compare in general, but also specifically with respect to translation of bns. although nmt has recently been reported to outperform smt even in low-resource settings, these findings were reported for systems translating between somewhat similar languages (e.g., languages that belong to the same language family, have overlapping vocabulary and similar script) @xcite @xcite such as @xmath0, @xmath1, @xmath2; languages for which large corpora exists. in this work, we collect a new dataset and use it to test whether the same nmt advantage persists by exploring two typologically dissimilar languages belonging to different language families: yorùbá and english. this work bears significance not only to yorùbá but to analytic languages, low-resource languages, and languages that have bns. our contributions are as follows: (1) we align a new dataset for the yorùbá and english lowresource setting. (2) we use our dataset to develop statistical and nmt @xmath3. (3) we study the linguistic ability of our models in disambiguating bns. the rest of the paper is organized as follows: section 2 is a description of disambiguation patterns of yorùbá bns. we discuss related work in section 3. section 4 presents the datasets, data collection process, and preprocessing. in section 5, we present our methods. we present results of our models and an evaluation of bn disambiguation in section 6. we conclude in section 7. ## disambiguating yorùbá bns a number of contextual variables are important when disambiguating bns in yorùbá. these in-clude the so-called familiarity and uniqueness conditions @xcite @xcite , as well as the category of the verb that occurs in the environment of the bn. familiarity refers to information which is already known from the previous textual context either explicitly or through inference. in yorùbá, a definite interpretation is permitted when the existence of the entity referred to has been established in the discourse, an indefinite occurs otherwise. in example (2), the first mention of book is indefinite, the second is definite. mo 1sg ra buy ìwé book fún for kó ˙lá. kó ˙lá. l'ó ˙jó i̇n day kejì, second kó ˙lá kó ˙lá ti perf so throw ìwé book nù away 'i bought a book for kó ˙lá. by the second day, kó ˙lá had lost the book.' the uniqueness condition claims that there exists one and only one entity that meets the descriptive content of the bn as in example (3). this entails that the bn will be interpreted as definite and singular. the sun shines in the afternoon.' the category of verb is also an important contextual element in correctly translating bns. stative verbs (verbs that describe the state of being or situation such as to own and to feel) introduce the generic description where a noun phrase is used to refer to a whole class as in example (4). eventive verbs (verbs that describe events such as to break and to appear) introduce all other disambiguation patterns as in example (1). tests suites that evaluate more than 100 linguistic phenomena was developed for english to german and german to english machine translation @xcite . the test sentences are given as input to the mt systems. the mt outputs are then evaluated by the set of rules which determine whether the output was correctly translated or not. @xcite assess the grammaticality of the output of a character-level nmt system by evaluating the mt model's capacity to correctly rank contrastive pairs of pre-existing translations, one of which is correct and the other one incorrect. this approach has also been applied to lexical disambiguation of english-german mt @xcite . @xcite explore automatic detection and classification of translation errors based on manual post-edits of mt output both for smt and nmt systems. they use a classification that evaluated outputs for morphological, lexical, and word order errors which was a simplification of those used in hjerson @xcite . hjerson detects word level error classes: morphological errors, re-ordering errors, missing words, extra words and lexical errors. neural models have also been evaluated for syntactic competence. for instance, @xcite probe the ability of lstm models to learn english subject-verb agreement. when provided explicit supervision, lstms were able to learn to perform the verb number agreement task in most cases, although their error rate increased on particularly difficult sentences. nmt systems have also been evaluated for morphological competence while translating from english to a morphologically rich language @xcite . certain linguistic phenomena have also been tested across language families. for instance, for four language families: slavic, germanic, finno-ugric and romance, the best nmt system outperformed the best phrase-based smt system for all language directions to english @xcite . the nmt systems produced fluent and more accurate inflections and word order but performed poorly when translating long sentences @xcite . morphologically rich languages have also been shown to have more fluent outputs @xcite @xcite . these studies have used metrics such as bleu @xcite , hter @xcite , ttr (dar-ley, 1959), yule (yule, 2014) to evaluate the output of mt models for fluency and adequacy. mt was also evaluated for discuss phenomena where the model's capacity to exploit linguistic context for co-reference and coherence was evaluated @xcite . evaluation frameworks like hope have also been used for task-oriented and human-centric evaluation for machine translation outputs @xcite . this framework uses professional post-editing annotations and contains commonly occuring error types and error penalty points. the error penalty points use geometric progression to reflect the severity level of errors for each translation unit. ## dataset we use the yorùbá bible bíbélì mímó ní èdè yorùbá òde-òní (bmeyo) and the new international version (niv) english bible. ## methods we train 3 sentence level models: smt, bilstm, and transformer models to translate from yorùbá to english. we choose english as our target lan-4 https://spacy.io/ guage because we are interested in analyzing how ambiguous bns in yorùbá are translated into english where (i.e., in english) nouns are typically marked both for number and determinacy. the hyperparameters and training procedure are described in the next subsections. ## evaluation in this section, we first evaluate using bleu. we provide details of this part of the evaluation in section 6.1. next, we provide details on our approach to evaluating bns in section 6.2. ## conclusion in this work, we showed how smt, bilstm, and transformer models translate bns in yorùbá, a resource scarce language. we compared the ability of smt and nmt models to correctly translate bns into various categories referenced in the syntax literature of yorùbá. we measured the performance of the mt models and the output using bleu scores, and by counting the percentage of correctly disambiguated bns compared with incorrectly disambiguated bns. we found a positive correlation between disambiguation accuracy, and bleu scores as well as a positive correlation between number of occurrences of a category and the accuracy in translation. we also found the transformer outperforming the smt and bilstm models in correctly translating bns. we also found that all 3 models best performed in translating a bn in yorùbá into an definite singular in english. this finding corroborates research that predicts that languages which lack overt definite and indefinite markers have larger cases of definites, and findings within the mt community that mt models improve with more data. we further analyzed the type of errors our systems produce. we identified cases of missing words, wrong word or spellings, grammaticality issues, and word-order errors. we found that even when certain bns have been correctly categorized by the models, the models still had semantic, and or syntactic errors. in order to further probe the capacity of smt and nmt models in disambiguating bns, further work can be carried out to improve the smt and nmt models and perform human based evaluations on the entire quality of the mt output. we can improve the mt systems with back-translation, cross-lingual word embeddings, increasing the size of data used for training, transfer learning, among other approaches. standard test sets can also be developed to aid automatic comparison of human evaluations and machine based evaluations. in addition, the yorùbá data we used for this work is a translated document; a translation from english to yorùbá and it will be interesting to use a text originally written in yorùbá but translated to english for this experiment.