A Manual Evaluation Method of Neural MT for Indigenous Languages (2023)

Abstract: Indigenous language expertise is not encoded in written text in the same way as it is for languages that have a long literal tradition. In many cases it is, on the contrary, mostly conserved orally. Therefore the evaluation of neural MT systems solely based on an algorithm learning from written texts is not adequate to measure the quality of a system that is used by the language community. If extensively using tools based on a big amount of non-native language this can even contribute to language change in a way that is not desired by the language community. It can also pollute the internet with automatically created texts that outweigh native texts. We propose a manual evaluation method focusing on flow and content separately, and additionally we use existing rule-based NLP to evaluate other factors such as spelling, grammar and grammatical richness. Our main conclusion is that language expertise of a native speaker is necessary to properly evaluate a given system. We test the method by manually evaluating two neural MT tools for an indigenous low resource language. We present an experiment on two different neural translations to and from North Sámi, an indigenous language of North Europe.

## introduction indigenous languages with few speakers are often left out in the development of high-level nlp tools that require a lot of data and have therefore not been subject to evaluation either. however, recently neural machine translation has become more effective and more available for even lesser resourced languages than before. while the technology has made the use of neural machine translators plausible, it is not clear whether the quality of the translation really is good enough for the common use cases within language communities. high-resource languages typically apply datahungry evaluation methods. the demand for big data is known to be problematic for smaller languages. an additional factor is, that while big languages with a long literary tradition have their language expertise encoded in large amounts of written texts, typically this is not the case for indigenous languages with a much shorter literary tradition. here language expertise is often transmitted orally and may not be reflected in written text at all, partly due to lack of literacy and tradition. it is problematic if we base our knowledge of a language on existing written text for a language community that does not have a long tradition in writing. written texts need to be treated much more critically with regard to who wrote it (was it even a native speaker?), if it was a translation, and which genre it belongs to. written texts can have systematic spelling and grammar errors. their authors can be second language learners instead of language experts, or they can be synthetically created by machine translation programs. taking into account the distribution of human resource and language expertise is an important factor in the thought process. language communities that put a great deal of work into preserving and strengthening their language typically use a lot of resources in teaching the younger generation. that also means that expertise may be found to a great deal in oral contexts rather than being reflected in text corpora. basing evaluation on algorithms that learn from written corpora is therefore a thinking error in these contexts. consequently, we find a manual evaluation of neural mt tools by language experts in this context unavoidable. by language experts we mean native speakers with a profound understanding of their own language, which allows them to make judgements about the grammaticality and idiomaticity of a sentence. especially since indigenous written grammars are far from exhaustive, good language intuition is a key qualification. in this article we suggest a grading system for a language expert evaluator that is an expert of both source and target language. the scale distinguishes between flow and content, where flow (which has a main focus on the target sentence) is evaluated before content (which again requires an analysis of the source sentence). our main hypothesis is, we need native language/linguistic expertise to even know how good the translation is. we do a small-scale but detailed manual evaluation of two neural mt tools for an indigenous low resource language (north sámi). our aim is to develop a workflow for future evaluations of similar languages and systems and those with even less resources, than the ones we work on, should they become available in the popular nmt toolkits. ## background methods of evaluating machine translation are often based on two approaches: automatic that requires high quality parallel texts and human-based, which requires a large amount of humans doing annotation or rating of large number of sentences for example. in a low-resource minority language situation, neither of these resources is easily available; there are no parallel texts and very few humans to do annotation or rating. that is to say, the amount of sentence-aligned parallel texts that is needed to automatically verify quality is larger than amount of any translated texts in the language in the foreseeable future and the amount of people required to do a meaningful comparison is well larger than available people as well, it is physically impossible to do perform such tests. the typical automatic evaluation metrics like word error rate require either post-editing or parallel corpora which typically are not available in large quantities in indigenous low-resource contexts. thus we will be able to identify the criteria that matter for a good translation of or into the language in question. based on their feedback, automatic processes to perform an adequate evaluation can be developed. also with regard to human resources the indigenous context is a challenging one. those that are language experts with a linguistic background and a high degree of literacy are typically recruited by schools, media, as translators or any other context where language knowledge is highly sought-after. generally, the machine translation use cases can be divided in two main categories: translations that can be read to understand the source texts (assimilation, gisting) and translations that can be edited for further use (dissemination). if the tools are useful as a basis for post-editing has to be decided by members of the the language communities, which is why we also think that feedback from the community is needed to evaluate the quality. because of the systems' fluency, new machine translation tools tend to get adopted quickly by businesses (e.g. facebook, google reviews) and even official bodies. an early and critical evaluation by language community is therefore essential. machinelearning mt is now almost a standard and being used in every day life without much thought. how does it look like in an extremely low resource language context? @xcite ## methods despite limited amount of corpora north sámi has in recent years gained some experimental neural machine translators. by evaluating their current state-of-the-art we present a manual evaluation method and relevant criteria. as a test case we looked at one system to and another one from north sámi. previously north sámi has been unreachable for neural approaches to language technology due to low resourcedness. the majority of resources are therefore rule-based tools. for machine translation, language pairs included other closely related sámi languages, as well as finnish, which is in same language family, but not closely related. there also exists translators for norwegian, which is another majority language in north sámi territory. many of the existing majority-tominority language translators are primarily developed in one direction first @xcite . the rule-based machine translators are based on other language technology resources, such as dictionaries, morphological analysers, syntactic analysers and so forth. we use these morphological analysers, as well as spell-checkers and grammar checkers as tools to find out if there are differences between the human and machine translated texts for potential spelling errors, grammatical errors as well as differences in distributions of the grammatical features. the systems for linguistic analysis and grammar and spell-checking have been acquired from the giellalt infrastructure foot_2 , that contains freely available open source language technology tools for minority languages @xcite . we used the existing neural machine translation systems as a black box, we fed in the source texts and evaluated the target translations without post-editing in between; only the cases where formatting went destructively wrong (line breaks and spaces added or disappeared in unusual places, like intra-word spaces) were corrected. ## evaluation method we evaluate separately for the from and to north sámi scenarios. ## conclusion we manually evaluated two neural machine translation systems in an indigenous low-resource context, one of which has north sámi as a source language and the other of which has north sámi as a target language. translation is done either into or from a higher resource language, i.e. norwegian and english, which are both morphologically simple compared to north sámi. the sámi to norwegian evaluation is done by a native north sámi speaker who has worked as a professional translator. we developed a scale according to which first the flow of the target language is evaluated and then the representation and exactness of the source language content in the target language. both scales have 7 grades. flow and content evaluation can differ very much from each other as flow mostly focuses on the target sentence, while content takes into account the source sentence to a much higher degree. the evaluation shows that flow typically scores higher than content, which means that a clear understanding of both source and target sentence is necessary to evaluate how well the matching is done. this supports our hypothesis that high-level language expertise is necessary to evaluate the quality of a translation. for the english to sámi evaluation we applied a different evaluation method. we applied highquality rule-based proofing tools for sámi for spellchecking and basic grammar checking of the target text. as human resources for indigenous languages are typically low, we find that this method -while it cannot replace human evaluation -can be revealing as regards certain shortcomings of the mt system, which affect its quality. we discovered that spelling errors in the neural translation are more than twice as much as in the sámi text collection sikor. additionally, a low-scale manual evaluation of the fictional text alice, showed that shortcomings of the system included a variety of different morpho-syntactic errors as well of nonidiomatic constructions and nonsense translations. the second system evaluation regards the newly released multi-lingual neural mt tool by tartu university, where we had a look at english-north sámi machine translation. none of the developers has knowledge of north sámi and is therefore not able to properly evaluate the results in all its relevant details. we regard it as important that these systems are evaluated by those that have knowledge of the language, and give a reliable picture of what can and what cannot be expected of such a system. as a user can have varying knowledge themselves about either source or target language,