C hr E n T ranslate: C herokee- E nglish Machine Translation Demo with Quality Estimation and Corrective Feedback (2021)

Abstract: We introduce ChrEnTranslate, an online machine translation demonstration system for translation between English and an endangered language Cherokee. It supports both statistical and neural translation models as well as provides quality estimation to inform users of reliability, two user feedback interfaces for experts and common users respectively, example inputs to collect human translations for monolingual data, word alignment visualization, and relevant terms from the Cherokee English dictionary. The quantitative evaluation demonstrates that our backbone translation models achieve state-of-the-art translation performance and our quality estimation well correlates with both BLEU and human judgment. By analyzing 216 pieces of expert feedback, we find that NMT is preferable because it copies less than SMT, and, in general, current models can translate fragments of the source sentence but make major mistakes. When we add these 216 expert-corrected parallel texts into the training set and retrain models, equal or slightly better performance is observed, which demonstrates indicates the potential of human-in-the-loop learning.

## introduction machine translation is a relatively mature natural language processing technique that has been de ployed to realworld applications. for instance, google translate currently supports translations between over 100 languages. however, a lot of lowresource languages are out there without the support of modern technologies, which might ac celerate their vanishing. in this work, we focus on one of those languages, cherokee. cherokee is one of the most wellknown native american languages, however, is identified as an "endan gered" language by unesco. cherokee nations have carried out language revitalization plans (na tion, 2001) and established language immersion programs and k12 language curricula. chero kee language courses are offered in some universi ties, including unc chapel hill, the university of oklahoma, stanford university, western carolina university. a few pedagogical books have been published @xcite and a digital archive of historical cherokee language documents has been built up @xcite . however, there are still very limited resources available on the in ternet for cherokee learners; meanwhile, first lan guage speakers and translators of cherokee are mostly elders and would likely benefit from ma chine translation's assistance. this motivates us to develop the first online cherokeeenglish machine translation demonstration system. extending our previous works @xcite , we develop the backbone statistical and neural ma chine translation systems (smt and nmt) on a larger parallel dataset (17k) and obtain the state oftheart cherokeeenglish (chren) and english cherokee (enchr) translation performance. besides translation, our system also supports quality estimation (qe) for both smt and nmt. qe is an important (missing) component of ma chine translation systems, which is used to inform users of the reliability of machinetranslated con tent @xcite . since our models are trained on a very limited number of parallel sen tences, it is expected that the translations will be poor in most cases when used by internet users. therefore, qe is essential for avoiding misuse and warning users of potential risks. existing best performance qe models are usually trained under supervision with quality ratings from professional translators @xcite . however, we are unable to easily collect a lot of human rat ings for cherokee, due to its state of endanger ment. nonetheless, we test both supervised and unsupervised qe methods: (1) supervised: we use bleu @xcite as the quality rat ing proxy and train a bleu regressor; (2) unsu pervised: following the uncertain estimation lit erature @xcite , we use the ensemble model's output probability as the es timation of quality. furthermore, to evaluate how well the qe models perform, we collect 200 human quality ratings (50 ratings for smt chren, smt enchr, nmt chren, and nmt enchr, respec tively). we show that our methods obtain mod erate to strong correlations with human judgment (pearson correlation coefficient γ ≥ 0.44). one main purpose of our system is to allow humanintheloop learning. since limited paral lel texts are available, it is important to involve humans, especially experts, in the loop to give feedback and then improve the models accord ingly. we develop two different user feedback interfaces for experts and common users, respec tively (shown in figure 2 ). we ask experts to pro vide quality rating, to correct the modeltranslated content, and to leave openended comments; for common users, we allow them to rate how help ful the translation is and to provide openended comments. upon submission, we collected 216 pieces of feedback from 4 experts. we find that experts favor nmt more than smt because smt excessively copies from source sentences; accord ing to their ratings and comments, current transla tion systems can translate fragments of the source sentence but make major mistakes. our naive humanintheloop learning, by adding these 216 expertcorrected parallel texts back to the training set, obtains equal or slightly better translation re sults. plus, the expert comments shine a light on where the model often makes mistakes. besides, our demo allows users to input text or choose an ex ample input to translate (shown in figure 1 ). these examples are from our monolingual databases, so that experts will annotate them by providing trans lation corrections. finally, to support an interme diate interpretation of the model translations, we visualize the word alignment learned by the trans lation model and link to cherokeedictionary to pro vide relevant terms from the dictionary. our code is hosted at chrentranslate and our online website is at chren.cs.unc.edu. common users need to accept agreement terms before us ing our service to avoid misuse; access the ex pert page chren.cs.unc.edu/expert requires autho rization. we encourage fluent cherokee speakers to contact us and contribute to our humaninthe loop learning procedure. a demonstration video of our website is at youtube. in summary, our demo is featured by (1) offering the first online machine translation system for translation between cherokee and english, which can assist both pro fessional translators or cherokee learners; (2) doc umenting human feedback, which, in the long run, expands cherokee data corpus and allows human intheloop model development. additionally, our website can be easily adapted to any other low resource translation pairs. ## conclusion & future work in this work, we develop a cherokeeenglish machine translation demonstration system that intends to demonstrate and support automatic translation between cherokee and english, col lect user feedback/translations, allow humanin theloop development, and eventually contribute to the revitalization of the endangered cherokee language. future work involves inviting more ex perts and common users to test/use our system and proposing more efficient and effective humanin theloop learning methods.