A Research-Based Guide for the Creation and Deployment of a Low-Resource Machine Translation System (2023)

Abstract: The machine translation (MT) field seems to focus heavily on English and other high-resource languages. Though, low-resource MT (LRMT) is receiving more attention than in the past. Successful LRMT systems (LRMTS) should make a compelling business case in terms of demand, cost and quality in order to be viable for end users. When used by communities where low-resource languages are spoken, LRMT quality should not only be determined by the use of traditional metrics like BLEU, but it should also take into account other factors in order to be inclusive and not risk overall rejection by the community. MT systems based on neural methods tend to perform better with high volumes of training data, but they may be unrealistic and even harmful for LRMT. It is obvious that for research purposes, the development and creation of LRMTS is necessary. However, in this article, we argue that two main workarounds could be considered by companies that are considering deployment of LRMTS in the wild: human-in-the-loop and sub-domains.

## introduction this research-based guide surveys the literature in order to provide a guide for companies that plan on deploying low-resource machine translation systems (lrmts) in the wild. the guide is meant to be used as a practical manner of knowing whether or not the lrmts meets the minimum requirements established by the literature to support those who live in regions where the respective lowresource language is spoken. much of the work in computational linguistics and machine translation (mt) focuses on high-resource languages, and especially english. in a recent acl-2022 conference @xcite and mt workshop (wmt-2022 @xcite ), there is considerable interest in "the bender rule" @xcite which states that the research community should move beyond english and even beyond highresource languages. there are a number of commercial mt products that support an amazingly large set of language combinations, and there are some research groups that are attempting to support even more combinations @xcite . of course, some language pairs are more successful than others. some of the low-resource language pairs that end up being deployed in the wild can be considered useful and others not so useful or even downright unethical @xcite due to their low quality. high-quality mt systems are more often than not back by neural networks; thus, neural machine translation (nmt) has advanced the state-of-the-art (sota) on many benchmarks. this is particularly true for high-resource languages like english and spanish because neural methods have been shown to work best with huge amounts of data @xcite . more traditional methods such as phrase-based statistical machine translation (smt) tend to work better than nmt when training data is limited. in this article we first explore in section 2 a list of challenges for companies that are considering deploying a lrmts in the wild. secondly, we discuss in section 3 the minimal requirements that a company should take into consideration when deploying an lrmts. after presenting the challenges and minimum requirements, we provide an overview of related work in section 4 to provide insight into the quality standards in section 5 and how to address them in section 6. ## challenge list we argue that, despite a popular opinion that deploying lrmts quickly is necessary for success @xcite , companies that deploy lrmts should consider reviewing literature such as this article to address ethical and responsible concerns in order to avoid outright rejection by the lowresource community that their system targets. from the company's perspective, successful lrmts require a compelling business case in terms of demand, cost and quality. companies are more likely to fund projects that address those concerns. but, since quality tends to increase with the size of the training set @xcite in nmt and even smt, it can be hard to determine whether or not a lrmts should be deployed in the wild. to avoid rejection of a lrmts's deployment from its targeted community, we propose two workarounds: (a) human-in-the-loop and (b) subdomains to address the following three challenges that a lrmts's creator must overcome as a first (not only) step: challenge 1. the business case needs to be compelling in terms of demand, cost, and quality. challenge 2. the lrmts's quality should be good enough to provide value to its target community. challenge 3. workarounds should be considered when mt quality is low. ## minimum viable product (mvp): minimal requirements while high-resource languages can be considered more reliable for mt, most lrmts are probably not up to par for deployment in their respective target communities. we argue that lrmts deployed for the wrong reason may cause more harm than help. if the needs of the of the low-resource community are not taken into account, results can be disastrous and difficult to turn around @xcite . at a minimum, the questions and statements below should be addressed. what if the low-resource community is not interested? risks associated with widespread adoption of digital system deployed in the wild, such as risks 1.0 and 2.0 defined by @xcite , can be costly. it is a mistake to deploy lrmts into the wild without sufficient demand. the mvp requires hundreds (if not thousands) of users in the low-resource community that are willing to use it. the ethical concerns could by far be more important than any other factor @xcite . when a company creates a business case for deploying a lrmts, it should at a minimum take the following into consideration: (1) demand (market size), (2) costs (memory footprint and computa-tion) and (3) high quality translations for ethical reasons. estimates of demand. demand for lrmts seems to be low due to the lack of funding from nations where low-resource languages are spoken. while there are exceptions such as the european low-resource projects horizon foot_0 and others, smaller countries with less governmental power like peru, for example, provide less funding in general. (camacho and zevallos, 2020) demand is focused on high-resource languages which have more speakers with more buying power. nonetheless, with the introduction of large-language models (llms), interest by larger private companies like meta @xcite in lrmt has increased. business demand, while not easily calculable for low-resource regions, can occur in unforeseen situations. crises situations, such as natural disasters, could constitute enough demand but much harder to forecast. @xcite unfortunately, these types of disasters can produce a higher demand in regions where low-resource languages are spoken and should be considered of upmost importance. ## related work this article is inspired by @xcite . their work was written at a time when neural-based systems were catching up to statistical alternatives, but their paper helped to close the gap by identifying six actionable challenges for advocates of neural-based systems. the hope is that this article provides clear goals for those developing lrmts to achieve in a similar way -a challenge list. this section will survey a few papers that take a similar approach. deployment. a number of papers have discussed minimum viable product (mvp) in the context of lrmt. @xcite discuss a number of challenges associated with creating systems for low-resource language communities. @xcite focus on the challenges of deployment related to multiple domains, a challenge covered later in section 6.2. their work discusses accuracy and other preconditions for deployment. @xcite comment on the effects of fewshot learning in lrmt when translating icelandic. other work @xcite assesses the quality of human effort as a metric for mt system deployment. their work addresses a few of our concerns but this article combines several subchallenges not covered by theirs into one (the de-ployment viability challenge). other task-specific mt work @xcite provides a "crisis cookbook" of terminology for a deployed lrmts in crisis situations but does not address issues from generic lrmts. lastly, one of the more important investigations @xcite sheds light on bloated models that use distillation as a form of compressing models in low-resource system deployment. their work encroaches on the same path as this because it takes into account the deployment of systems that use llms for low-resource settings, by far the most popular approach in current times. quality. much has been written about lrmt quality. initial work @xcite introduce indices and software that were promising and included both the mt system and the human; while their work is notable, we focus on the following seminal work. mentioned before as a human value challenge resource, work by @xcite extends previous work @xcite by introducing a translation quality assessment metric that we use in this work along with other measurements. other automated methods such as the one from @xcite focus more on creating predictors for quality rather than the challenge of measuring human versus machine. evaluation. when it comes to evaluation for lrmt, an article of this nature could report on many. however, there are some main resources that are used in determining the challenges consisting of the following work. the default standard measurements which cover string-based and embedding-based methods are already mentiond by @xcite : bleu @xcite , chrf @xcite , bertscore @xcite , comet @xcite , bleurt, @xcite and meteor @xcite . several major lrmt projects like gourmet @xcite , google research @xcite , flores @xcite and more @xcite currently use the standard metrics. one previous investigation @xcite used bleu @xcite to determine that 70k sentences was sufficient to provide decent quality for a neural lrmts. the assumption from lrmt developers is that including humans is expensive and time-consuming avoiding inclusion of more human-like measurements such as adequacy @xcite , hter @xcite , and fluency @xcite . this article describes uses of those metrics along with the following others to help better overcome the evaluation challenge. for the bias and culture challenge, we report two evaluation frameworks used for guidance: winomt @xcite and mt-geneval @xcite . for evaluating human parity value with lrmt, seminal work @xcite provides insight into translations as a whole used a translation quality assessment. one quality metric used for evaluation by projects like the one from (bayón and sánchez-gijón, 2019) called the multidimensional quality metric (mqm) @xcite identifies errors from the wide range of possibilities mentioned. however, it does not seem to take into account bias and culture, something that we address in this article. ## quality the quality expectations of a lrmts should be similar to those of a professional translator. an unfortunate by-product of the increasing amount of digital resources available is that they dampen performance due to higher search spaces. we consider the following attributes of a high-quality translation for different domains as highly important: (1) verified by humans and (2) adjusted to their domain (3) free of bias and (4) evaluated for accuracy. there are several techniques to guarantee quality of which the main two methods are: involving humans and estimating quality. quality estimation of machine translation must have used a human-in-the-loop regardless if it is for the ground truth translations or the approval of mt system suggestions. we highly recommend the use of a framework such as the translation quality assessment framework @xcite which should include several of the metrics mentioned in section 5.1. what determines if lrmts translations are of high quality? generally speaking, humans determine whether or not a translation is of high quality. of course, in a lrmts, the quality expectation are generally lower since most lrmts do not tend to be of high quality. one way of measuring is called the translation edit rate (ter) @xcite and it is the amount of edits that a professional translator would take for improving it. as for an acceptable ter score, acceptable ranges from previous work @xcite @xcite for lrmts should be ≈ 50-70 and by no means should they be more than 90 (a near useless translation). other metrics such as hter (human ter), meteor @xcite @xcite are considered correlationary with humans and discussed further in section 5.1. are there methods for estimating quality in a lrmts without a human? although there are automated methods for estimating the quality of an lrmts, the methods generally use some form of reference (ground-truth) data as is the case of quest @xcite , a framework that uses word and sentence-level features for estimating quality similar to a human. we discourage the use of quality estimation and other automated techniques during the initial phases of the creation of a lrmts that is intended to be deployed in the wild. as mentioned in this article, a human should always be involved despite the higher time and expense required, this is even more important during the initial development stage. can a machine determine lrmt quality better than a human? simply put, there is not substitute for a human in the lrmt creation loop. at this point in time, to our knowledge, there does not exist a lrmts that has achieved nearly the same performance as high-resource language pairs like english-german. while some bert-based @xcite mt systems that use transformers @xcite have achieved nearhuman performance when measured by bleu @xcite , it is not clear that is the case for lrmts or domain-specific situations as was shown in recent work @xcite in the clinical domain. ## plan b: workarounds given the realities of lrmt, it may be necessary to consider various workarounds in order to achieve quality that is good enough to deploy a minimum viable product. the next two sections consider two workarounds of many possible: (1) human-in-theloop and (2) subdomains. ## conclusions we provided a practical guide of challenges for companies considering the deployment of a lrmts. much of the work in our field focuses on english and other high-resource language, but recently, there has been more interest in low-resource languages. a number of systems support an amazingly large set of languages. that said, it is a mistake to deploy a non-viable system. adoption of lrmt can be limited by many factors and the question therein lies if the risks are worth the rewards. a company's minimum viable product requires sufficient demand with hundreds (if not thousands) of users in the low-resource community that are willing to use it. in addition to demand, we also discussed costs and quality. quality includes standard metrics in