E nglish to M anipuri and Mizo Post-Editing Effort and its Impact on Low Resource Machine Translation (2020)

Abstract: We present the first study on the post-editing (PE) effort required to build a parallel dataset for English-Manipuri and English-Mizo, in the context of a project on creating data for machine translation (MT). English source text from a local daily newspaper are machine translated into Manipuri and Mizo using PBSMT systems built in-house. A Computer Assisted Translation (CAT) tool is used to record the time, keystroke and other indicators to measure PE effort in terms of temporal and technical effort. A positive correlation between the technical effort and the number of function words is seen for English-Manipuri and English-Mizo but a negative correlation between the technical effort and the number of noun words for English-Mizo. However, average time spent per token in PE English-Mizo text is negatively correlated with the temporal effort. The main reason for these results are due to (i) English and Mizo using the same script, while Manipuri uses a different script and (ii) the agglutinative nature of Manipuri. Further, we check the impact of training a MT system in an incremental approach, by including the post-edited dataset as additional training data. The result shows an increase in HBLEU of up to 4.6 for English-Manipuri.

## introduction in our increasingly globalized world, communication plays a vital role and with it, demand for translation between different languages is on the rise. despite much progress, machine translation (mt) on its own may not always be sufficient to meet the demand. mt output may sometimes be erroneous and needs to be checked and corrected. the use of translation technology such as mt systems, transla-tion memories (tm) and cat tools can boost translation productivity @xcite . however, limited numbers of professional translators for a language pair can be a major challenge, especially for low resource languages. raw mt output is not always exempt from errors. often post-editing mt output (where a human translator reviews and where required corrects mt output) is the most productive approach to translation. pe effort is the amount of effort required to generate a reasonable target text from mt output. following krings (2001) pe effort can be subdivided into temporal effort, technical effort and cognitive effort. temporal effort represents the overall time taken to complete a pe task. technical effort can be measured tracking keyboard and mouse interactions, including insertion, deletion, mouse movement, etc. cognitive effort (considered the most difficult to measure) involves mental effort such as reading and understanding the text, identifying errors, and the decision making process towards correcting errors. to date, pe research has mainly concentrated on a few well-studied languages. in this work, the same source data in english are machine translated to low resource languages to carry out a pe task. the dataset used in the experiment consists of news articles collected from a local daily newspaper, imphal free press 1 in manipur, a northeastern state of india with a population of around 3 million 2 and a geographic size of 22,327 sq. km. the collected news corpus is originally in english and then machine translated into three english manipuri mizo hindi svo sov osv sov roman bengali roman devanagari acronyms: @xmath0, @xmath1, @xmath2. we conduct a study on the pe effort required to produce reasonable target text in english-manipuri and english-mizo. as there is no commercially available machine translation system for these two languages, for comparison we also studied english-hindi pe effort on commercial mt output on the same dataset. two levels of pe are generally distinguished: light and full. for our experiments, we instruct our post-editors to carry out light pe to achieve the desired level of output quality. with various pe effort indicators computed using the data captured from the cat tool, we carry out an experiment to measure the pe effort for english-manipuri, english-mizo and english-hindi mt systems. lexical words and function words are observed to have a different impact on the pe effort on the mt output for the three language pairs. we also carry out an experiment to test the impact of training a machine translation system for the low resource language pairs english-manipuri and english-mizo in an incremental manner, i.e. by adding the peed dataset to the original mt training data. the rest of the paper is structured as follows: section 2 reviews previous research, section 3 desrcibes the pe task and the human peers, section 4 presents our approach and system set up. section 5 details our findings. section 6 summarizes our main results and avenues for further research. ## related work early studies on the correlation between pe effort and various aspects of pe include @xcite . o'brien studied the temporal, technical and cognitive effort involved in pe by analyzing keyboard-data using translog and choice network analysis (cna). several studies investigated bi-lingual pe and monolingual pe. in bilingual pe @xcite post editors have access to the source text, while in monolingual pe @xcite mt output is edited without the source text. @xcite studied the use of tms generated by mt output and their effect on human translation. the authors reported a significant increase in translation speed while using the tm as compared to translating from the scratch. similarly, @xcite show that postediting an mt dataset involves less effort than translating from scratch. post-editing mt output increases the productivity of the translators. @xcite examine various types of mt errors and the challenges they present for pe. @xcite compile a corpus consisting of english to german translation generated by different types of mt systems. the dataset is then annotated for translation errors using the mqm error typology, with only one error in each sentence. as the dataset is already annotated, the posteditors could skip the effort of identifying the errors and concentrate only on the highlighted error text segment in the pe process. focusing on how pe effort changes with the different types of mt errors, the authors reported a weak correlation between pe time and pe effort. the authors also report that no direct dependency was found between the temporal and technical pe effort. investigating the various types of pe operations for french to english and english to spanish translation outputs, @xcite reported lexical edits as the main factor in pe time. @xcite , study the cognitive effort of post-editing mt output based on measuring pe time and hter @xcite . hter (human-targeted translation edit rate), is an automatic metric that computes the minimum number of edits required to change mt output into the post-edited version. the authors reported that the absolute pe time increases with the number of printable keystrokes and sentence length while seconds per word remain relatively constant. despite the fact that hter captures the difference between the final translation and raw mt, it does not disclose much of the time and keystroke effort required to produce the final result. a similar study is also reported by @xcite where the human (or h-) variants of the reference based similarity measure such as bleu @xcite is used to analyze pe effort. ## description of the pe task two post-editors who are native speakers of the target languages and also proficient with the source language are employed for each of the language pairs to carry out the pe task. for english-manipuri and english-hindi, the post-editors are undergraduate students of computer science and engineering and for english-mizo, the post-editors are postgraduate students of science. when peing machine translated text, it is important to clearly define what level of output quality is to be achieved. generally two pe levels are distinguished: light or complete. in our work, the post-editors are asked to carry out light pe with the following instructions: 1) using the maximum possible amount of raw mt text in the output of pe. 2) ensure no addition or omission of source content. 3) restructuring output, where the meaning is inaccurate. ## methodology and experimental design we use an english language corpus collected from a local daily newspaper as the source text. we normalize the data in a pre-processing step. the normalized text is then machine translated into three different target languages using different mt systems. after post-editing a sample dataset of the machine translated text using a cat tool, we study the data collected ## results and discussion to measure pe effort, pe logs are collected from the cat tool after post-editing the raw mt output (in manipuri, mizo and hindi) of the sample dataset, d p e . we compare the general distribution of pet and pee for each of the 200 sentences in the language pairs in the form of box plots. mean and standard deviations for the rest of the source pe effort indicators (tt, nn, lw, ft) and the target pe effort indicators (pet, pee, spw ) are computed and we investigate correlations between indicator variables. ## conclusions using log-information gathered from our cat tool, an analysis of the pe effort and pe time is carried out for three target languages: manipuri, mizo and hindi with english as the source language. to our knowledge, this is the first pe analysis conducted on english-manipuri, english-mizo and english-hindi. our analysis shows that current state of the art in commercially available mt for english-hindi requires small pe effort and pe time. while mt systems for low resource languages such as manipuri and mizo are under development, mt training data for the languages is very scarce. using a pbsmt system built in-house, a study on the pe effort and pe time is carried out for english-manipuri and english-mizo. our findings show that, compared to english-manipuri and english-mizo, pee is low for english-hindi. by contrast, for english-manipuri and english-mizo, the problems in mt output are far more serious requiring heavy pe effort. interestingly, while there is a significant correlation between pee and ft for the language pair english-manipuri and english-mizo, there is a significant negative correlation between pee and nn for the english-mizo language pair. the pee for english-mizo decreases with the increase in noun words in the source text, which might be because of mizo sharing the same script as the source language. also, a significant negative correlation is observed between spw and tt for english-mizo. this suggests that with the increase in the number of tokens in source text, the average time taken per word decreases for english-mizo. we identify mt quality as well as script and ease of typing script as a factor in pe effort for languages like manipuri and mizo. we also made a first attempt to address the scarcity of a parallel training data of english-manipuri and english-mizo mt by training the mt systems in an incremental manner using additional data created by the pe experiment. the result indicates an improvement of up to 4.6 in hbleu for english-manipuri.