How do LSP s compute MT discounts? Presenting a company’s pipeline and its use (2020)

Abstract: In this paper we present a pipeline developed at Acolad to test a Machine Translation (MT) engine and compute the discount to be applied when its output is used in production. Our pipeline includes three main steps where quality and productivity are measured through automatic metrics, manual evaluation, and by keeping track of editing and temporal effort during a post-editing task. Thanks to this approach, it is possible to evaluate the output quality and compute an engine-specific discount. Our test pipeline tackles the complexity of transforming productivity measurements into discounts by comparing the outcome of each of the above-mentioned steps to an estimate of the average productivity of translation from scratch. The discount is obtained by subtracting the resulting coefficient from the per-word rate. After a description of the pipeline, the paper presents its application on four engines, discussing its results and showing that our method to estimate post-editing effort through manual evaluation seems to capture the actual productivity. The pipeline relies heavily on the work of professional post-editors, with the aim of creating a mutually beneficial cooperation between users and developers.

## introduction over the last few years, the number of companies starting to integrate machine translation (mt) c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. in their workflow has increased dramatically. in 2018, for the first time, more than half of the companies and individual professionals taking part in the language industry survey stated that they used mt @xcite . the same survey repeated in 2019 showed that mt was one of the highest priorities for companies, with 51% of them willing to increase its use and 62% stating that they were planning investments on mt @xcite . at the same time, all categories involved in the 2019 language industry survey mentioned price pressure as the main negative trend, with mt and post-editing identified as one of the major causes. in the previous year, one of the main concerns of the language industry components were the "technological advances that are not initiated or controlled by the respondents" @xcite . these surveys point out that individuals and companies are embracing a technology that they themselves see as a threat to the sustainability of the translation industry. this seems to suggest that there might be a strong disagreement regarding the price models to be adopted when mt is included in the workflow. one of the priorities of mt users and developers should therefore be the creation of shared models and/or methods to measure quality and productivity with a view to computing discounts to be applied when mt is used. however, to the best of our knowledge this topic has been underinvestigated so far. the "pricing machine translation post-editing guidelines" published by taus have tried to fill the gap in this field. 1 they claim that a model to price post-editing should be predictive (able to es-timate prices in advance), fair (involving all parties and providing them with a reliable estimate), and appropriate for both the language pair and the content characteristics. to achieve this aim, three main indicators should be combined: human evaluation, automatic metrics, and productivity assessments. despite the usefulness of these guidelines, many questions on how to convert quality scores and/or judgements into rates -especially wordbased ones -or discounts remain unanswered. for example, if automatic metrics are used, what would a ter of 0.4 imply in terms of productivity? if a sentence is evaluated as very fluent by a human translator, how could this judgement be converted into a discount? at the same time, if productivity assessments on a machine translated sample reveals that, e.g., 200 words per hour were post-edited, what would be the most reliable and consistent way to turn this productivity rate into a fair per-word rate for post-editing? indeed, measuring post-editing productivity might be a difficult task per se, since post-editing effort is multidimensional and composed of time needed to edit a text (temporal effort), number of edits performed (editing effort) and identification and correction of issues (cognitive effort) @xcite . computing discounts adds an additional challenge, i.e. converting productivity measures into post-editing rates. building upon taus guidelines, in the present work we introduce a pipeline to estimate an engine-specific discount to be applied to per-word rates for post-editing. the pipeline is composed of different steps that involve the use of automatic metrics, post-editing effort measurements and human evaluation processes, carried out as follows: ## related work quality estimation (qe) models -i.e. models trained to output a quality indicator of a translation without the need of comparing it to a referencehave been examined by a large amount of papers in the last decade and would serve the need of estimating the output quality, the post-editing time or the amount of editing needed on a whole text or on each of its sentences @xcite @xcite . despite the relevance of this line of research for the present paper, we are not aware of qe models aimed at predicting discounts or post-editing effort that are ready to be integrated in cat tools, which would be key for translation companies. one exception is memsource mtqe functionality, which computes a match rate for machine translated sentencessimilarly to what cat tools usually do for translation memory (tm) matches. however memsource mtqe is still in a beta version, and as of today it is not able to estimate quality for all language pairs and sentences. 2 companies operating in the translation field have published articles on their experience in computing discounts for post-editing. @xcite describes an experiment carried out in the framework of the matecat project aimed to understand the maximum discount translators are willing to accept for post-editing tasks. 3 translators working on different language combinations were asked to choose between translating 1,000 words from scratch or post-editing a lower number of words at the same per-word rate. the number of words paid for post-editing was gradually increased until at least 75% of the translators chose post-editing over translation from scratch. however, the author claimed that this method was not suitable for establishing discount rates, since for some language combinations translators chose post-editing over translation only when the former was paid more than the latter. the article then suggests a combination of editing effort, temporal effort and output quality to understand the usefulness of mt for post-editors. a rather straightforward approach at computing post-editing discounts is the one presented by @xcite in an sdl blog post. 4 the author suggests that a post-editing test is set up where a linguist is asked to post-edit a text sample for one hour. after that, the post-editor hourly rate is divided by the number of translated words to obtain the discount. however, since post-editing rates should be computed based on the content and language pair (see section 1), this test should be carried out every time a post-editor is working on a new content, or on a text belonging to a different domain or on a target language they have never post-edited. this would not be the most efficient solution. the different approaches presented here suggest that developing a reliable method to compute postediting discounts is not an easy task. as a matter of fact, to the best of our knowledge, as of today the translation world lacks a shared model to establish mt discounts upfront. in the next sections we will present the pipeline developed and currently in use at acolad to compute an engine-specific discount (see section 3). to gain a better insight into its functioning, four use cases are also presented (see section 4). ## pipeline description our pipeline is mainly composed of the three steps that were introduced in section 1. these will be detailed in the following sections, together with a final step in which the final coefficient is computed. ## discussion we have presented four use cases of our pipeline to test an mt engine and estimate the productivity increase that can be reached when its output is used in post-editing tasks. results in the small differences between the rct coefficient and the manual evaluation coefficient seem to suggest that the approach adopted to estimate post-editing effort through manual scoring reflects the actual temporal (and cognitive) effort. this might not always be the case as testified by the fashion en-de engine. therefore, keeping both coefficients and computing an average of the two seems to be the best solution. ## conclusions and future work the present work aimed at introducing our test pipeline to measure mt quality and enginespecific discounts. future work might present a higher number of use cases than those described in section 4.2, in order to provide a better overview of how the pipeline works. also, adding exam-ples of raw sentences together with the score assigned to them during the manual evaluation and their post-edited versions can better explain differences between the two coefficients or between the two outcomes of the rct. however, it has to be taken into account that it is often not possible to provide such examples due to confidentiality issues. the feedback module described in section 3.3 has been recently added to our pipeline. before that, post-editors were only asked to provide a general feedback on the output quality and its main issues. however, such general feedback was often too generic and/or difficult to interpret. for this reason, in this paper we are not reporting the feedback received, except for the fashion en-de engine in section 4.2. future work could compare hter and the coefficients with the results of the new feedback module, discussing the accuracy and fluency ratings assigned to the four engines. our mt group is constantly working on the improvement of our processes, including improvements of our test pipeline. in the future, languagespecific or domain-specific formulas to compute the two coefficients could be developed, since productivity can be highly influenced by the text and/or domain post-editors are working on. also, as introduced in section 3.3, using an average productivity increase to compute discounts to be applied to the work of any post-editor does not take into account differences between professionals. several studies on post-editing concluded that there is a high degree of variability between post-editors when it comes to productivity @xcite @xcite . however, we fully agree with guerberof (2014), who argues that using an average productivity increase to compute discounts for all post-editors is not beneficial for less productive post-editors, but further maintains that this is what happens with fuzzy match discounts as well, and finding a solution is not an easy task. to improve the fairness of our discounts, when allowed by deadlines and by the budget, we plan to involve more than one linguist per stage. to conclude, the authors are aware of the need to develop a pipeline that can be applied to all production tasks involving post-editing. producing continuous reports on the output quality of an engine -and how this influences productivity -is of the essence for an increasingly efficient and fair use of mt, similarly to what has been introduced by taus dynamic quality framework (dqf). 6 however, developing such a complex pipeline will require a long research and pilot phase. these possible improvements should not hide the fact that, to the best of our knowledge, this is the first work presenting a company's approach to test engines and method of computing its specific discount rates, reporting also on examples of the practical use of this pipeline.