Machine Translation Post-Editing Levels: Breaking Away from the Tradition and Delivering a Tailored Service (2020)

Abstract: While definitions of full and light post-editing have been around for a while, and error typologies like DQF and MQM gained in prominence since the beginning of last decade, for a long time customers tended to refuse to be flexible as for their final quality requirements, irrespective of the text type, purpose, target audience etc. We are now finally seeing some change in this space, with a renewed interest in different machine translation (MT) and post-editing (PE) service levels. While existing definitions of light and full post-editing are useful as general guidelines, they typically remain too abstract and inflexible both for translation buyers and linguists. Besides, they are inconsistent and overlap across the literature and different Language Service Providers (LSPs). In this paper, we comment on existing industry standards and share our experience on several challenges, as well as ways to steer customer conversations and provide clear instructions to post-editors.

## introduction as one of the largest multilingual lsps, we have been offering machine translation post-editing services for many years, and our team supports more than 30 of our largest customers in the enterprise or regulated space with mt and postediting programs in often 30+ language pairs. when implementing machine translation for a new customer, we always provide a post-editing training to the linguists working on the program. during this training, among other relevant topics, we focus on the basics of post-editing and we explain what the client's requirements are regarding final translation quality. based on experience, we found that it can be very complicated to communicate what is expected of linguists in different post-editing levels. while it is easier to explain what is expected of light versus full post-editing, there are some grey areas that don't fall either into the full post-editing or the light post-editing service. furthermore, our customers will often not be experts of translation quality assurance methodologies, and also not be familiar with the common definitions of the different levels of post-editing. as such, they are themselves often not entirely sure which approach would meet, exceed or fall short of their requirements. it is therefore crucial to guide them and define their requirements from the very outset, also in order to be able to clearly communicate them to the post-editors. this is extremely important since post-editors might feel confused if they do not receive clear instructions, and will probably end up delivering a quality that is either too highin this case they will not be productiveor too low and the clients' quality requirements will not be met. ultimately, the effort of the post-editor depends strictly on clients' quality requirements, therefore, it is not always advisable to rely exclusively on the current, most commonly used post-editing guidelines. in order to precisely define the quality requirements for each postediting task, we reference the dqf-mqm error matrix and the taus dqf content types to align all parties on what types of errors are acceptable for a translation request given its purpose, target audience etc. in this paper, we would like to share our experience on existing industry standards, challenges, and ways to steer customer conversations and provide clear instructions to post-editors. ## existing industry definitions and standards as for quality, two error typologies were proposed in the last years: dqf by @xcite and mqm by qt21, 2014 @xcite . these provide more flexible and dynamic ways to assess quality, and apply the same approach to machine translation and human translation. while they can also be used separately, the two typologies were brought together in 2015 into the mqm-dqf quality framework. these error typologies also aim to move away from the lisa qa model @xcite , used for a long time in the localization industry to rate translation quality. we like the dqf-mqm error typology because the error hierarchy, made up of well-structured main criteria and sub-criteria, allows for a granular categorisation of the quality issues in the translation. regarding post-editing, as mentioned above, it is common knowledge and generally accepted in the translation industry that there are different levels of post-editing, aimed at obtaining a final text that satisfies diverse predefined purposes and quality standards. however, there appears to be no recognised industry-wide standard and the definition and guidelines of each level of postediting are inconsistent across the literature and different lsps. currently, the most commonly used and referenced definitions of light and full post-editing in the localization industry are probably those provided by the international organization for standardization, gala, taus and sharon @xcite . while the last two were already analysed by hu and cadwell, we would like to summarize what iso and ga-la say on the different levels of post-editing, before we proceed with the comparative analysis. ## challenges with existing definitions of post-editing the main challenge with terms like "light", "medium" and "full" is that they remain very abstract. hu and cadwell showcased already in 2016 that the literature seems to offer inconsistent and/or overlapping nomenclature, definitions and guidelines for post-editing. having said this, it seems to be broadly accepted that light post-editing should focus on conveying the meaning of the source text in an accurate way. therefore, if we had to use the dqf-mqm error types as a referenceinstead of the categories from the lisa qa model (localization industry standards association quality assurance model) and sae (society of automotive engineers) j2450 translation quality metric, as suggested by hu and cadwellwe could say that light post-editing should focus on fixing accuracy error types, while it should not focus on style, design, locale convention and verity error types, as long as the information is delivered accurately. whether terminology, spelling and grammar errors should be penalized in a light post-editing task seems to remain controversial and unclearthe requirements as for these error types are inconsistent (see appendix a). on the other hand, if we consider the findings from hu and cadwell as well as the iso standard no. 18587, it seems to be broadly accepted that full post-editing should focus on readability. however, there appears to be no common agreement as to whether full-post editing should be of equal quality to conventional human translation from scratch. considering the different guidelines we analysed, we could say that full post-editing should focus on accuracy, fluency and terminology error types. style is discussed controversially, as there is no agreement on its importance between all the different guidelines (see appendix b). according to the taus guidelines, the style "may not be as good as that achieved by a native-speaker human translator", 1 while stylistic and textuality problems should be ignored according to o'brien. on the other hand, we read that the iso standard no. 18587 recommends that client's stylistic guidelines are followed, and highlights that the style should be appropriate for the text type. lastly, gala simply points out that the style should be consistent and appropriate. some lsps also provide "medium postediting" services, but the guidelines for this quality level are even more vague and inconsistent, and this level of post-editing is mentioned only sporadically in the literature. generally speaking, when performing medium post-editing, we expect the post-editor to put more effort into editing terminology, fluency and style compared to light post-editing, but not to the same extent as they would for full post-editing. there appear to be no medium post-editing definitions in the literature that we could reference here; the easiest way to derive a distinction between full and medium post-editing, for instance, might be via the text type and translation purpose, i.e. texts that 1 https://www.taus.net/academy/bestpractices/postedit-best-practices/machine-translation-postediting-guidelines are more stylistically challenging and complex by definition would always require full postediting, whereas text types with a simpler structure (often technical manuals) could fold into medium post-editing. however, this might ultimately be an unnecessary definition as such, that could also be covered by the full post-editing requirement for "appropriate" style. as we can see, these guidelines leave some grey areas when it comes to a hands-on postediting task. for example, if i am performing full post-editing, should i check that bullets are consistent in the same list of items? that the headers are all title case? and what happens if i notice that the target language is using masculine form for example, "amigo" in spanishwhen the source language might refer both to feminine and masculine genderfor example, "friend" in english? should i edit all of these, or is it ok to leave those as they are? there is also a potential problem in that existing definitions appear to assume that only one linguist should ever post-edit the machine translated outputirrespective of full or light. in other words, there appear to be no guidelines specifying how many linguists should be involved in the different post-editing levels, and the iso standard no. 18587 does not set any requirement in this sense. it just mentions the requirement of a process to make sure that the final product meets the specifications. in the localization industry, however, it is still very common for translation buyers to enshrine a so-called "4-eye process" in the contract, i.e. irrespective of mt, that content needs to undergo post-editing plus a separate review or revision step, and potentially even a quality assurance step, which in some cases might be performed by a third party. in other words, customers still tend to buy a specific process (tep, translation only, etc.), rather than an agreed service level or translation quality. this becomes even more stringent in the regulated sectors, i.e. patent, life sciences, finance etc., where these additional steps can be mandatory to comply with other iso standards and certification requirements. another challenge with the terms "light" and "full" post-editing is that often people misunderstand that these describe how much editing needs to be done, or in other words, how much effort the post-editor should put into the task, rather than what the final translation quality should be. more precisely, some people might erroneously think that, if they were to translate the same content in multiple languages, depending on the quality of the raw mt output, some languages will require light post-editing while others will require full post-editing. for example, usergenerated content machine translated into spanish will require light post-editing as the raw output's quality is good, while finnish will require full post-editing, because the raw output isn't as good as for spanish. this is a fairly common misunderstanding and yet another reason why we think it is better to focus on final translation quality requirements, than the vaguer definitions of light and full pe. ## challenges with error typologies since the lack of a clear, common approach highlighted by @xcite is still a very present issue, translation service providers need to define their own methodology, in order to provide a flexible service offering, linked to transparent pricing for the client and fairer rates for the post-editors. what is ultimately needed is a highly flexible and granular approach, since the effort of the post-editor is essentially decided by the exact quality requirements of a given customer. rather than working with the somewhat vague definitions of "light", "medium" and "full" postediting, we find that it is easier for all parties to define quality requirements by aligning on what types of errors are admissible for a translation request given its purpose, target audience etc. considering the purpose of the text and the document type, and referencing the dqf-mqm error matrix, we help the clients choose what error categories are acceptable for them and what are not. also, for each error category they decide how many (if any) major and/or minor errors they are admissible. we use the same framework for quality assurance (qa) steps to understand if the quality of the mtpe projects meets client's requirementsthis way the linguists performing this task are fully aware of what they should focus on and we get full consistency as for quality requirements from the start until the end of the process. in order to make this possible, first we created different groups of domains, considering the purpose of the document and the text type (based on taus dqf content types), and then we created different sets of standard checks for each one of these groups, aimed at getting a translation which is free of certain predefined unacceptable errors. for example, for user-generated content, we could propose a set of post-editing checks that focuses purely on the accurate transfer of meaning. user-generated content would be an example of text type typically accepting a high error thresholdespecially in light of the source input itself being known for being characterized by errors (o'curran, 2014). at the other end of the spectrum we might find text types such as marketing materials with a focus on brand's style and tone of voice. we like to call the abovementioned sets of standard checks "full", "medium" and "light" post-editing too, as our guidelines show some similarity with the most popular industry mtpe guidelines mentioned above. then, building on these sets of pre-defined standard checks, we add or remove applicable error categories as per client's preferences, and we raise or lower the threshold of the acceptable number of minor and/or major errors. as mentioned above, the error categories are also based on the dqf-mqm error typology. the dqf-mqm framework involves the use of a list of error categories, and the content quality is judged based on the amount and severity of the errors found. the errors can have different severity levels: critical, major, minor and neutral. "neutral" applies when an issue should be flagged to the translator but is not counted as an error and does not influence whether the translation is considered a pass or a fail. during our qa step, a post-edited text (or a sample of it) is evaluated by a linguist who marks the errors; all errors are added up, based on severity, and output a pass or fail score, depending on the defined threshold. the thresholds are flexible and depend on content type, text purpose and perishability of the text. in practice, this strategy is extremely helpful, as we can agree with customers, post-editors and reviewers at a very granular level what issues need to be addressed during postediting, and which are of purely preferential nature. however, while error typologies for quality assurance are fairly common among professional translators and reviewers, it can be trickier to agree on error categories and severities with translation buyers. this is primarily due to the fact that the owner of a given machine translation initiative on customer side may not be an expert in translation quality assurance methodologies. on the other hand, on post-editor and reviewer side, the main challenge is changing the mindset, and getting professional translators to accept that for certain content types and translation purposes, it is acceptable to leave certain types of issues in the machine translation output unedited. however, by providing a granular breakdown of what constitutes an error in a given translation request, it is much easier to train and support post-editors, and to monitor their actual productivity for the task at hand. ## use cases examples and strategies as we have seen on a high level, clients often have specific requirements that cannot easily and universally be categorized with the typical definitions of full, medium or light post-editing. in the following section, we will showcase some examples to explain our approach: based on the purpose of the document, the content type and the error types that the client is willing to accept or not, we build custom requirements and instructions for post-editors. ## how we provide guidance and set expectations in order to help clients understand what service level best fits their needs and to make it clear to post-editors what is required of them, there are different strategies an lsp can put in place. ## conclusions there is no gold standard for post-editing guidelines nor universally applicable definitions of different post-editing services. while still being useful for initially steering conversations, we saw that the generic guidelines overlap in key aspects. at this point in time, we find that in order to effectively communicate with different stakeholders in the localization industry, it is necessary to refer both to definitions of light, medium, full post-editing, but to also supplement these with very hands-on, practical definitions of what constitutes an error in a given scenario, and how quality assurance is provided. instructions, error categories and penalty thresholds need to be defined on a case-by-case basis with customers and need to be communicated very clearly to post-editors. metrics such as ter or edit distance can help analyse and monitor the actual post-editing effort, and can be used to fine-tune and revisit requirements, productivity expectations and fair compensation. o'brien 2010 rowda 2016 taus 2016 iso 2017 addition the mes s a ge tra ns ferred s houl d be a ccura te accura cy i s key ens ure tha t no i nforma ti on ha s been a cci denta l l y a dded or omi tted ens ure tha t no i nforma ti on ha s been a dded or omi tted omission the mes s a ge tra ns ferred s houl d be a ccura te accura cy i s key ens ure tha t no i nforma ti on ha s been a cci denta l l y a dded or omi tted ens ure tha t no i nforma ti on ha s been a dded or omi tted mistranslation the mes s a ge tra ns ferred s houl d be a ccura te accura cy i s key ai m for s ema nti ca l l y correct tra ns l a ti on res tructure s entences i n the ca s e of i ncorrect or uncl ea r mea ni ng over-translation the mes s a ge tra ns ferred s houl d be a ccura te accura cy i s key ens ure tha t no i nforma ti on ha s been a cci denta l l y a dded or omi tted ens ure tha t no i nforma ti on ha s been a dded or omi tted under-translation the mes s a ge tra ns ferred s houl d be a ccura te accura cy i s key ens ure tha t no i nforma ti on ha s been a cci denta l l y a dded or omi tted ens ure tha t no i nforma ti on ha s been a dded or omi tted untranslated text the mes s a ge tra ns ferred s houl d be a ccura te accura cy i s key improper exact tm match punctuation va ri a ti ons i n s tyl e, punctua ti on, a nd s pel l i ng a re ok spelling al l ba s i c rul es rega rdi ng s pel l i ng s ti l l a ppl y va ri a ti ons i n s tyl e, punctua ti on, a nd s pel l i ng a re ok ba s i c rul es a ppl y grammar not a bi g concern, unl es s gra mma ti ca l probl ems i nterfere wi th a ccura cy ma y not be perfect grammatical register inconsistency link/cross-reference character encoding inconsistent with termbase do not s pend ti me res ea rchi ng terms inconsistent use of terminology do not s pend ti me res ea rchi ng terms awkward ignore s tyl i s ti c probl ems va ri a ti ons i n s tyl e, punctua ti on, a nd s pel l i ng a re ok no need to i mpl ement correcti ons tha t a re of a s tyl i s ti c na ture onl y need not be s tyl i s ti ca l l y a dequa te company style ignore s tyl i s ti c probl ems va ri a ti ons i n s tyl e, punctua ti on, a nd s pel l i ng a re ok no need to i mpl ement correcti ons tha t a re of a s tyl i s ti c na ture onl y need not be s tyl i s ti ca l l y a dequa te inconsistent style ignore s tyl i s ti c probl ems va ri a ti ons i n s tyl e, punctua ti on, a nd s pel l i ng a re ok no need to i mpl ement correcti ons tha t a re of a s tyl i s ti c na ture onl y need not be s tyl i s ti ca l l y a dequa te third-party style ignore s tyl i s ti c probl ems va ri a ti ons i n s tyl e, punctua ti on, a nd s pel l i ng a re ok no need to i mpl ement correcti ons tha t a re of a s tyl i s ti c na ture onl y need not be s tyl i s ti ca l l y a dequa te unidiomatic ignore s tyl i s ti c probl ems va ri a ti ons i n s tyl e, punctua ti on, a nd s pel l i ng a re ok no need to i mpl ement correcti ons tha t a re of a s tyl i s ti c na ture onl y need not be s tyl i s ti ca l l y a dequa te length local formatting markup missing text truncation/text expansion address format date format currency format measurement format shortcut key telephone format veri ty culture-specific reference edi t a ny offens i ve, i na ppropri a te or cul tura l l y una ccepta bl e i nforma ti on edi t a ny offens i ve, i na ppropri a te or cul tura l l y una ccepta bl e content edi t a ny i na ppropri a te content other fi x ma jor/bl a ta nt errors onl y. mi nor i s s ues a re a ccepta bl e. avoi d s tyl i s ti c a nd preferenti a l cha nges . throughput expecta ti ons : very hi gh qua l i ty expecta ti ons : l ow fa s t turn-a round us e a s much of the ra w mt output a s pos s i bl e us e a s much of the ra w mt output a s pos s i bl e error type loca l e conventi on accura cy fl uency termi nol ogy styl e des i gn appendix b. comparative analysis of full pe guidelines based on dqf-mqm framework o'brien 2010 rowda 2016 taus 2016 iso 2017 addition the mes s a ge tra ns ferred s houl d be a ccura te no i nforma ti on ha s been a cci denta l l y a dded or omi tted no i nforma ti on ha s been a dded or omi tted omission the mes s a ge tra ns ferred s houl d be a ccura te no i nforma ti on ha s been a cci denta l l y a dded or omi tted no i nforma ti on ha s been a dded or omi tted mistranslation the mes s a ge tra ns ferred s houl d be a ccura te al l mi s tra ns l a ti ons fi xed ai m for gra mma ti ca l l y, s ynta cti ca l l y a nd s ema nti ca l l y correct tra ns l a ti on res tructure s entences i n the ca s e of i ncorrect or uncl ea r mea ni ng over-translation the mes s a ge tra ns ferred s houl d be a ccura te al l mi s tra ns l a ti ons fi xed no i nforma ti on ha s been a cci denta l l y a dded or omi tted no i nforma ti on ha s been a dded or omi tted under-translation the mes s a ge tra ns ferred s houl d be a ccura te al l mi s tra ns l a ti ons fi xed no i nforma ti on ha s been a cci denta l l y a dded or omi tted no i nforma ti on ha s been a dded or omi tted untranslated text the mes s a ge tra ns ferred s houl d be a ccura te untra ns l a ted terms bel ong to the cl i ent's l i s t of "do not tra ns l a te" terms improper exact tm match punctuation al l ba s i c rul es rega rdi ng s pel l i ng, punctua ti on a nd hyphena ti on s ti l l a ppl y ba s i c rul es rega rdi ng s pel l i ng, punctua ti on a nd hyphena ti on a ppl y appl y s pel l i ng, punctua ti on a nd hyphena ti on rul es spelling al l ba s i c rul es rega rdi ng s pel l i ng, punctua ti on a nd hyphena ti on s ti l l a ppl y deta i l ed correcti ons , no gra mma r or s pel l i ng errors s houl d be i gnored ba s i c rul es rega rdi ng s pel l i ng, punctua ti on a nd hyphena ti on a ppl y appl y s pel l i ng, punctua ti on a nd hyphena ti on rul es grammar gra mma r s houl d be a ccura te deta i l ed correcti ons , no gra mma r or s pel l i ng errors s houl d be i gnored ai m for gra mma ti ca l l y, s ynta cti ca l l y a nd s ema nti ca l l y correct tra ns l a ti on produce gra mma ti ca l l y, s ynta cti ca l l y a nd s ema nti ca l l y correct ta rget l a ngua ge content grammatical register ai m for gra mma ti ca l l y, s ynta cti ca l l y a nd s ema nti ca l l y correct tra ns l a ti on ens ure tha t the s tyl e a ppropri a te for the text type i s us ed a nd tha t s tyl i s ti c gui del i nes provi ded by the cl i ent a re obs erved inconsistency link/cross-reference character encoding inconsistent with termbase ens ure tha t key termi nol ogy i s correctl y tra ns l a ted accura te termi nol ogy key termi nol ogy i s correctl y tra ns l a ted adhere to cl i ent a nd/or doma i n termi nol ogy inconsistent use of terminology ens ure tha t key termi nol ogy i s correctl y tra ns l a ted accura te termi nol ogy correcti ng i ncons i s tenci es i n termi nol ogy, termi nol ogy di s a mbi gua ti on adhere to cl i ent a nd/or doma i n termi nol ogy awkward ignore s tyl i s ti c a nd textua l i ty probl ems styl e s houl d be cons i s tent a nd a ppropri a te ma y not be a s good a s tha t a chi eved by a na ti ves pea ki ng huma n tra ns l a tor ens ure tha t the s tyl e a ppropri a te for the text type i s us ed a nd tha t s tyl i s ti c gui del i nes provi ded by the cl i ent a re obs erved company style ignore s tyl i s ti c a nd textua l i ty probl ems styl e s houl d be cons i s tent a nd a ppropri a te ma y not be a s good a s tha t a chi eved by a na ti ves pea ki ng huma n tra ns l a tor ens ure tha t the s tyl e a ppropri a te for the text type i s us ed a nd tha t s tyl i s ti c gui del i nes provi ded by the cl i ent a re obs erved inconsistent style ignore s tyl i s ti c a nd textua l i ty probl ems styl e s houl d be cons i s tent a nd a ppropri a te ma y not be a s good a s tha t a chi eved by a na ti ves pea ki ng huma n tra ns l a tor ens ure tha t the s tyl e a ppropri a te for the text type i s us ed a nd tha t s tyl i s ti c gui del i nes provi ded by the cl i ent a re obs erved third-party style ignore s tyl i s ti c a nd textua l i ty probl ems styl e s houl d be cons i s tent a nd a ppropri a te ma y not be a s good a s tha t a chi eved by a na ti ves pea ki ng huma n tra ns l a tor ens ure tha t the s tyl e a ppropri a te for the text type i s us ed a nd tha t s tyl i s ti c gui del i nes provi ded by the cl i ent a re obs erved unidiomatic ignore s tyl i s ti c a nd textua l i ty probl ems styl e s houl d be cons i s tent a nd a ppropri a te ma y not be a s good a s tha t a chi eved by a na ti ves pea ki ng huma n tra ns l a tor ens ure tha t the s tyl e a ppropri a te for the text type i s us ed a nd tha t s tyl i s ti c gui del i nes provi ded by the cl i ent a re obs erved length ens ure tha t forma tti ng i s correct appl y forma tti ng rul es local formatting ens ure tha t forma tti ng i s correct appl y forma tti ng rul es markup for ta gged forma ts , ens ure a l l ta gs a re pres ent a nd i n the correct pos i ti ons ens ure tha t forma tti ng i s correct appl y forma tti ng rul es missing text ens ure tha t forma tti ng i s correct appl y forma tti ng rul es truncation/text expansion ens ure tha t forma tti ng i s correct appl y forma tti ng rul es address format ha ndl i ng of mea s urements a nd l oca l e-s peci fi c punctua ti on, da te forma ts a nd a l i ke date format ha ndl i ng of mea s urements a nd l oca l e-s peci fi c punctua ti on, da te forma ts a nd a l i ke currency format ha ndl i ng of mea s urements a nd l oca l e-s peci fi c punctua ti on, da te forma ts a nd a l i ke measurement format ha ndl i ng of mea s urements a nd l oca l e-s peci fi c punctua ti on, da te forma ts a nd a l i ke shortcut key ha ndl i ng of mea s urements a nd l oca l e-s peci fi c punctua ti on, da te forma ts a nd a l i ke telephone format ha ndl i ng of mea s urements a nd l oca l e-s peci fi c punctua ti on, da te forma ts a nd a l i ke veri ty culture-specific reference edi t a ny offens i ve, i na ppropri a te or cul tura l l y una ccepta bl e i nforma ti on edi t a ny offens i ve, i na ppropri a te or cul tura l l y una ccepta bl e content edi t a ny i na ppropri a te content other reta i n a s much ra w tra ns l a ti on a s pos s i bl e us e a s much of the ra w mt output a s pos s i bl e us e a s much of the mt output a s pos s i bl e throughput expecta ti ons : hi gh qua l i ty expecta ti ons : medi um cl os e to huma n tra ns l a ti on qua l i ty produce a n output whi ch i s i ndi s ti ngui s ha bl e from huma n tra ns l a ti on output error type loca l e conventi on accura cy fl uency termi nol ogy styl e des i gn