Persistent MT on software technical documentation - a case study (2020)

Abstract: We report on the features and current challenges of our on-going implementation of a Persistent MT workflow for Citrix Product Documentation, to increase localization coverage to 100% content in docs.citrix.com into German, French, Spanish, Japanese and Simplified Chinese. By the end of 2019, we had processed seven million words of English documentation with this model, across 24 doc sets, and raised localization coverage from 40% to 100% of the content of our documentation repositories. This has boosted our global reach across the entire Citrix portfolio (Digital Workspace, Networking, and Analytics). The current implementation requires a process of Light Post-editing (LPE) for all languages, in order to fix over-translations, out-of-domain words, inline tags and markdown errors in the raw output.

## background the localization team at citrix systems introduced machine translation in its documentation workflows in 2013, with an in-house implementation of moses engines from english into german, french, and spanish. statistical mt was the basis for a full post-editing workflow performed by inhouse linguists and a small pool of trusted contractors. with the advent of neural machine translation in 2017, we switched to google generic enc 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 citrix tier-1 languages: german, french, spanish, japanese and simplified chinese. productivity increased dramatically since introducing nmt as a base for full post-editing. this had a positive impact on time-to-market and reduction of localization costs. however, the actual coverage of our localization infrastructure remained at 40% of all the citrix product documentation volume, mainly into tier-1 languages, with a few doc sets into korean and brazilian portuguese. the cost of sustained localization of the remaining content using mt plus human full post-editing was prohibitively high. it was evident that just integrating mt in the workflows was not enough to produce global-ready content at the increasing speeds and volumes necessary in today's competitive market. ## the plan encouraged by the better fluency and adequacy of nmt output, at the end of 2018, it was proposed to expand our localization coverage to the remaining 60% wordcount of our documentation by using raw mt (mt without post-editing). by then we were familiar with microsoft's successful implementation of raw mt in their technical and end-user support documentation, which microsoft now considers "proven for all support content types" @xcite . we particularly liked some of the front-end features of their deployment in docs.microsoft.com, like the switch-to-english toggle and the banner with the mt warning, and we looked at the microsoft case as a good reference for our implementation. we have since come across the adobe foot_0 case, where similar features have been used to implement raw mt in their documentation web site. our focus was on usability as the criteria for publication, and the philosophy behind the idea was to offer this mt content as a service to the customer. the plan was to use the current technology we already used for full post-editing: a generic mt provider and the tms system in place. we remained open to exploring bespoke mt engines and other tms, depending on the results. this solution would only be applied to technical documentation of products where the user interface is not localized, and therefore their documentation would not have been a candidate for human/fully post-edited translation in the first place. ## evaluation phase the mt engine of choice for the persistent mt implementation was amazon translate, as we had been using it successfully for the full post-edit workflow. we found that overall quality in terms of fluency and adequacy was similar between the nmt pro-vided by google and amazon, but the latter offered a custom terminology feature which was crucial to solve the issue of over-translation of product names. this issue was severe in doc sets related to products where generic nouns are used as part of the product name (for example: "app layering", "workspace", "endpoint management"). in order to collect data on the current quality gap between the raw output of generic nmt and our human translations, we started measuring edit effort 3 in the full post-edit workflow. the data thus obtained corroborated our human assessment of the quality that could be expected from the amazon generic mt engine on the tier-1 languages (see figure 1 ). edit effort was chosen over bleu scores, as we understand bleu tends to underestimate the quality of nmt @xcite . we also ran quality assessment checks on the mt output using a popular qa tool, xbench foot_1 , in order to scrutinize the raw mt output and identify quality shortcomings, error patterns, their estimated frequency, and their severity. in addition, we ran a human evaluation test on 55 articles (topics) of citrix documentation, where in-house linguists, contractors, and native speakers of the target languages were asked to rate the quality of the mt output with a criteria of usability, using scores from 1 to 5. the scores were to be given at topic-level, rather than segment-level. the idea was to assess whether the translated content would be enough for the user to complete the task described in a particular topic. evaluators were asked to penalise output that would confuse the user to the point of being unable to follow or understand the text. the averaged result across all 55 topics was 4 for the european languages, and 2 for the asian languages. however, there were some discrepancies in the ratings, consistent with the subjectivity of human evaluation based on usability criteria. the persistent mt workflow is very similar to the default full post-edit workflows: source files are taken from bitbucket repositories in markdown format and uploaded to the tms, where the translation memory is leveraged before processing new content with mt. the target files are then downloaded from the tms and uploaded to bitbucket for publication in docs.citrix.com. unlike the default human post-editing workflow, only in-context and 100% matches are leveraged from the tm. all fuzzy matches (99% and lower) are sent to mt for processing, together with new words. in order to maximize the usability of the machine-translated content, and to manage customer expectations, the documentation website features the following: these mt docs usage data is collected via google analytics events. since the addition of the customer feedback feature is fairly recent, we are unable to report significant insights at this time. ## conclusion we have achieved localization coverage of 100% of citrix product documentation using persistent mt for spanish, german, french, japanese, and simplified chinese. however, sustaining the update cadence of the english documentation proves challenging without an automation platform, and without a solid quality estimation process in place for each mt output. the same factors make scalability into other languages difficult. besides, we are not truly ready to move to a raw mt without review workflow. a process of lpe is still considered necessary, even for the languages where linguistic quality, fluency, and adequacy of the raw mt is very good. this lpe process targets mainly inline tags, over-translations, and blacklisted terms. we are aware that other successful implementations of raw mt in docs, like microsoft's, rely on a tighter control and customisation capabilities of their mt engines, and this is a crucial difference between their case and ours. in this respect, we are currently testing the customisation capabilities of google automl and microsoft custom translator, to train english-japanese and english-chinese engines. in addition, due to the constraints of the source format (markdown), we have encountered further limitations in the quality of the raw output. the markdown errors in the output affect formatting and rendering, and some of them cannot be fixed with automated processes. they require human intervention to pass the validators in place, before publication in the docs web site. quality estimation remains an important challenge, as we observe a lack of readily available tools that we can integrate into mt workflows in a production environment. as a workaround, we use current qa tools in the market to perform estimation tasks, but these tools are best used in traditional localization workflows where the quality checks are aimed at helping human intervention (pe), rather than bypassing it. a tms-agnostic, reliable and customisable qe tool that we could plug in our current workflows would significantly reduce the burden on our translation project managers and post-editors, and accelerate the end-to-end workflow by allowing the implementation of an automated quality gate. also, we still depend on a third-party tms to process the mt content. this dependency adds a certain degree of uncertainty to our processes and restricts the mt engines we can test and use for optimal output. we believe an in-house custom tool to support the entire persistent mt pipeline would be beneficial and this is part of the automation platform we are currently working on.