Insights from Gathering MT Productivity Metrics at Scale (2020)

Abstract: In this paper, we describe Dell EMC’s framework to automatically collect MT-related productivity metrics from a large translation supply chain over an extended period of time, the characteristics and volume of the gathered data, and the insights from analyzing the data to guide our MT strategy. Aligning tools, processes and people required decisions, concessions and contributions from Dell management, technology providers, tool implementors, LSPs and linguists to harvest data at scale over 2+ years while Dell EMC migrated from customized SMT to generic NMT and then customized NMT systems. For content in two quality tiers, we ranked language pairs by productivity, graphed trendlines, compared the time needed to edit machine translations versus fuzzy matches, studied the time spent on segments with no post-edits, and going by the post-edit density, re-viewed segment distribution on a post-edit scale of 1 to 10 and any correlation between the extent of edits and segment length.

## the metrics we will discuss the following mt metrics: productivity and post-edits. we chose to measure both metrics at the first linguistic step, translation, although our linguists may have made further changes downstream at the editing, proofing, client review and feedback implementation steps. the taus dqf dashboard expresses productivity in words post-edited per hour. this number is calculated from the number of words in segments and the milliseconds these segments are active in the cat tool for editing. as for post-edits (ped), the taus dqf dashboard distinguishes between post-edit density (pede) and post-edit distance (pedi). both are calculated with the @xcite . pede expresses changes across the entire sample in percentages, in the average number of characters changed per 100 characters. at the segment level, pedi expresses the changes in absolute numbers, i.e., in characters changed per segment; we will call it apedi. normalized to the length of the segment, the pedi expresses changes on a scale of 0 to 10; we will call it npedi. as an example: 10 characters changed in a 20 character-long segment will result in an apedi of 10 and an npedi of 5. ## caveats as we analyzed the accumulated data for this paper, we found that post-edits made in a single cat tool session at the translation step correctly capture the full extent of post edits, even if the linguist revisits a segment multiple times in the same session. if the linguist edited the same segment in separate cat tool sessions, only the edits made in the last session are captured. because of this, we are underreporting the postedit distance for an unknown number of segments. another caveat is that we decided not to track productivity for human translations (ht). we originally expected that for a given quality tier, we would either mtpe or ht all jobs. and comparing productivity between mtpe and ht jobs across quality tiers would not result in a fair comparison. later on, we found that our pms did apply ht workflows selectively to mtpe quality tiers. had we adjusted our setup, we would now have data to benchmark mtpe against ht productivity. ## ranking by productivity the most basic exercise is to rank our major language pairs by productivity. these hourly word numbers below result from dividing cumulative mt words by cumulative post-editing time between august 2017 and december 2019. the hourly throughput does not account for elapsed time, such as research while segments in the cat tool are inactive and the time tracker is not running. therefore these productivity numbers are somewhat theoretical and do not mean that our pt-br linguists post-edit 2147 x 8 = 17,176 words per day. but they surely suggest that the historic translation output of 2000 words per day is outdated and the actual productivity is significantly higher due to translation technology. while the language ranking is roughly as expected, there are surprises: zh-cn ranks relatively high compared to ja-jp and ko-kr; and es-mx (latin american spanish) ranks low compared to fr-fr, it-it and pt-br. productivity clearly varies by quality tiers, being higher for good enough than high quality content. this may be due to varying levels of linguistic complexity and expectations, or the simple fact that good enough jobs are bigger than high quality jobs, think of product documentation vs. marketing material. and the more volume in a given job, the easier it is for linguists to pick up speed. within the overall correlation between bleu and productivity, the top-ranking bleu score for es-mx failed to predict average post-editing productivity; similarly, the better-than-expected bleu score for ja-jp failed to predict low postediting productivity. it has been observed before that productivity and bleu scores do not correlate necessarily @xcite . there is a caveat to our correlation of productivity and bleu since productivity was calculated using data collected since august 2017, while the bleu score is for customized nmt systems deployed only since march 2019. productivity ranking of mt systems provides helpful context when triaging linguist feedback on mt output quality, especially when combining the ranking with npedi distributions for a particular language or across languages for a given job. please zoom in. also, knowing your numbers allows you to place your mt technology relative to potential alternatives in the market. ## productivity and pede trendlines we wanted to understand if our mt output is getting better, worse or is stable over time. for this, we compiled productivity and pede trends by transcribing monthly averages from the taus dqf dashboard into msft excel and applying linear trendlines. we then noted start and end values of these trendlines along with the number of words machine translated for statistical context. the higher the productivity, the better; the lower the ped, the better. in the above table a positive percentage for productivity (prod) means that the hourly edited words increased by x%, while a negative value indicates productivity loss. conversely, a negative pede value means that average number of edits fell over time (good), while a positive value means an increase in edits (bad). for good enough content, pede fell for our top nine languages. likewise, productivity increased for these languages for high quality content. productivity for good enough content and pede for high quality content, however, have outliers. ideally, falling pede should result in rising productivity. while de-de exemplifies hopedfor results, there are obvious exceptions. korean, for example, at the good enough quality tier has pede falling by 48%, while productivity is dropping by 7%, when it should be rising in correlation. the table below shows how the numbers above came about. the following graph compares productivity trendline and pede for en-us to dutch, a language pair for which we never customized nmt systems. ## cut-off between mt and tm in the following exercise, we wanted to find out if post-editing machine translations is faster than editing fuzzy matches. depending on the language pair and nmt system, customized vs. generic, machine translated segments require fewer post-edits than fuzzy matches @xcite . assuming that fewer post-edits mean shorter post-editing times, i.e., higher productivity, we should raise the mt-to-tm cutoff from 75% to x. we looked at jobs machine translated against our customized nmt system from en-us to de-de between march 2019 and 2020. as expected, post-editing times diminish with increasing match rates. but it takes significantly longer to post-edit mt segments than 75%-84% fuzzies: 44 vs. 25 seconds. increasing the mt-to-tm cut-off to 85% would drastically reduce productivity. sorting segments by post-edit time, we noticed outliers that were active much longer than typically necessary for editing. the biggest outlier was active in the cat tool for 16 minutes. when only considering segments active for 60 seconds which means 81% of all mt segments -the editing time gap shrinks from 76% to 25%. for the en-us-to-fr-fr language pair, the post-editing time gap between fuzzy matches and mt shrinks to 9% when only considering segments active for 60 seconds. this convergence likely applies to other well-performing mt language pairs as well: english to brazilian portuguese, chinese, french, italian and spanish. but as long as editing time for mt segments doesn't fall below editing time for fuzzy matches, raising the mt-to-tm cut-off would be counterproductive. let's see if the ped tells a different story. for en-us to fr-fr, the average ped for mt segments is 17 characters vs. 21 characters for 75%-84% fuzzy matches. while mt segments require 19% fewer post-edits, it takes 9% longer to edit them. it has been noted before, that post-edits and post-editing time, i.e., technical and temporal efforts do not necessarily correlate @xcite . while we couldn't demonstrate that our mt segments can be edited faster than our fuzzies, we did gain a couple of useful insights. for one, we need to optionally exclude outliers from our data to produce a richer picture of our mt productivity. about 10% of mt segments inflate both average post-editing time and ped noticeably. the tables above contain outliers, segments with inline tags relatively easy to handle by linguists in the cat tool, but with large character counts to the levenshtein algorithm. in one sample, a segment with 7 words and 6 tags resulted in an apedi of 1015 characters. going by the standard ratio of 1:5 for words to characters, the calculated apedi vastly overstates the human effort of placing a few tags and minor textual changes. for two, 99% fuzzy matches deserve special consideration in slas, assuming they constitute a good portion of overall word count. in our 12month sample, they account for 9% of total words, and require a fraction of editing time compared to other match bands. they ought to be broken out for dedicated costing. ## time spent on 0 npedi segments in this section we discuss the time linguists spent on segments that required no post-editing. ideally, cat tools should flag these segments to linguists so that they can skip them. ## correlating npedi and segment length we approached the exercise of correlating npedi and segment length with the assumption that mt systems translate segments of a certain length best, segments that are not too short and not too long. similar to linguists, mt systems may struggle with short segments for lack of context and with long segments because of complexity. we tried to confirm this assumption with two different methods on segments machine translated with our customized nmt systems between march 2019 and february 2020. in the first exercise, we simply expanded the npedi distribution table, by adding the total source language word count for each npedi mark and dividing it by the number of segments. while short segments appear at both ends of the 1 -10 npedi scale, 75% of them are in the 0 npedi bracket. this does mean that shorter strings machine translate more successfully. in the next method, we grouped de-de segments by word length and, calculated the average apedi and npedi for each length. with the exception of segments 1, 2, 5 and 40 words long, the npedi hovers around 3 for segments of any length, even for the longer ones. we assume that the average npedi for longer segments doesn't increase noticeably because linguists may revisit these long and complicated segments in separate cat sessions and therefore only record a portion of the actual post-edits. the lighter (orange) line in the graph below illustrates the lower npedi for the very short segments and the otherwise relatively stable npedi: the first exercise clearly indicates that it is the majority of shorter segments that machine translate better than longer strings. a minority of shorter strings fails to translate well, probably for lack of context. the second exercise is inconclusive, but highlights one of the imperfections in our productivity tracking setup. ## future work users will benefit from several enhancements to globallink to more accurately record postediting efforts in this particular translation management system. above all, the setup needs to capture all edits, whether performed in one or multiple cat tool sessions. mt segments should only be included in the metrics if linguists signed off on them. to track editing time comprehensively, the cat tool should optionally require activation also for segments that do not need post-edits. batch changes that do not trigger the time tracker should be accounted for somehow. tracking editing effort for ht segments is essential to establish a baseline. the comprehensive audit trail capabilities of globallink allowed us to pinpoint these opportunities for improvement. we hope that taus will use our findings to develop certification test plans for tools integrators to ensure that productivity metrics are consistently calculated across organizations using different cat tools. also, the taus dqf dashboard should allow users to exclude outliers for an alternate productivity view. for full access to our productivity data currently stored in the taus dqf dashboard, dell needs to integrate its bi tools. ## conclusion tracking mt productivity at scale needs to become an integral capability in the translation industry to be available regardless of which tools and services providers we partner with. even though our productivity metrics are after the fact, they are a statistically robust addition to small-scale human evaluations, bleu scores and emerging risk calculation models. together, these mt quality assurance methods help us focus our continuous improvement efforts. our numbers show that we are on the right track: productivity is steadily rising and postedits falling. our challenge will be to turn the many segments requiring few post-edits to ones that require none and to flag these segments in the cat tool so that linguists can skip them. we like to think that the imperfections we discovered in our setup balance each other out as some inflate and some deflate our productivity numbers. we are also reminded that within mtpe jobs, we apply machine translations to new words only, about 10% of total word count. the remaining 90% are leveraged from translation memories. while mt is an important productivity aid, it is not the only one in a linguist's tool chest. overall, 70% of our production translation jobs use mt to pre-translate new words. we will expand mt usage by starting to pre-translate software as well. the biggest expansion of mt usage at dell, however, occurs somewhere else. to operate within a global enterprise, many of our colleagues produce raw mt in self-service mode. and data scientists machine translate vast amounts of data into english for processing by bi engines. this last use case dwarfs all others by volume. closely monitoring machine translation quality in our human-assisted production workflows will benefit dell's two use cases of unedited mt output as well. the following individuals and organizations were instrumental in creating an environment to harvest mt metrics automatically for dell emc: nancy anderson, head of the emc translation team at the time supported the proposal to take translations "online". she negotiated with our lsps the necessary process and tools concessions. keith brazil and his team at translations.com optimized globallink as a collaborative platform for a multi-vendor supply chain. jaap van der meer proposed an integration with the taus dqf dashboard. taus and