Translation Analytics for Freelancers: I . Introduction, Data Preparation, Baseline Evaluations (2025)

Abstract: This is the first in a series of papers exploring the rapidly expanding new opportunities arising from recent progress in language technologies for individual translators and language service providers with modest resources. The advent of advanced neural machine translation systems, large language models, and their integration into workflows via computer-assisted translation tools and translation management systems have reshaped the translation landscape. These advancements enable not only translation but also quality evaluation, error spotting, glossary generation, and adaptation to domain-specific needs, creating new technical opportunities for freelancers. In this series, we aim to empower translators with actionable methods to harness these advancements. Our approach emphasizes Translation Analytics, a suite of evaluation techniques traditionally reserved for large-scale industry applications but now becoming increasingly available for smaller-scale users. This first paper introduces a practical framework for adapting automatic evaluation metrics — such as BLEU, chrF, TER, and COMET — to freelancers’ needs. We illustrate the potential of these metrics using a trilingual corpus derived from a real-world project in the medical domain and provide statistical analysis correlating human evaluations with automatic scores. Our findings emphasize the importance of proactive engagement with emerging technologies to not only adapt but thrive in the evolving professional environment.

## introduction this is the first in a series of papers exploring the rapidly expanding new opportunities arising from recent progress in language technologies for individual translators and language service providers (lsps) with modest resources. ## the christopher & dana reeve foundation trilingual corpus translation analytics methods with the resources from a large translation-editingproofreading project completed in summer 2024 for the christopher & dana reeve foundation. foot_1 specifically, the foundation's paralysis resource guide is "a free comprehensive 392-page book designed to empower individuals living with and impacted by paralysis to lead healthy and fulfilling lives." a shorter (80k words) international edition of the guide was recently translated into several languages. the guide (referred to below as 'prg') is a coherent structured document divided into chapters and sections, complete with a descriptive glossary of about 200 technical terms. the translation project (en-ru and en-ja) came in the form of idml (indesign markup language) files for separate chapters. the pdf layouts of the en, ru, and ja versions of prg are included in our corpus for reference. as the first step in data preparation, we took the versions of our tms which preserve the order of source sentences in the original full document. we removed idml and other tags from the tms, discarded repetitions, and produced a spreadsheet that combined the source text (en) and our reference translations (ru, ja). next, we performed additional cleanup operations to remove: to make the translation task more challenging for mt engines and llms, we also decided to remove segments shorter than 6 source words from our set and generated "short" versions of the data (1-10 en-ru-ja short.xlsx, etc.). the source sentence length (len) is calculated in column f. an additional minor reduction was necessitated by the limitations mateo imposes on the input file size (≤ 1mb) for evaluation (section 4 below). to preserve the natural order of the segments, we met this requirement by removing the last two parts of prg ("glossary" and "back cover"), which brought the segment count down to 3555 (1-8 en-ru-ja short.xlsx). the resulting excel document was used to prepare tri-and monolingual unicode text files for our experiments. the materials referenced above comprise the christopher & dana reeve foundation trilingual corpus (alternatively, the reeve foundation trilingual corpus, rftc), complete with the pdf layouts. additional corpus details can be found in appendix b. with the client's permission, we make the corpus described here available for noncommercial/academic use. ## translation outputs in this section we describe how we obtained mt and llm translation outputs for our corpus. ## automatic quality evaluation in this section we report and discuss the automatic metric scores for the entire corpus. as already noted in section 2 above, we had to reduce the length of our corpus by about 9% to 3555 segments to meet the file size requirements of mateo @xcite , the tool we utilized to calculate the bleu, chrf2, ter, and comet scores for our outputs. we provide additional details in appendix d. the evaluation scores for 1-8 en are represented in since the distinction between the linguistic concepts of character and word is blurred in japanese, questions may be raised about the separate significance of chrf for translation directions involving this language. we do not have a considered view on this. but we calculated pairwise pearson correlation values for bleu-chrf2, bleu-ter, and bleu-comet between the scores for our six systems for both language pairs ( in our experiments, performed on a 13th gen intel(r) core(tm) i9-13900kf 3.00 ghz 64.0 gb pc, mateo took roughly 20 minutes to compute the four scores for a single output against a reference; for ja it took slightly longer than for ru. of these 20 minutes, roughly 16 minutes go into computing the comet scores, 2 minutes into ter, and 2 minutes into bootstrap resampling at the very end. the calculation of blue and chrf2 is very fast. in light of the above-noted considerations, the time spent on computing the comet scores is the time well spent. users should be aware of this. ## what sample size is needed for reliable automatic quality evaluation? another important question that may arise for freelancers inclined to use automatic evaluation of mt/llm outputs in choosing the best system for a new project is the minimal size of a sample required to make a reliable decision. a freelancer may have a good tm from a previous project in the same domain or for the same client that could be used for reference. alternatively, a freelancer may complete a representative part of a new project and decide to add the best-performing mt and llmbased system to their workflow going forward. one can imagine similar scenarios. such deliberations should, of course, take into account typological differences between target languages which may affect the automatic scores for string-and neuralbased metrics differently. in all cases of this sort, the size of the sample to be used for mt/llm translation quality evaluation must be statistically significant. what is the minimal size that meets this requirement? to approach this question empirically we generated additional sets of automatic mt/llm evaluation scores for the outputs from three distinct parts of our corpus, 229 en (229 segments, identical to 5 en short), 1143 en (1143 segments, identical to 3 en short), and 2183 en (2183 segments comprising the rest of 1-8 en short) to see how well they correlate with each other. tables 11 and 12 in appendix e feature the four sets of scores, including those for 1-8 en short (3555 = 228 + 1143 + 2183 segments). the lack of overlap among 229 en, 1143 en, and 2183 en (cumulatively comprising the entire 1-8 en short document), which is evidenced in our memoq analysis ( we observe that the correlations are very strong in all cases, across all the metrics. we are thus led to conclude that computing automatic scores for a small part of our document (229/3555 = 6.4%) would give us a good sense of the relative performance of several mt/llm systems. however, this approach has its limitations. see appendix f where we also provide additional details regarding the use of statistical methods for freelancers and discuss the prospects for future work. ## manual evaluation of select translation outputs we selected 180 and 360 mt-and llm-translated sentences from the outputs for 5 en short and 3 en short respectively for each of our language pairs (i.e. 1080 segments overall) to perform manual evaluation of their quality with a simple linear scale in order to estimate whether the outputs' sentence-level comet scores correlate with our "human grades" for them. below we outline our selection process, the evaluation scale, and the results. ## limitations of our study and future plans translation directions. we had an opportunity to experiment with two interestingly different language pairs because we ourselves produced the translations for them and the client gave us permission to use them. we hope that other language directions and document families will be added to our corpus in the future. automatic metrics. we limited our choice of them to bleu, chrf2, ter, and comet, to maximize efficiency and ease of use. more advanced users should consider other metrics and consult the current best practices @xcite . correlation experiments with sample sizes reported in section 5 need to be complemented with power analysis to determine the minimal size of the statistically significant sample. ideally, future experiments should also include other contrasting pairs-from different domains, registers etc. human evaluation requires independent raters and a uniform blinding and randomization protocol. while extensive, our reported results must be taken with a grain of salt. we do believe they serve as a proof of concept. in future experiments with our corpus (rftc) we want to explore the potential of various dedicated systems and llms for (i) extracting a bilingual glossary from a set of parallel sentences, and (ii) using a glossary thus obtained to improve the quality of translation in the context of the freelancer's workflow. other ideas are briefly described in section 1.1 above. we may pursue some of them and invite fellow translators and other interested parties to join us in this effort. ## conclusions this study demonstrates the potential of translation analytics to help freelance translators and smaller language service providers (lsps) thrive in a rapidly evolving industry. by adapting evaluation metrics such as bleu, chrf, ter, and comet to individual workflows, we provide methods for assessing mt and llm outputs with rigor and precision. the findings underscore several critical insights: utility of automatic evaluation metrics. automatic metrics, particularly comet, consistently align with human assessments, reinforcing their value as robust tools for translation quality evaluation. translators can confidently leverage these metrics to make informed decisions about incorporating mt and llm systems into their workflows.