MTU ncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning O pen AI LLM s (2024)

Abstract: Translation Quality Evaluation (TQE) is an essential step of the modern translation production process. TQE is critical in assessing both machine translation (MT) and human translation (HT) quality without reference translations. The ability to evaluate or even simply estimate the quality of translation automatically may open significant efficiency gains through process optimisation.This work examines whether the state-of-the-art large language models (LLMs) can be used for this uncertainty estimation of MT output quality. We take OpenAI models as an example technology and approach TQE as a binary classification task.On eight language pairs including English to Italian, German, French, Japanese, Dutch, Portuguese, Turkish, and Chinese, our experimental results show that fine-tuned can demonstrate good performance on translation quality prediction tasks, i.e. whether the translation needs to be edited .Another finding is that simply increasing the sizes of LLMs does not lead to apparent better performances on this task by comparing the performance of three different versions of OpenAI models: , , and with 13B, 175B, and 175B parameters, respectively.

## introduction most modern translation projects include postediting (pe) of machine-translation (mt) output @xcite . instead of translating from scratch, the mt+pe process increases productivity and allows to speed up global content delivery @xcite . however, in regulated industries and many other scenarios raw mt output is not suitable for final publication due to the inevitable errors caused by the inherently stochastic nature of neural mt (nmt) @xcite @xcite . hallucinations, incorrect terminology, factual and accuracy errors, small and large, as well as many other types of mistakes are inevitable to varying degrees of extent, and therefore for premium quality publication human revision is required. mt output serves as input for a professional human translator, who reviews and revises the mt proposals to eliminate factual errors and ensure that the quality of translated material conforms to the customer specifications. at the same time even with those languages that are not handled well by mt, there is a significant portion of segments that are not changed after human review. this portion varies from 10% to 70% in some cases foot_0 , and the question arises, "is it possible to use machine learning (ml) methods to mark these segments and save time for human reviser and make them focus on those segments that need attention instead"? in other words, is it possible to capture editing distance patterns from data of prior editing of this material, which already has been made? this could further speed up the translation process and decrease the costs while preserving the premium quality of the translated product. this problem is also closely related to the traditional mt quality estimation (qe) shared task that has been held with the workshop of mt (wmt) series since 2012 @xcite @xcite @xcite , where both token-level and segment-level qe were carried out. from practical application and industrial usage, we formulate the problem into a single classification task, i.e. we are trying to solve a classification task to answer if the translated segment (sentence) needs to be edited, or not. with the development of current large language models (llms), we choose openai models as state-of-the-art llms to examine their capabilities for this task. in this work, our first experimental investigation is on "predictive data analytics with ai: assessing the need for post-editing of mt output by fine-tuning openai llms". we also follow up with an experiment that explores "if the size of sample or llm matters in such a task" by experimenting with three openai models: curie, davinci, and gpt3.5, with parameter sizes varying from 13b to 175b. the rest of this paper is designed as below. section 2 introduces related work to ours including mt-qe-related shared task and challenge events, section 3 presents our methodology design and pilot study using two language pairs, section 4 extends the experimental investigation with six more language pairs, section 5 discusses experiment on english-japanese news content with the increasing sizes of training and testing corpus and explores two more openai llms with varying model sizes, and section 6 concludes this paper with future work and research perspectives. ## related work the quality evaluation (qe) of mt output has always been a critical topic for mt development due to its critical role in assessing quality in the process of training. in many cases, evaluation has to be done without seeing the reference translations. in many practical situations, reference translations are not available or even impossible to acquire, i.e. it is not practical to "manufacture" them for evaluation. the earliest qe shared task with the annual wmt conference started in 2012 when word level qe was introduced by @xcite to estimate if the translated tokens need to be edited or not, such as deletion, substitution, or keeping it as it is. in the later development of qe, a sentence-level task was introduced to predict the overall segment translation scores, which are to be correlated with human judgement scores, such as using direct assessment @xcite . in wmt-2022, a new task on binary sentencelevel classification was also introduced to predict if a translated output has critical errors to be fixed on english-german and portuguses-english language pairs @xcite . the recent methods used for such qe tasks included prompt-based learning using xlm-r by ku x upstage (korea university, korea & upstage) from @xcite , direct assessment and mqm features integration into fine-tuning on xlm-r and infoxlm @xcite by the alibaba team @xcite , and incorporating a word-level sentence tagger and explanation extractor on top of the comet framework by @xcite , in addition to historical statistical methods such as support vector machine (svm), naive bayes classifier (nb), and conditional random fields (crfs) by @xcite . however, to the best of our knowledge, this work is the first to investigate the openai llms with varying sizes on such mt error prediction tasks with positive outcomes. ## methodology and experiments as shown in the system diagram in figure 1 , we first collect the historical post-editing data from our past projects on eight languages of enterprise resource planning (erp) content translation on @xmath0, french, italian, japanese, dutch, portuguese, turkish, and chinese (de, fr, it, ja, nl, pt, tr, zh). this project was completed by using an mt engine to automatically translate the source into the eight languages, followed by post-editing by professional linguists. two examples of mt and pe in english-italian and english-german languages as pilot experiments are shown in figure 2 and 3. regarding mt system selection, since the content was from the erp domain, we used the sap sth as our mt en-usa en-it mt mt pe have affected personnel in the immediate area of the hazardous work been notified of the work to take place? il personale interessato nell'area immediatamente interessata dal lavoro pericoloso è stato informato dei lavori da svolgere? il personale interessato dai lavori nell'area vicina all'intervento pericoloso è stato informato sui lavori da svolgere? have all energy sources been restored per restart procedures and isolations removed? sono state ripristinate tutte le fonti energetiche per ogni ripresa e rimozione degli isolamenti? sono state ripristinate tutte le sorgenti di energia secondo le procedure di riavvio e sono stati rimossi gli isolamenti? have all equipment and safeguards been completely restored tutte le attrezzature e le misure di sicurezza sono state completamente ripristinate? attrezzature e salvaguardie interamente ripristinate? have all equipment and safeguards been completely restored? sono state completamente ripristinate tutte le attrezzature e le salvaguardie? attrezzature e salvaguardie interamente ripristinate? have all safeguards been completely restored? tutte le salvaguardie sono state completamente ripristinate? salvaguardie interamente ripristinate? have conditions changed? le condizioni sono state modificate? sono cambiate le condizioni? hazardous energy control (hec) procedural format formato procedurale per controllo energetico pericoloso (hec) format procedurale per controllo energetico pericoloso (cep) hazardous energy control permit permesso di controllo dell'energia pericolosa permesso di controllo energia pericolosa (hec) hazardous energy control permit completed as planned permesso di controllo dell'energia pericoloso completato come previsto permesso cep (hec) completato come pianificato hazardous energy control permit template modello di permesso di controllo dell'energia pericolosa modello di permesso di controllo energia pericolosa (hec) hazardous operations within 35ft (11m) of work area are shut down le operazioni pericolose all'interno di 35 ft (11 m) dell'area di lavoro vengono chiuse le operazioni pericolose entro 35 ft (11 m) dell'area di lavoro vengono chiuse figure 2: en-it examples on mt and post-editing en-usa en-de mt mt pe chemical/liquid protective boots chemische/flüssige schutzboote schutzstiefel für chemikalien und flüssigkeiten child location unterlokation untergeordneter standort child permit kindgenehmigung untergeordnete genehmigung child state untergeordneter bundesstaat untergeordneter zustand child workflow untergeordneter workflow untergeordneter arbeitsablauf child workflow state untergeordneter workflow-status untergeordneter arbeitsablaufstatus chilled water -specify pressure kühlwasser -geben sie den druck an. kühlwasser -druck angeben choose workflows to import zu importierende workflows auswählen zu importierende arbeitsabläufe auswählen the file size exceeds the limit allowed die dateigröße überschreitet das zulässige limit die dateigröße überschreitet die zulässige grenze qualified person returned service to full operation and notified all affected workers? qualifizierte person hat den dienst in betrieb genommen und alle betroffenen mitarbeiter benachrichtigt? hat die qualifizierte person den betrieb wieder vollständig hergestellt und alle betroffenen mitarbeiter benachrichtigt? qualified persons are first aid/cpr certified qualifizierte personen sind erste-hilfe-zertifizierung/cprzertifiziert qualifizierte personen verfügen über erste-hilfe-zertifizierung/cpr-zertifizierung record results every 30 minutes for length of work. erfassen sie die ergebnisse alle 30 minuten für die dauer der arbeit. ergebnisse alle 30 minuten für die dauer der arbeit erfassen. record worst case reading of any meter at any level in the space. erfassen sie den worst case reading eines beliebigen zählers auf einer beliebigen ebene des bereichs. aufzeichnung des worst-case-messwerts eines beliebigen zählers auf einer beliebigen ebene im raum. the symbol displayed at the top left of the application, used as the home button. das oben links in der anwendung angezeigte symbol, das als home-drucktaste verwendet wird. das oben links in der anwendung angezeigte symbol, das als startseiten-schaltfläche verwendet wird. engine. 2 with this data from a real-world translation project, we used api to fine-tune the openai curie model for our classification task. the input is the triple set (english source, mt outputs, postedited "gold standard") we prepared in phase 1. the goal of this step is to optimise the weights of the model parameters for our classification task. the custom fine-tuned model produced as a result of llmb2pen (llm for binary prediction of post-editing need) method is created in our private space on the openai account. we did not apply "prompt engineering" for this task by doing zero-shot, one-shot, or few-shot training; we did a full-scale fine-tuning of openai llms via api. it is important to note that we did not simply train the llm for edit distance either; instead, the model was trained to learn whether the strings were edited or not taking into account the full content of the string and the entire context of the training data. one of the reasons that we did not use prompting is that "prompt engineering" of chatgpt-3 is limited by 3,000 tokens, and 2 https://www.sap.com/ sap is an enterprise resource planning, automation and business software company. with chatgpt-4 the context has been increased to 25,000 tokens, but still very significant limitation remains. openai documentation states that 100 @xmath1, meaning that the average sentence is 20 tokens, therefore 3000 tokens is only 150 sentences, or 75 translation units of bilingual text, or 50 segment triples of source, target and reference. the context of 25,000 sentences is only about 150 segment triples. also, fine-tuning is a deeper process of adjusting the model's weights, and not just an incontext learning. that's why we chose fine-tuning method, which is not constrained by such limitations. for our classification experiment we took about 4000 lines of bilingual data in triples of source, target, and reference, and split it into train (large) and test (smaller) sets with a ratio of 9:1. there were no specific selection criteria for the data because we took the entire project dataset after project completion. (please, note that since we used the entire data from the actual project, and split the data set as 9:1, the sizes of test sets are not round and slightly different for different languages.) we also combined source sentences in groups of length, so that the test data set has the same distribution of sentences by their length as the training dataset. since the average sentence size is about 17 words, the training dataset contained about 35000 words of source data, 35000 words of mt output, and 35000 words of post-edited human reference. it is also important to note what the model learns in this case -in such an experiment it learns not to translate, but to spot mt translation errors that were made by the specific mt engine in a specific language pair on particular content. ## extended experiments on six more language pairs we hereby also present extended experimental results using six more language pairs obtained with lmb2pen method for translation editing distance prediction. these language pairs include englishto-french, japanese, dutch, portuguese, turkish, and chinese (@xmath2, whose results are listed in figure 6 , 7, 8, 9, 10, and 11 respectively. from the results presented in the figures, in general, the ratio of correct prediction (tp+tn) is much higher than the one from mis-prediction (fn+fp) across all these language pairs, as for english-italian and english-german in the pilot studies. on one hand, the following language pairs have more true positive than true negative predicted segments than for english-german/italian: english-japanese, english-portuguese, and english-chinese. on the other hand, the rest of the language pairs have more tn than tp: english-french, and english-dutch, except for english-turkish which has a comparable number of segments between tp (347) and tn (353) labels. this finding also indicates that such language pairs with a high number of tn labels are still much more challenging for mt system development to produce more correct outputs, i.e., english to french, dutch, and turkish. earlier research findings from @xcite on tqe conclude that 200+ segments can be enough amount of data to reflect the mt system quality. 5 different llms on en-ja news domain in the subsequent experiment on data, we used different news items translation corpus from different projects, translated from english to japanese. figure 8: en-nl confusion matrix of llmb2pen, curie model: clockwise from top-left corner (tn, fp, tp, fn) 5.1 using openai gpt3.5turbo in this experiment, we have repeated experiments of fine-tuning the openai gpt3.5turbo model on datasets of different sizes: 2000 pairs, 4000 pairs, and 6000 pairs. figure 12 shows the confusion matrix for the training set of 6000 bilingual en-ja translation pairs in the news domain. we ran several experiments with varying training set sizes, with results shown in figure 13 . these results are interesting because although false positive prediction does not improve with the increase of training set, in the context of the figure 10: en-tr confusion matrix of llmb2pen, curie model: clockwise from top-left corner (tn, fp, tp, fn) need for post-editing the false negative category is much more interesting, because we are interested in better prediction of those segments which do not require post-editing. and, as we see from the experimental data, the prediction of fn improves from almost 20% to 12%-15% with the increase of training set from 2000 bilingual segments to 6000 bilingual segments. we, therefore, can recommend the training set in that range, since larger sizes of training set will be more expensive and will take significant time for models with the size of gpt3.5turbo. figure 12: en-ja news items confusion matrix of llmb2pen, gpt3.5turbo model: clockwise from top-left corner (tp, fn, tn, fp) figure 13: en-ja news items predictions with fine-tuning completed on different training dataset sizes, gpt3.5turbo model 5.2 comparison of performance on different openai models figure 14 shows the comparison of these three llms regarding their confusion matrix and parameter sets. surprisingly, their performances on predicting mt errors are very close, i.e. the larger-sized davinci model and extra-large sized gpt3.5turbo did not demonstrate much improvement on model classification accuracy. their correct labels (tp+tn) are (694, 699, 706) respectively out of 842 all labels, which results in the accuracy ratios 82.42%, 83.02%, and 83.85%. in comparison to the much smaller curie model with 12 layers of transformer and 768 hidden units, the xllm gpt3.5turbo only achieved 1.43 points (83.85%-82.42%) increase of accuracy score despite using 175 layers of transformer and 4096 hidden units. the explanation for this may probably be found because the fine-tuning loss on this classification task drops down very quickly. figure 15 shows the fine-tuning loss on the gpt3.5turbo model. as can be seen from this graph, only 100 steps are sufficient to bring the loss to almost zero, and then all other steps contribute very little to the classification quality improvement. as we can see, there is no need to use larger models since results hardly improve as compared with curie model. ## conclusions and future work in this work, to investigate the llm's capability of predicting mt output errors, we fine-tuned gpt models via openai api. we formulated the task as a classification challenge using prepared historical post-editing data on english-italian and english-german for pilot studies. the experimental output using fine-tuned llmb2pen demonstrated promising results. we also analysed the possible solutions for addressing the error rates, i.e. whether prediction errors can be ignored and published without the review, or letting them be reviewed by the linguists at a lower rate, and how much saving can be achieved for the client who uses this process, in comparison to 100% postediting without using llmb2pen method. in the extended experiments, we added six more language pairs including english-to-french, japanese, dutch, portuguese, turkish, and chinese, in total resulting in eight, and summarised our findings by classifying the language pairs. we also compared gpt models from different sizes and the experimental results surprisingly show that the larger llms (davinci and gpt3.5turbo) do not improve the accuracy performance of much smaller curie model with apparent margins but with much more cost. in the future, we are going to work on response rate and training times to see whether the model can continue learning as being fed with more consecutive chunks of data for the same languages, to implement an ongoing learning of prediction. in addition, we plan to carry out the llmb2pen fine-tuning on other language pairs for which we have historical data. we intend to explore to what extent the model is capable of absorbing data for several languages, i.e. one fine-tuned multilingual model serving several language pairs. to further extend this project, it will also be interesting to explore and check whether the llmb2pen method can help to identify humanintroduced errors or translationese.