cush LEPOR : customising h LEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model L a BSE (2021)

Abstract: Human evaluation has always been expensive while researchers struggle to trust the automatic metrics. To address this, we propose to customise traditional metrics by taking advantages of the pre-trained language models (PLMs) and the limited available human labelled scores. We first re-introduce the hLEPOR metric factors, followed by the Python version we developed (ported) which achieved the automatic tuning of the weighting parameters in hLEPOR metric. Then we present the customised hLEPOR (cushLEPOR) which uses Optuna hyper-parameter optimisation framework to fine-tune hLEPOR weighting parameters towards better agreement to pre-trained language models (using LaBSE) regarding the exact MT language pairs that cushLEPOR is deployed to. We also optimise cushLEPOR towards professional human evaluation data based on MQM and pSQM framework on English-German and Chinese-English language pairs. The experimental investigations show cushLEPOR boosts hLEPOR performances towards better agreements to PLMs like LABSE with much lower cost, and better agreements to human evaluations including MQM and pSQM scores, and yields much better performances than BLEU. Official results show that our submissions win three language pairs including English-German and Chinese-English on News domain via cushLEPOR(LM) and English-Russian on TED domain via hLEPOR. (data available at https://github.com/poethan/cushLEPOR )

## introduction machine translation (mt) is a rapidly developing research field that plays an important role in nlp area. mt started from 1950s as one of the earliest artificial intelligence (ai) research topics and gained a large improvement in the output quality in large resourced language pairs after the introduc-tion of neural mt (nmt) in recent years @xcite @xcite . however, the challenge still remains in achieving human parity of mt output @xcite . thus mt evaluation (mte) continues to play an important role in aiding mt development from the aspects of timely and high quality evaluations, as well as reflecting the translation errors that mt systems can take advantages of for further improvement @xcite . on one hand, human evaluations have long been criticised as expensive and unrepeatable. furthermore, the inter-and intra-agreement levels from human raters may struggle to achieve a consistent and reliable score, unless done in rigour with highly trained and skilled evaluators @xcite . on the other hand, even though researchers have claimed that the automatic evaluation metrics have reached much better performances in the category of system level evaluations of mt outputs, with high correlation to human judgements, the segment level performance is still a large gap from human experts' expectation @xcite @xcite @xcite . in the meantime, many pre-trained language models have been proposed and developed in very recent years and showing big advantages in different nlp tasks, for instance, bert @xcite and its further developed variants @xcite . in this work, we take the advantages of both high performing automatic metric and pre-trained language model, aiming at one step further towards higher quality performing automatic mt evaluation metric from both system level and segment level perspectives. among the evaluation metrics developed recent years, hlepor @xcite is an augmented metric that include many evaluation factors with tunable weights assigned including precision, recall, word order (via position difference factor), and sentence length. it has also been applied by researchers from different nlp fields including natural language generation (nlg) @xcite , natural language understanding (nlu) @xcite , automatic text summarization (ats) @xcite , and searching @xcite , in addition to mt evaluation @xcite . however, original hlepor has disadvantage of the manual tuning of its parameter weights which take a lot of human efforts. we choose hlepor @xcite as our baseline model, and use the very recent language model labse @xcite to achieve automatic tuning of its parameters thus aiming at reducing the evaluation cost and further boosting the performance. this system description paper is based on our earlier work, especially the training models @xcite . the rest of the paper is organised as below: section 2 revisits hlepor metric, its factors, advantages and disadvantages, section 3 introduces our python ported version of hlepor and the further customised hlepor (cushlepor) using language models, section 4 presents our experimental development and evaluation that we carried out on cush-lepor metric using wmt historical data, section 5 reserves space for our submission to this year wmt21 metrics task, and section 6 finishes this paper with discussions of our findings and possible future work. the factor hpr is the harmonic mean of precision and recall values. hp @xmath0 we refer the work @xcite @xcite for detailed factor calculation with examples there. the basic version of hlepor carries out similarity calculation between mt system outputs and reference translations, in the same language setting, based on the word surface level tokens. the hybrid hlepor metric also carries out similarity calculation based on pos sequences from systemoutput and reference text. to do this, pos tagging is needed as the first step, then hlepor(pos) calculation uses the same algorithms used for the word level similarity score hlepor(word). finally, hybrid hlepor is a combination of both word level and pos level score. in this system submission work, with the time limitations, to make an easier to use customised hlepor, we take the basic version of hlepor, i.e. the word level similarity calculation and leave the hybrid hlepor into the future work. the weighting parameters for the three main factors in original hlepor metric, i.e the (w lp , w n p osp enal , w hp r ) set, in addition to the other parameters inside each factor, were tuned by manual work based on development data (see appendix for detailed parameter value sets on each language pair for word-surface level evaluation, en ⇔ cs/fr/de/es/ru). this is very time consuming, tedious, and costly. in this work, we will introduce an automated tuning model for hlepor to customise it regarding deployed language pairs, which we name as cushlepor. ## experimental evaluations the training and development data we used regarding mqm scores and psqm labels is from the recent work by google research team on investigating into human evaluations based on wmt2020 shared task @xcite (data available at https://github.com/ google/wmt-mqm-human-evaluation ). we first focus on english-to-german language (en-de) pair, which includes mqm and psqm labels, acquired from 10 submission of wmt 2020, then take zh-en data-set. we refer to the paper @xcite for detailed mt system names and offering institutions. firstly, a multi-parameter optimisation against labse for en-de language pair gave the following values for cushlepor parameters: this also reflected that labse similarity is indeed a feasible goal for cushlepor optimisation. the correlations of hlepor and cushlepor to labse are shown in fig. 1 and 2 . however, we found out that we were not able to decrease much on rmse (root mean square error) score for cushlepor towards psqm , in comparison to original hlepor, (0.28 vs 0.29)which does indicate that original hlepor empirically shows very good fit for psqm type human evaluation, using the suggested parameter settings for en-de @xcite as bellow. optuna did optimise cushlepor against labse very well, halving the rmse distance between labse and cushlepor as compared to original hlepor, shown in fig. 5 . the performances of tuning on labse and psqm are shown in fig. 6 and 7 respectively. the horizontal axis is the score value (0, 1) and the vertical axis is the sentence number that falls into the corresponding score intervals. figure 4: rmse: bleu vs cushlepor to psqm (lower score is better) figure 5: rmse: hlepor vs cushlepor to labse (lower score is better) from the score distribution visualisation, it reflects the tuning on psqm has a larger covered error types while labse is less sensitive to some errors that human experts would spot out. as shown on these charts, psqm human rating shows much wider "tail" of "low score ratings", while labse rating is much more focused. the reason is that labse similarity model underestimates the severfigure 7: score distribution: tune on psqm ## submission to wmt21 for wmt2021 metrics task, we submitted our cushlepor system scores for @xmath1, both segment-level and systemlevel evaluation. the training and development set we used are exact the ones from last section (section 4). we can not tune our cushlepor model parameters on @xmath2, because the human labelled mqm and psqm scores as validation data that cushlepor requires do not exist from last year wmt20 set. instead, we submitted hlepor metric for @xmath3. we carried out evaluation on all four official data-sets: newstest2021 (traditional task), florestest2021 (sentences translated as part of the wmt news translation task), tedtalks (additional sets of sentences translated by wmt21 translation systems in the ted talks domain), and challengeset (synthetic outputs generated specifically to challenge automatic metrics). ## discussions and future work in this work, we described cushlepor, a customised hlepor metric which can be automatically trained and optimised using both human labelled mqm scores, as well as large scale pretrained language model (lm) labse towards better agreement to human experts level judgements and distilled lm performance respectively, and reducing cost at the meantime, e.g. the manual tuning from hlepor and high computational demand from lms. we also optimised cushlepor towards human translators' evaluation scores, i.e. psqm, which showed much improved performance than bleu and original hlepor (with default parameters). our research is in line with the mt evaluation guideline suggestions from the very recent work @xcite that better evaluation metrics in correlation to human judgement shall be tested and deployed. or human judgements shall be carried out directly wherever possible. we have some findings during the experimental investigation: 1) cushlepor trained on labse can replace labse to carry out similarity calculation task in mt evaluation, which is much more light weighted and low cost from computational power and complexity point of view. 2) we can choose alternative pre-trained language models (lms) in the future to boost performance. 3) this cushlepor optimisation framework proves to be functional, offering high performance towards pretrained lms, much improved agreement of cush-lepor to labse scores in comparison to hle-por (as in figure 1 and 2). 4) optimised cush-lepor achieves better agreement towards professional translator's evaluation (psqm). optuna, the hyper parameter optimisation toolkit we used, can generate different set of cushlepor parameter values in different runs, which could be an consistency issue. however, we believe it optimises the performance of cushlepor towards the highest agreement to the reference scoring (pretrained lms or human evaluations), but not to ensure the same set of parameter values to be generated, so this will not be a problem. we will carry out further analysis on this aspect in the future work. the hybrid version of hlepor @xcite use pos features to function as pseudo synonyms to capture alternative correct translations. however it relays on pos taggers for target language, which does not exist for newly proposed languages, and its tagging accuracy may be low, and it costs extra processing steps. in the future work, we plan to carry out integrated model which combine the pos tagging as a command function in data pre-processing for hybrid cushlepor. overall, cushlepor achieved the first cluster performances in news domain data on chinese-english and english-german in wmt2021 metrics task, while hlepor wins ted domain data on english-russian @xcite . in the future work, we plan to carry out optimisation of cush-lepor on more language pairs as well as more domains. we will keep our updated parameter set for extended languages and domains available on our cushlepor open-sourced platform.