Partial Could Be Better than Whole. HW - TSC 2022 Submission for the Metrics Shared Task (2022)

Abstract: In this paper, we present the contribution of HW-TSC to WMT 2022 Metrics Shared Task. We propose one reference-based metric, HWTSC-EE-BERTScore*, and four referencefree metrics including HWTSC-Teacher-Sim, HWTSC-TLM, KG-BERTScore and CROSSQE. Among these metrics, HWTSC-Teacher-Sim and CROSS-QE are supervised, whereas HWTSC-EE-BERTScore*, HWTSC-TLM and KG-BERTScore are unsupervised. We use these metrics in the segment-level and systemlevel tracks. Overall, our systems achieve strong results for all language pairs on previous test sets and a new state-of-the-art in many sys-level case sets.

## introduction due to the expensive cost of manual evaluation, automatically evaluating the outputs of translation systems is critically important in the field of machine translation (mt) @xcite . therefore, a lot of automatic metrics have been proposed to approach this task. according to whether the reference sentences are required or not, the metrics are categorized into two classes: (1) referencebased metrics like bleu @xcite , meteor @xcite , bertscore @xcite and bleurt @xcite , which evaluate the hypothesis by referring to the golden reference; (2) reference-free metrics like yisi-2 @xcite and comet-qe @xcite , which are also referred as quality estimation (qe). these metrics estimate the quality of hypothesis only based one source sentences without using references. in this paper, we present the contribution of hw-tsc to the wmt 2022 shared task on metrics. we participated in the segment-level and system-level tracks with 1 reference-based metric (hwtsc-ee-bertscore*) and 4 reference-free metrics (hwtsc-teacher-sim, hwtsc-tlm, kg-bertscore and cross-qe). details of our metrics are illustrated in hwtsc-ee-bertscore* (entropy enhanced metrics)is built upon existing metrics, aiming to achieve a more balanced system-level rating by assigning weights to segment-level scores produced by backbone metrics. the weights are determined by the difficulty of a segment, which is related to the entropy of a hypothesis-reference pair. a translation hypothesis with a significantly high entropy value is considered difficult and receives a large weight in aggregation of ee-metrics' system-level scores. hwtsc-teacher-sim is a supervised referencefree metric with the framework of bertscore @xcite , which is obtained by fineturning the multilingual sentence-bert model @xcite . both the unsupervised tearchersim @xcite and the implicit multilingual word embedding alignment @xcite have shown that the pretained multilingual sentence-bert model is very effective for both reference-based and referencefree mt evaluations on wmt da (direct assessment) data. however, its performance on wmt mqm (multidimensional quality metrics) data is poor. we propose an effective training strategy for the pretrained multilingual sentence-bert and a novel normalization method for the da and mqm scores. hwtsc-tlm @xcite ) is an unsupervised reference-free metric which only uses the system translations as input and calculates the scores by a target-side language model. although source sentences are not considered, the results of this metric with xlm-r @xcite ) on wmt19 are very promising. kg-bertscore @xcite is an unsupervised reference-free metric, which incorporates multilingual knowledge graph into bertscore @xcite . the score of this metric is calculated by linearly combining the results of bertscore and bilingual named entity matching. cross-qe is an application of "qe as a metric". based on our previous work @xcite @xcite , we propose a reference-free metric, like comet-qe architecture. ## metrics this section introduces our metrics for wmt metrics 2022 shared task including reference-based and reference-free. ## conclusions in this paper, we present one reference-based metric and four reference-free metrics. we apply the