Empowering a Metric with LLM -assisted Named Entity Annotation: HW - TSC ’s Submission to the WMT 23 Metrics Shared Task (2023)

Abstract: This paper presents the submission of Huawei Translation Service Center (HW-TSC) to the WMT23 metrics shared task, in which we submit two metrics: KG-BERTScore and HWTSC-EE-Metric. Among them, KG-BERTScore is our primary submission for the reference-free metric, which can provide both segment-level and system-level scoring. While HWTSC-EE-Metric is our primary submission for the reference-based metric, which can only provide system-level scoring. Overall, our metrics show relatively high correlations with MQM scores on the metrics tasks of previous years. Especially on system-level scoring tasks, our metrics achieve new state-of-the-art in many language pairs.

## introduction due to the expensive cost of human evaluation, automatic metrics @xcite for machine translation (mt) @xcite is critically important for mt research and development. while human evaluation is still very important, automatic metrics allow the rapid evaluation and comparison of mt systems on large collections of text and facilitate expansion to low resource languages @xcite and domains @xcite . depending on whether the references are required or not, automatic metrics are categorized into two categories: the wmt23 metrics shared task invites submissions of reference-free metrics and reference-based metrics to find automatic metric scores for translations at the segment-level and system-level. this paper presents the contribution of hw-tsc to the wmt23 metrics shared task. slightly different from our participation last year @xcite , we only submit two metrics this year. details of our metrics (kg-bertscore and hwtsc-ee-metric) are illustrated in ## metrics this section introduces our metrics for wmt23 metrics shared task, including kg-bertscore and hwtsc-ee-metric. ## experiments this section introduces the experimental results of kg-bertscore and hwtsc-ee-metric on previous metrics shared tasks. ## conclusion this paper presents hw-tsc's submission to the wmt23 metrics shared task, in which we summit a reference-free metric (kg-bertscore) and a reference-based metric (hwtsc-ee-metric). we have made some improvements to these two metrics compared to last year's submission. one of the most critical improvements is on kg-bertscore, we empower the metric with llm-assisted ne annotations, significantly improving its correlation with mqm scores. the experimental results on previous wmt metrics tasks show great effectiveness of our research direction and the superiority of our metrics.