G uo F eng: A Benchmark for Zero Pronoun Recovery and Translation (2022)

Abstract: The phenomenon of zero pronoun (ZP) has attracted increasing interest in the machine translation (MT) community due to its importance and difficulty. However, previous studies generally evaluate the quality of translating ZPs with BLEU scores on MT testsets, which is not expressive or sensitive enough for accurate assessment. To bridge the data and evaluation gaps, we propose a benchmark testset for target evaluation on Chinese-English ZP translation. The human-annotated testset covers five challenging genres, which reveal different characteristics of ZPs for comprehensive evaluation. We systematically revisit eight advanced models on ZP translation and identify current challenges for future exploration. We release data, code, models and annotation guidelines, which we hope can significantly promote research in this field (https://github.com/longyuewangdcu/mZPRT).

## introduction zero pronoun (zp) is a discourse phenomenon that appears frequently in pronoun-dropping (pro-drop) languages such as chinese and japanese. specifically, pronouns are often omitted when they can be pragmatically or grammatically inferred from intraand inter-sentential contexts @xcite . since recovery of such zps generally fails, this poses difficulties for several generation tasks, including dialogue modelling @xcite , question answering @xcite as well as machine translation (mt) @xcite . although neural machine translation (nmt) @xcite @xcite has achieved great progress in recent years, translating zps from a pro-drop to a non-pro-drop language is a significant and challenging task @xcite . as shown in ## mzprt: new benchmark dataset in general, the process of manual labeling totally spends five annotators and two translators two months, which costs us $10,000 dollars. ## experiment we systematically evaluate existing models on the proposed mzprt benchmark to 1) investigate effects of zpr on zpt; 2) build a benchmark of zpt methods; 3) revisit document-level nmt models. ## conclusion we revealed data-and evaluation-level gaps in previous works on translating zps. accordingly, we proposed a benchmark testset and evaluation metric for target evaluation on zpt. our benchmark emphasizes the large gap between performances of existing models and an upper bound from the perspective of zpt. we release data (benchmark testset, human judgments and collected training data), code (target metric and implemented approaches) and models (comparative models), which we hope can significantly promote research in this field.