Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLM s (2023)

Abstract: Translating literary works has perennially stood as an elusive dream in machine translation (MT), a journey steeped in intricate challenges. To foster progress in this domain, we hold a new shared task at WMT 2023, the first edition of the Discourse-Level Literary Translation. First, we (Tencent AI Lab and China Literature Ltd.) release a copyrighted and document-level Chinese-English web novel corpus. Furthermore, we put forth an industry-endorsed criteria to guide human evaluation process. This year, we totally received 14 submissions from 7 academia and industry teams. We employ both automatic and human evaluations to measure the performance of the submitted systems. The official ranking of the systems is based on the overall human judgments. In addition, our extensive analysis reveals a series of interesting findings on literary and discourse-aware MT. We release data, system outputs, and leaderboard at http://www2.statmt.org/wmt23/literary-translation-task.html.

## introduction in past decades, the evolution of machine translation (mt) has undergone significant improvements in accuracy and efficiency, leading to many practical applications in various fields @xcite @xcite . despite its success, mt still struggles in certain intricate scenarios to deliver translations that meet high standards @xcite . translating literary texts is considered to be the greatest challenge for mt due to its complex nature @xcite @xcite ): • rich linguistic and cultural phenomena: literary texts contain more complex linguistic and cultural knowledge than non-literary ones @xcite . to generate a cohesive and coherent output, mt models require an understanding of the intended meaning and structure of the text at discourse level @xcite @xcite . furthermore, it demands skillful adaptation of cultural references, idioms, and subtle expressions to capture the essence of the original work in target languages. with the swift progression of mt and the notable advancements in large language models (llm) @xcite , our curiosity is piqued regarding the efficacy of mt and llm in the realm of literary translation. we aim to explore the extent to which these technologies can aid in addressing the intricate challenges of translating literary works. therefore, we hold the first edition of the discourse-level literary translation in wmt 2023. literary texts encompass a wide range of forms, including novels, short stories, poetry, plays, essays, and more. among these, web novels, also known as online or internet novels, represent a unique and rapidly growing subset of literature. their popularity, accessibility, and diverse genres set them apart. as they provide not only an extensive volume of text but also exhibit distinctive linguistic features, cultural phenomena, and simulations of societies, web novels can serve as valuable resources and challenging for mt research. this year, the shared task mainly focuses on document-level web novels, and we introduce a document-level benchmark dataset and establish human evaluation criteria specifically tailored to address the challenges of literary translation: ## the guofeng webnovel corpus we release a copyrighted and high-quality chinese-english corpus on web novels. additionally, we provide in-domain pretrained models as supplementary resources. as shown in figure 1 , a total of 45 institutes and companies from various regions have downloaded our dataset, showing that the prposed tasks and data have garnered widespread interest. ## evaluation methods it is still an open question whether human and automatic evaluation metrics are complementary or mutually exclusive in measuring the document-level and literary translation quality. thus, we report both automatic and human evaluation methods, and officially rank the systems based on the overall human judgments. ## participants' and baseline systems here we briefly introduce each participant's systems and refer the reader to the participant's reports for further details. ## conclusion and future work we believe that the wmt2023 shared task on discourse-level literary translation will be a valuable contribution to the field of machine translation and will encourage further research in this area. we discuss the potential limitations of this edition of the shared task as follows: