MAX - ISI System at WMT 23 Discourse-Level Literary Translation Task (2023)

Abstract: This paper describes our translation systems for the WMT23 shared task. We participated in the discourse-level literary translation task - constrained track. In our methodology, we conduct a comparative analysis between the conventional Transformer model and the recently introduced MEGA model, which exhibits enhanced capabilities in modeling long-range sequences compared to the traditional Transformers. To explore whether language models can more effectively harness document-level context using paragraph-level data, we took the approach of aggregating sentences into paragraphs from the original literary dataset provided by the organizers. This paragraph-level data was utilized in both the Transformer and MEGA models. To ensure a fair comparison across all systems, we employed a sentence-alignment strategy to reverse our translation results from the paragraph-level back to the sentence-level alignment. Finally, our evaluation process encompassed sentence-level metrics such as BLEU, as well as two document-level metrics: d-BLEU and BlonDe.

## introduction this paper introduces our submissions to the wmt23 shared task: discourse-level literary translation (zh-en), constrained track. our submission comprises three translation systems: a primary system employing a paragraph-level transformer, a first contrastive system utilizing a sentence-level transformer, and a paragraph-level mega model as the second contrastive system. until very recently, the predominant focus of context-aware neural machine translation (nmt) research has been on parallel datasets that align at the sentence level, such as iwslt17 @xcite and opus @xcite . more ## model architectures we select the following two model architectures for our systems, taking into account their strong performance in the context of context-aware machine translation. transformer the transformer architecture, as introduced by @xcite , utilizes an encoder-decoder framework, leveraging a selfattention mechanism. this mechanism enables each position within a given sequence to interact with every other position, facilitating the computation of a comprehensive representation for the entire sequence. in all our experiments, we employ the transformer base version which consists of 6 encoder layers, 6 decoder layers, a model dimension of 512, and a ffn hidden dimension of 2048. ## results the results of our experiments are presented in the results of the test set are presented in transformer vs. mega as per the outcomes presented in sent-level vs. paragraph-level based on the results presented in as delving into the four distinct categories in blonde, a consistent trend of enhancement emerges across each category with the adoption of paragraph-level translation. particularly, marked improvements are observed within the pronoun and tense categories. this can be attributed to the inherent reliance of pronouns and tenses on contextual information. these empirical results demonstrate that paragraph-level data provides more useful contextual signals than sentence-level data. ## discussion limitation of sentence alignment literary texts often rely on context that spans beyond individual sentences, making strict sentence alignment impractical. as evidenced in our results, paragraphlevel translation excels in preserving contextual information, like pronouns and tenses. however, the insistence on maintaining sentence-level alignment imposes constraints on model selection, hindering flexibility and adaptability. ## conclusion this paper describes the submission to the wmt23 literary translation shared task -constrained track. we compare traditional transformer models to the newer mega model, integrating paragraph-level data into both. transformer models outperform mega in both sentence and paragraph translation on this literary dataset. we observe a discrepancy between bleu and blonde evaluations, with the latter favoring paragraph-level translation. these results emphasize the challenges of document-level translation and the importance of more contextaware evaluation metrics.