Don’t Go Far Off: An Empirical Study on Neural Poetry Translation (2021)

Abstract: Despite constant improvements in machine translation quality, automatic poetry translation remains a challenging problem due to the lack of open-sourced parallel poetic corpora, and to the intrinsic complexities involved in preserving the semantics, style and figurative nature of poetry. We present an empirical investigation for poetry translation along several dimensions: 1) size and style of training data (poetic vs. non-poetic), including a zero-shot setup; 2) bilingual vs. multilingual learning; and 3) language-family-specific models vs. mixed-language-family models. To accomplish this, we contribute a parallel dataset of poetry translations for several language pairs. Our results show that multilingual fine-tuning on poetic text significantly outperforms multilingual fine-tuning on non-poetic text that is 35X larger in size, both in terms of automatic metrics (BLEU, BERTScore, COMET) and human evaluation metrics such as faithfulness (meaning and poetic style). Moreover, multilingual fine-tuning on poetic data outperforms bilingual fine-tuning on poetic data.

## introduction american poet robert frost once defined poetry as "that which gets lost out of both prose and verse in translation" @xcite . indeed, the task is so complex that translators often have to "create a poem in the target language which is readable and enjoyable as an independent, literary text" @xcite . but even though poetry is destined to lose its accuracy, integrity, and beauty even in human translation, the process conceives new opportunities to stress-test the ability of machine translation models to deal with figurative language. while most computational work has focused on poetry generation @xcite il ny avait que sable et boue où sétait ouverte la tombe. le long des murs de la prison on ne voyait aucune tombe human for where a grave had opened wide, there was no grave at all: only a stretch of mud and sand by the hideous prison-wall, gck but there was only sand and mud. to where the grave was laid. along the walls of prison wall. ## experimental setting we experiment with several systems to evaluate performance across several dimensions: poetic vs non-poetic data; multilingual fine-tuning vs. bilingual fine-tuning; language-family-specific models vs. mixed-language-family models. ## results our results based on automatic metrics are summarized in however, when comparing the bilingually finetuned models (poetic vs. non-poetic bi(opus)) the pattern is not as clear based on automatic metrics. we see comparable performance, but not a clear winner across languages and metrics. however, as with the multilingual case, the size of poetic data is much smaller than the non-poetic data (20x to 50x smaller depending on the language). we also mixed poetic and non-poetic data in equal proportion and fine-tuned mbart by framing it as a domain adaption problem, however it did not lead to significant improvements and degenerated in a few languages. we also tried intermediate finetuning @xcite , where we first fine-tune a pre-trained mbart model on our non-poetic data and then fine-tune the best model checkpoint on our poetic data. the results for this experiment also did not lead to any significant difference in performance. the third insight is that language-family-specific multilingual fine-tuning (poetic langfamily) helps in some of the languages when compared to mul- last but not least, we notice that the multilingual fine-tuned model on poetic data (poetic all) is consistently better than the bilingual fine-tuned model on poetic data (poetic) across all languages. while we show that multilingual fine-tuning is an effective way to improve performance on low resource poetic data, we believe techniques like iterative backtranslation @xcite with sophisticated techniques for data selection (dou the zero-shot performance of our model fine-tuned multilingually on poetic data (excluding the unseen languages) to the non-poetic multi (opus) and non-poetic multi (ml50) model. we chose ukrainian, romanian and swedish poetry given the fact that our model is fine-tuned on poetry belonging to languages from the slavic, romance, and germanic families. ## shortcomings of style transfer techniques as a post-editing tool we evaluate whether style transfer techniques could help attenuate the shortcomings of translation models trained on non-poetic data. we use the romantic poetry style transfer model provided by @xcite to paraphrase our non-poetic translations. this is the only available poetic style transfer model to our knowledge.to control for faithfulness, we generate 20 outputs for each input (i.e., non-poetic translations) using nucleus sampling (@xmath0.6), we then select the sentence that has the highest similarity score with input using the sim model by @xcite . the style transfer experiments decrease performance across all languages on both bleu and bertscore metrics as evaluated on the multi(opus) model (see results in qualitatively, this may happen due to errors cascading from incorrect translations by the nonpoetic model, introduction of archaic language where it is not appropriate, and change in meaning. an example output is provided in ## analysis it is well-known that occasionally nmt systems have a tendency to generate translations that are grammatically correct but unrelated to the source sentence particularly for low-resource settings (e.g., hallucinate words that are not mentioned in the source language) @xcite . pre-trained multilingual language models and techniques like multilingual training or fine-tuning can indeed be effective for dealing with low-resource data such as poetry as seen in figures 1 , 2 , 3 , showing examples of poetic translations by poetic all and multi(opus) configurations. however, it is surprising that even a model trained on 6m parallel lines from opus(100) performs worse than models trained on in-domain data that is 35x smaller. gold: i love the lavish withering of nature, the gold and scarlet raiment of the woods, the crisp wind rustling o'er their threshold, the sky engulfed by tides of rippled gloom, the sun's scarce rays, approaching frosts, and gray-haired winter threatening from afar. ## conclusion and future work we release poetic parallel corpora for 6 language pairs. our work shows the clear benefit of domain adaptation for poetry translation. it further shows that improvements can be achieved by leveraging multilingual fine-tuning, and that the improvements transfer to unseen languages. future directions include addition of new languages and larger corpora, adapting low-resource machine translation techniques for poetry translation, translating to languages that are morphologically richer than english, as well as working on better evaluation metrics to detect hallucinations. while computational methods for poetry translation may never outperform the human standard, we hope our contributions spark interest in the machine translation community to take up this rather challenging task. additionally, by open-sourcing our work we hope to provide a helpful resource for professional translators.