Songs Across Borders: Singable and Controllable Neural Lyric Translation (2023)

Abstract: The development of general-domain neural machine translation (NMT) methods has advanced significantly in recent years, but the lack of naturalness and musical constraints in the outputs makes them unable to produce singable lyric translations. This paper bridges the singability quality gap by formalizing lyric translation into a constrained translation problem, converting theoretical guidance and practical techniques from translatology literature to prompt-driven NMT approaches, exploring better adaptation methods, and instantiating them to an English-Chinese lyric translation system. Our model achieves 99.85%, 99.00%, and 95.52% on length accuracy, rhyme accuracy, and word boundary recall. In our subjective evaluation, our model shows a 75% relative enhancement on overall quality, compared against naive fine-tuning (Code available at https://github.com/Sonata165/ControllableLyricTranslation ).

## introduction with the globalization of entertainment, it is becoming increasingly common for people to appreciate songs in foreign languages. meanwhile, artists are internationalizing their work and building territories worldwide. nevertheless, an unfriendly barrier exists between the artists and the audience: most commercial songs are not written in multiple languages. worse still, most existing song translations entirely ignore the music constraints, rendering 1 code available at https://github.com/sonata165/controllablelyrictranslation them unsingable alone with the music. as a result, the language barrier complicates the interaction between artists and their audience. obtaining singable lyric translations can facilitate the globalization of the music publishing industry to further promote the growth of its $5.9 billion usd market size @xcite . however, song translation is unusually difficult for human translators, due to music constraints and style requirements. if we can construct lyricspecific machine translation (mt) systems that can produce drafts that satisfy these constraints and requirements, the difficulty and cost of lyric translation will be largely reduced, as lyricists and translators can start with such automatic drafts and can focus on post-processing for quality and creativity. however, obtaining singable lyrics from mt systems is challenging. figure 1 shows two sentences of lyrics from the song let it go, together with an mt output and a singable translation. we observe a notable quality gap between them. while the mt output correctly translates the source, it ignores all the criteria that matter to make the output singable: (1) the second sentence of the mt outputs is unnatural because of incoherent vocabulary selection and lack of aesthetics. (2) overcrowded syllables in the first sentence of the mt outputs force performers to break music notes in the orange box into multiple pieces to align them with lyrics. the rhythm pattern consequently diverges from the composer's intention. (3) the two-syllable word in the red box is situated across a musical pause (blue box), let it go in intro subtitle composer / arranger causing an unnatural pronunciation. (4) the endsyllables (purple text) are not of the same rhyme pattern, making the output miss a key chance for being poetic. in contrast, the singable translation in the third row outperforms the mt output in all four aspects, all while maintaining translation fidelity: it perfectly aligns with each musical note, has the same end-rhyme pattern for the two sentences (green text), a natural stop at the musical pause, and higher naturalness. these properties make it a significantly more performable translation. to address these quality gaps to obtain singable lyric translations from neural machine translation (nmt) systems, we formalize singable lyric translation as an instance of constrained translation, identify useful constraints, and propose a languagepair independent approach that combines translatology theoretical guidance with prompt-driven nmt. our contributions are: ## method to bridge the gaps of previous research, we identify comprehensive controlling aspects from the translatology literature, propose prompt-based solutions for each aspect, and explore more effective foundation models and adaptation methods. ## experiment we tested our methods with english-chinese lyric translation. we obtained a small amount of parallel data (about 102k paired sentences after deduplication) by crawling data of both english-chinese and chinese-english pairs from an online lyric transla-tion sharing platform foot_2 . for target-side monolingual data, we adopted lyric data from three publiclyavailable datasets 567 , resulting in about 5.5m sentences after deduplication. for details of dataset statistics and splits, data preprocessing, and back translation, please refers to appendix a. ## results ## conclusion we discussed how to obtain singable translations with prompt-driven nmt systems with the guid-10 translation outputs are available at https://www.oulongshen.xyz/lyric_translation ance of translatology theories. specifically, we used back-translation to enhance translation quality and naturalness. we compared the effectiveness of different prompt methods in different controlling aspects and showed their advantage over biased decoding. we designed an effective word boundary control approach and presented a training strategy without the help of music data. we demonstrated the effectiveness of reverse-order decoding in nmt models for rhyme control and showed how it helps users to choose the best suitable rhymes for a paragraph of source text. this work does not explore more detailed prompt manipulation, such as using varied prompts for the same constraint or examining prompt order's impact on performance. we leave these investigations for future research.