Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics (2023)

Abstract: Song translation requires both translation of lyrics and alignment of music notes so that the resulting verse can be sung to the accompanying melody, which is a challenging problem that has attracted some interests in different aspects of the translation process. In this paper, we propose Lyrics-Melody Translation with Adaptive Grouping (LTAG), a holistic solution to automatic song translation by jointly modeling lyric translation and lyrics-melody alignment. It is a novel encoder-decoder framework that can simultaneously translate the source lyrics and determine the number of aligned notes at each decoding step through an adaptive note grouping module. To address data scarcity, we commissioned a small amount of training data annotated specifically for this task and used large amounts of automatic training data through back-translation. Experiments conducted on an English-Chinese song translation data set show the effectiveness of our model in both automatic and human evaluations.

## introduction song translation is a meaningful human endeavor to climb high tower of babel for inter-culture exchange. yet it has not received much attention in the natural language processing (nlp) community despite the advancement of machine translation technologies, especially neural machine translation (nmt) @xcite @xcite , and the expanding interests of solving real-world problems using artificial intelligence techniques. challenges include the lack of efficient means to collect parallel lyrics and alignment data, the difficulty of modeling the complex interaction between texts and melody and rolling in the deep imperceptive evaluation of scores. while closely related to text translation, song translation is a more involved task. in addition to the general considerations of word choice and word order in translation, human translators of songs need to have a mastery of cultural traditions and the poetic usage of both source and target languages. furthermore, the translated lyrics need to be properly aligned with the melody, as shown in figure 1 , to maintain the intact beauty of the song, a factor that is indispensable in song translation @xcite . researchers have explored singing voice synthesis (svs) @xcite to automate the vocal singing of songs given the input lyrics and scores, which laid the foundation of convenient and perceptive evaluation and a prospective empirical usage of automatically generated songs. however, there is very few previous studies in the direction of automatic song translation (ast). the sole work @xcite we are aware of focuses on matching tones and rhythms for the translated target words for tonal languages, by imposing constraints during nmt inference. their direct use of text translation models and strict mapping between notes and tokens, however, is unable to capture the more involved nature of song translation. while the number of notes provides an easy upper bound on the length of translation, the delicate alignment between lyrics and melody, as observed in @xcite , should not be dictated solely by simple rigid rules. in this paper, we propose lyrics-melody translation with adaptive grouping (ltag), the first comprehensive solution to the ast problem, by jointly modeling of both lyrics translation and lyrics-melody alignment within the transformerbased encoder-decoder framework. ltag incorporates both lyrics and melody in an end-to-end manner and employs an adaptive grouping module to explicitly model the alignment between lyrics and melody. to facilitate training, we produce the first (chinese-english) bilingual lyrics-melody alignment data set. to address the data scarcity problem, we also generate a large amount of bilingual lyricsmelody data through back-translation of monolingual lyrics-melody alignment data, which is used together with the high quality manual annotations through a curriculum training strategy. our experiments show that songs translated by ltag are both faithful to the original lyrics and singable to the melody, as measured by both automatic metrics and human judges majoring in music. main contributions of this work are as follows: ## related works lyrics and song translations have recently drawn attention from the nlp community. automatic lyrics translation has been approached by rulebased methods, statistical machine translation methods, finite-state methods with rhythmic and lexical constraints @xcite @xcite , and more recently by neural methods @xcite @xcite . traditional song translation research has made progress in terms of lyrics translation and lyrics-melody alignment through linguistic knowledge @xcite @xcite @xcite . often, the object of these research was artificial songs. these methods pursue lyrics-melody alignment and lyrics translation in separate tracks. @xcite pursue song translation as a type of constrained text translation. previous works @xcite @xcite on constraining the decoding process are shown to be effective in performance and convenient in implementation. others that impose constraints during training, such as adding format embedding @xcite , introducing special tags and rescoring length control @xcite , are data-driven methods and show good performance. in this paper, we propose the lyrics translation model with lyricsmelody alignments for domain shift and length control, and overcome the problem of domain mismatch and data sparsity by using monolingual data. lyrics generation with alignment prediction, one of the most important tasks in automatic song production, has received much attention recently. in recent works of machine translation, graph neural networks (gnns) are widely used for alignment prediction between the source and target sentences @xcite . in our researched domain, most of the current works @xcite @xcite @xcite adopt the sequence generation method, but with different objectives. some constrain rhythmic alignment, others theme and target genre. other works @xcite apply the attention mechanism and find the lyrics-melody alignment via dynamic programming on the attention weights matrix. this method sometimes results in nonmonotonic output. most importantly, it seems that their alignment component is akin to a postprocessing module rather than an integrated unit that learns the dynamic alignment such that the lyrics generation is constrained. in our proposal, we take advantage of the monotonic nature of the lyrics-melody alignments and design a light neural network for alignment prediction in parallel to the translation process. ## methodology in this section, we first describe the ltag as shown in figure 2 . then, we detail the adaptive grouping method for alignment prediction and explain how we adapt back-translation for the ast task. ## experiments in this section, we describe the experiment setup, results and analysis on chinese↔english song translation. ## conclusion in this work, we propose ltag, a lyrics translation model with lyrics-melody alignments that allows simultaneous generation of target text and alignment to the music notes. we propose an adaptive grouping method that fits in the auto-regressive translation process. to better train and evaluate our model, we also annotate a new song translation data set popcv containing english and chinese songs with their cover version in both languages and with lyrics-melody alignments. for training, we also employ back-translation to leverage the more abundantly available monolingual lyrics data with lyricsmelody alignments in a curriculum learning way. evaluations with both the automatic and human metrics show that ltag is capable of producing natural, singable and enjoyable translation results.