Automatic Song Translation for Tonal Languages (2022)

Abstract: This paper develops automatic song translation (AST) for tonal languages and addresses the unique challenge of aligning words’ tones with melody of a song in addition to conveying the original meaning. We propose three criteria for effective AST—preserving meaning, singability and intelligibility—and design metrics for these criteria. We develop a new benchmark for English–Mandarin song translation and develop an unsupervised AST system, Guided AliGnment for Automatic Song Translation (GagaST), which combines pre-training with three decoding constraints. Both automatic and human evaluations show GagaST successfully balances semantics and singability.

## introduction suppose you are asked to translate the lyrics "let it go" from the disney musical frozen into mandarin chinese. some good, literal translations of this would be a) "fàng shǒu", b) "fàng shǒu ba" or c) "ràng tā qù ba" (figure 1 ); these get the meaning across and are the domain of traditional machine translation. however, what if you needed to sing this song in mandarin? these literal translations simply do not work: translations a and c do not match the number of notes and break the original rhythm; while the tones of translation b does not match the pitch flow of the original melody. song translation, unlike translation lyrics for understanding (subtitling), aims to translate the lyrics so that it can be sung with the original melody. therefore, the translated lyrics must match the prosody of the pre-existing music in addition to retaining the original meaning. in singable translations of songs, @xcite says, this is an uncommon of these, only the official human song translation is something a singer could actually sing: it fits the length of the notes and matches the tones with the pitch of notes. gagast finds translations that satisfy these constraints. and an unusually complex task, a translator consider rhythm, notes' pitches, phrasing, and stress. nonetheless, there are cultural and commercial incentives for more efficient song translation; frozen alone made over a half a billion dollars in non-english box office receipts foot_0 and the musical les misérables has been performed in over a dozen languages on stage. as we discuss in section 2, while translating western songs resembles poetry translation, translating into tonal languages (e.g., mandarin, zulu and vietnamese) introduces new problems. in tonal languages, a word's pitch contributes to its meaning (figure 2 ); when singing in tonal languages, the figure from @xcite . this paper builds the first system for automatic song translation (ast) for one tonal language-mandarin. section 3 proposes three criteriapreserving semantics, singability and intelligibility-needed in an ast system. guided by those goals, we propose an unsupervised ast system, guided alignment for automatic song translation (gagast). gagast begins with an out-of-domain translation system (section 4.1) and adds song alignment constraints that favor translations that are the appropriate length and whose tones match the underlying music (section 4.2). naturally, such constraints trade-off between semantic meaning and singability/intelligibility. section 5.4 discusses this trade-off between song alignment scores and the standard machine translation metric, bleu. these criteria also form the evaluation for our initial evaluation (section 5.3). however, we go beyond an automatic evaluation through a humancentered evaluation from musicology students. gagast creates singable songs that make sense given the original text, and our proposed alignment scores correlate with human judgements (section 5.4.3). foot_1 ## background: prose, poetry, and song translation a spoken language can be divided into two forms: prose, which corresponds to natural conversa- misheard lyrics (consistent tone) original lyrics (inconsistent tone) 似 在 眼 前 死 在 眼 前 sì zài yǎn qián sǐ zài yǎn qián appear where eye front death where eye front as if before my eyes die before my eyes inter-syllable pitch alignment score: 0.5 inter-syllable pitch alignment score: 0.75 figure 3 : if a song's music doesn't match the tones of the lyrics, it can cause the hearer to misunderstand the lyrics. in this example, someone can hear "s ǐ zài" instead of "sì zài", because the notes are going up and "sì zài" is going down. the vast majority of machine translation research has been focused on prose translation and has made huge progress; in contrast verse translation is more difficult as it must obey the rhythmic constraints and is less developed. in his tour de force work le ton beau de marot, douglas hofstadter presents eighty-nine translations of a single poem to capture the panoply of considerations of what makes the task difficult @xcite . in western verse, the rhythmic structure are mostly defined by meter, such as the iambic pentameter for sonnets, which defines the length of each line, the patterns of long syllables versus short ones and the stressed ones versus weak ones. existing work @xcite use finite-state constraints to encode both meter and rhyme. song translation, on the other hand, can be viewed as a translation where the melody defines the constraints. reproducing all of the essential values of a song-perfectly matching the meaning, perfectly singable, and perfectly understandable-is an impossible ideal @xcite . thus, tradeoffs are unavoidable. @xcite argues for prioritizing singability over other qualities such as sense and rhyme since "effectiveness on stage" is a practical necessity. tonal languages (e.g., mandarin, zulu and vietnamese) dramatically increases the complexity of singability, and introduces a new factor that could hamper intelligibility. ## ast for tonal languages this section formally defines automatic song translation (ast) for tonal languages and introduce three criteria for what makes for a good song translation. these criteria form the foundation for the quantitative metrics we use in the experiment. ## gagast ideally, we would build an ast system for english-mandarin song translation with data-driven models from parallel data, i.e., aligned triples (m , x, y ). however, these data are not available in the quantity or quality necessary for mandarin: there is not enough data of any quality, and those that do exist have errors in the syllable-notes alignment. thus, we propose an unsupervised ast system, guided alignment for automatic song translation (gagast). for the pre-training, we collect non-parallel lyrics data in both english and mandarin, as well as a small set of lyrics translation data (section 5.1). ## generating melody-constrained lyrics and validating singability this section details data sets, model configuration, and proposed evaluation metrics. then we analyze the results and the trade-offs inherent in song translation. our code and data are open-sourced at https://github.com/gagast . ## related work verse generation and translation. generating verse text began through rule-based implementations @xcite and developed through the next forty years. @xcite design a chart system that generate strings that match a given stress pattern. gervás (2000) build a forward reasoning rule-based system. manurung (2004) address poetry generation with stochastic search based on evolutionary algorithms. oliveira (2012) create a template-based platform that allows user to define features and create templates. he et al. (2012) adopt statistical machine translation models for chinese poetry generation. yan et al. (2013) compose poetry based on generative summarization framework. zhang and lapata (2014), wang et al. (2016), and hopkins and kiela (2017) adopt recurrent neural networks for poetry generation and incorporate rhythmic constraints. ghazvininejad et al. (2016, 2017) represent rhythm and rhyme with finite-state machines. poetry translation using these frameworks and statistical machine translation thus offers elegant solutions: genzel et al. (2010) intersect the finite state representation of the meter and rhyme scheme with the synchronous context-free grammar of the translation model under the phrase-based machine translation framework. ghazvininejad et al. (2018) apply the finitestate constraints to neural translation model. however, these representations of the rhythmic and lexical constraints are not flexible enough to encode the real-valued representation of a song as required for translation in tonal languages. ## conclusion this paper addresses automatic song translation (ast) for tonal languages and the unique challenge of aligning words' tones with melody. and we build the first english-mandarin ast system -gagast. both objective and subjective evaluations demonstrate that gagast successfully improves the singability and intelligibility of translated songs. more constraints are left in the future work such as rhymes and style. we aim to build a systematic framework that address all constraints. with the help of newly developed singing voice synthesize tools such as x studio, 9 we can perform human evaluation with actual singing voice with a larger scale to provide more reliable analysis. moreover, our system can also be applied in lyrics and song generation applications without translation input.