Sing it, Narrate it: Quality Musical Lyrics Translation (2024)

Abstract: Translating lyrics for musicals presents unique challenges due to the need to ensure high translation quality while adhering to singability requirements such as length and rhyme. Existing song translation approaches often prioritize these singability constraints at the expense of translation quality, which is crucial for musicals. This paper aims to enhance translation quality while maintaining key singability features. Our method consists of three main components. First, we create a dataset to train reward models for the automatic evaluation of translation quality. Second, to enhance both singability and translation quality, we implement a two-stage training process with filtering techniques. Finally, we introduce an inference-time optimization framework for translating entire songs. Extensive experiments, including both automatic and human evaluations, demonstrate significant improvements over baseline methods and validate the effectiveness of each component in our approach.

## introduction have you ever heard of hamilton in chinese, or mamma mia in swedish @xcite ? advancements in cultural globalization allow musicals to reach universal audiences, but language barriers still hinder full comprehension. translating musicals into performing country's language could enhance audience experience @xcite and expand commercial outreach @xcite , as it allows audiences to enjoy theatrical elements without heavily relying on subtitles @xcite . however, musical translation is labor-intensive and time-consuming, requiring adjustments for musical framework, stage performance, and cultural references beyond mere verbatim translation @xcite . to alleviate this burden, we aim to automatically translate musical lyrics from english to chinese. song translation, a closely related field, requires aligning the translated text with the music to ensure the translated lyrics can be sung @xcite . however, musical translation requires an even higher standard of translation quality because lyrics play a crucial role in the story-telling of a musical @xcite @xcite . to preserve the original narration, the translations must accurately convey the meaning and nuance of the source lyrics. this high fidelity ensures that the translated musical maintains its artistic integrity and allows the story to unfold as intended for the target audience. thus, musical translation demands a rigorous approach to translation quality, focusing on maintaining the narrative function to create a faithful rendition of the original work. to the best of our knowledge, there is no existing work on automatic musical translation, and existing works on automatic song translation @xcite @xcite ) mainly focus on the alignment of text and music, sacrificing translation quality and often produce unnatural and inaccurate translations unsuitable for musicals, as shown in figure 1 . to distinguish our work from existing art, we focus on improving translation quality, which would contribute to maintaining the narrative function, while reasonably following singability constraints. we define translation quality using the well-established criteria for literature translation: fluency, accuracy, and literacy @xcite . additionally, we consider the singability constraints of length and rhyme following previous works @xcite . figure 1 shows our considered aspects, with examples demonstrating their significance. to depict translation quality, we collect english-chinese lyric pairs using large language models (llms), label them according to our scoring figure 1 : aspects we considered include length, rhyme, and translation quality. the proper length of translated lyrics is the number of notes, and the end rhyme of each line (shown in parentheses) is better to have the same type (shown in the same color). google translation fails to follow the length constraint and misaligns with music, as shown in red boxes, and its rhyme does not match. both the baseline and our results meet length and rhyme constraints, but the baseline has inaccurate translations and inappropriate phrases, while our model generates higher-quality lyrics. to sum up, we make the following contributions: (1) we propose the task of musical translation, which requires a higher level of translation quality than song translation; (2) we create a dataset musicaltranseval for scoring musical translation, which could serve as a valuable resource for future research; (3) we propose a two-stage translation model training approach that leverages reward models for data filtering and introduces a novel inference-time optimization framework, both aimed at improving translation quality while maintaining satisfactory singability performance. however, this principle is not developed specifically for songs on the musical stage and is not completely suitable for it @xcite . the functional approach @xcite is more suitable for songs in musicals @xcite , which emphasizes that the translated lyrics should replicate the function of the source text. in musicals, songs are "story-telling" elements @xcite , and the translated lyrics must carry out this role @xcite @xcite . thus a basic yet necessary constraint in musical translation is that lyrics must maintain the original narrative function, and thus should have high quality. ## problem formulation we formulate the problem of musical translation as: given a paragraph of english lyrics from a song, the task is to produce a chinese translation that has high translation quality while adhering to singability constraints. by treating each paragraph independently, we can process an entire song. to ensure singability constraints, we consider the following aspects. (1) length: the number of syllables in the english lyrics and the number of characters in the chinese lyrics should match the number of musical notes to ensure proper alignment. since we lack direct access to sheet music but can easily obtain the english lyrics, we use the number of syllables in the english lyrics as the reference length for alignment. (2) rhyme: the translated sentences within each paragraph should maintain the same end rhyme as much as possible, particularly aligning with the end rhyme of the last sentence in each paragraph. to evaluate translation quality, we focus on the following three aspects (yan, 1898). (1) fluency: the naturalness and readability of the translated lyrics in chinese. (2) accuracy: how well the translation conveys the same meaning as the original english lyrics. (3) literary quality: the aesthetic appeal and literary merit of the translated lyrics. we further categorize fluency and accuracy as basic translation quality, while considering literary quality as advanced translation quality, to differentiate between mandatory and supplementary aspects. to enable machines to evaluate these aspects of translation quality, we train reward models using human annotations as learning data. ## method our method consists of three key components: reward models trained to evaluate the quality of the translated language (section 4.1), a translation model trained using a two-stage pipeline (section 4.2), and an inference-time optimization framework that composes sentence-level results into paragraph-level output (section 4.3). figure 2 illustrates how these components work together. ## experiments in our experiments, we investigate the following research questions: rq 1 how well does our method perform in generating high-quality musical lyrics translations, as measured by automatic evaluation metrics? rq 2 how well do the generation results of our method align with human preference? rq 3 how does each component contribute to our performance improvements? ## conclusion in conclusion, our work successfully balances translation quality and singability in musical lyrics translation. to solve this task, we leverage trained reward models, a two-stage translation model training approach, and an inference-time optimization framework. our approach ensures that translated lyrics meet the criteria of fluency, accuracy, and literary quality while adhering to the critical constraints of length and rhyme. the substantial improvements over the baseline, as evidenced by both automatic metrics and human evaluations, demonstrate the efficacy of our method in delivering highquality translations that retain the essence of musical expression. this work paves the way for future advancements in the field, and advances the crosscultural appreciation of musicals.