REFFLY : Melody-Constrained Lyrics Editing Model (2025)

Abstract: Automatic melody-to-lyric (M2L) generation aims to create lyrics that align with a given melody. While most previous approaches generate lyrics from scratch, revision—editing plain text draft to fit it into the melody—offers a much more flexible and practical alternative. This enables broad applications, such as generating lyrics from flexible inputs (keywords, themes, or full text that needs refining to be singable), song translation (preserving meaning across languages while keeping the melody intact), or style transfer (adapting lyrics to different genres). This paper introduces REFFLY (REvision Framework For LYrics), the first revision framework for editing and generating melody-aligned lyrics. We train the lyric revision module using our curated synthesized melody-aligned lyrics dataset, enabling it to transform plain text into lyrics that align with a given melody. To further enhance the revision ability, we propose training-free heuristics aimed at preserving both semantic meaning and musical consistency throughout the editing process. Experimental results demonstrate the effectiveness of REFFLY across various tasks (e.g. song translation), showing that our model outperforms strong baselines, including Lyra (CITATION) and GPT-4, by 25% in both musicality and text quality.

## introduction music acts as an important universal language that facilitates social connection and strengthens community involvement @xcite . automatic melody-to-lyric (m2l), creating lyrics that are aligned with a given melody, has emerged as a promising task and received interest by the ai community, because it makes the process of music creation more accessible to a wider audience. @xcite @xcite . in practice, amateur songwriters may wish to craft lyrics that goes with their favorite melody figure 1 : human singers naturally emphasize certain words when singing, which align with prominent notes to ensure musical flow (details in § 3.1). however, llms like gpt-4 often misalign these prominent words (e.g."arm" with non-prominent notes) or omit syllables (e.g. no words for the last four notes), lowering lyric quality( § 2). our model, reffly, refines less singable drafts into melodically-aligned lyrics while preserving the meaning. listen to the audios for an intuitive sense. with desired content @xcite , or translate songs into different languages for a wider audience or adapt existing lyrics to fit a different melody @xcite . however, existing m2l approaches and ai-assisted songwriting frameworks fall short to support these use cases, due to insufficient control over sentence-level semantic. most prior works generate lyrics with zero or limited user input such as keywords or topics @xcite @xcite . some prior works rely on in-filling text templates without providing sufficient automacy to the user @xcite . in contrast, our revision framework refining plain text into melody-aligned lyrics offers greater flexibility and control. in addition, both state-of-the-art llms and prior works on lyrics generation struggle with producing singable lyrics that align well with a specific melody. for instance, as illustrated in figure 1 , @xcite generates coherent lyrics but fails to synchronize syllables with the last four music notes. moreover, prominent words like 'arm' and 'eye' are paired with less prominent notes, disrupting the overall musical flow and resulting in low prosody. similarly, prior works on lyrics generation generation either don't consider melody as a constraint @xcite @xcite @xcite , or overlooked the important relationship between prominent note in melody and prominent lyric words @xcite @xcite , lowering the generated lyric quality. addressing these challenges, we propose a novel revision framework, reffly, which transfers a draft prose to structured and singable lyrics align with a piece of melody. to enhance melody-lyric alignment, we develop a training-free heuristic for capturing prominent lyrical words and musical notes ( §3.1). since the melody-aligned lyric data is scarce due to copyright constraints, we design an instruction-based mechanism to guide llms towards highly singable lyrics by training on a synthetic dataset ( § 4.1). reffly can generate fulllength songs with lyrical verses that develop the song's plot and message, and choruses that repeat a memorable musical motif. our contributions are summarized as follows: • we propose the first melody-constrained lyric revision framework that, given a predefined melody, transfers an arbitrary text (also referred to as a draft or unsingable lyrics) to a full-length, melody-aligned lyrics with high singability and prosody (also referred to as revised or singable lyrics), with sentence-level semantic control. ## a revision framework for lyric generation figure 2 illustrates the inference process of reffly. to manage the complexity of lyric revision, the revision process is conducted at the sentence level. we iteratively revise each sentence from the unsingable draft to lyric that fits the melody, aligning the prominent words with prominent notes while maintaining the overall coherence. in this section, we detail each component of reffly. we develop a training-free heuristic for capturing prominent lyrical words and musical notes (section §3.1). then, §3.2 introduces our lyrics revision module that refines unsingable drafts based on musical constraints. last, §3.3 provides an overview of the inference process to achieve optimal lyric-melody alignment. ## experiments setup in this section, we introduce dataset setup, experiment setting, baseline models, and evaluation setup. ## results in this section, we present results for validation of proposed heuristic, automatic evaluation, human evaluation, and ablation studies. ## conclusion we introduced reffly, the first melody constrained revision framework to generate highquality lyrics from plain text drafts while retaining original meaning. to enhance the lyrics-melody alignment, we designed a heuristic to identify and align prominent notes and words. finally, we show reffly excel across diverse applicability.