A Data-Driven Method for Analyzing and Quantifying Lyrics-Dance Motion Relationships (2025)

Abstract: Dancing to music with lyrics is a popular form of expression. While it is generally accepted that there are relationships between lyrics and dance motions, previous studies have not explored these relationships. A major challenge is that the relationships between lyrics and dance motions are not constant throughout a song but are instead localized to specific parts. To address this challenge, we hypothesize that lyrics and dance motions that co-occur across multiple songs are related. Based on this hypothesis, we propose a novel data-driven method to detect the parts of songs where meaningful relationships between lyrics and dance motions exist. We use clustering to transform lyrics and dance motions into symbols, enabling the calculation of co-occurrence frequencies and detection of significant correlations. The effectiveness of our method is validated by a dataset of time-synchronized lyrics and dance motions, which showed high correlation values for emotionally salient lyrics such as “love”, which is expressed in heart-shaped motions. Furthermore, using our relationship detection method, we propose a method for retrieving dance motions from lyrics that outperforms previous text-to-motion retrieval methods, which focus on prose and non-dance motions.

## introduction platforms like youtube and tiktok have popularized dance videos, allowing amateur dancers to express their creativity, but choreographing these performances is challenging for those without professional training. consequently, previous studies have focused on synthesizing or retrieving dance motions that match the rhythms and timbres of music @xcite . however, choreographic design is influenced by both audio attributes and lyrics, as dancers often interpret lyrics literally, such as by making heart shapes to express "love". however, no empirical studies have quantitatively analyzed the relationships between lyrics and dance, and no studies have specifically focused on retrieving dance motions associated with lyrics. quantitative analysis of lyrics and dance motions offers two main contributions. academically, this analysis provides clear evidence of how lyrics and dance motions are related. practically, it enables the development of systems that suggest dance motions based on user-input lyrics, offering choreographic ideas to support dancers, particularly amateurs, in creating or enhancing their choreography. previous studies have quantified the relationship between prose and corresponding non-dance gestures (e.g., "a man walks in a quarter circle to the left") @xcite @xcite . these methods assume that text and motion are analyzed in units of sentences and sequences, respectively, and that all text-motion pairs are inherently related. however, these methods fall short for analyzing interactions between lyrics and dance motions, because those interactions are often localized to specific parts of songs rather than uniformly distributed across the entire song. for example, functional words like "the" or "is" may not correspond to any dance motion, while emotionally charged words like "love" might correlate with specific motions. to analyze lyrics and dance motions, we need a method that can detect which frames foot_0 have motions related to lyrics. this study proposes a novel data-driven method to analyze and quantify the relationship between lyrics and dance motions as illustrated in figure 1 . we hypothesize that lyrics and dance motions that co-occur in different songs are related. for example, if "jump" is sung while a jumping motion is performed in multiple songs, the co-occurrence suggests a meaningful relationship. to quantify these relationships, we first transform lyrics and ## analyzing lyrics and dance motion relationships in this section, we present a method for detecting frames that demonstrate meaningful relationships between lyrics and dance motions within our timesynchronized data. based on our hypothesis that the frequent co-occurrence of lyrics and dance motions across multiple songs indicates meaningful relationships, we use co-occurrence frequencies to quantitatively analyze these interactions. to analyze the co-occurrence relationships at each frame, we utilize pmi, a metric used in natural language processing. pmi is calculated with the formula: to investigate the effectiveness of our method, we have prepared a novel dataset of timesynchronized lyrics and dance motions, including the dancer's finger motions. applying our method to this dataset, we have detected frames with high pmi values between lyrics and dance motions. for example, the lyric "love" corresponds to dance motions that form heart shapes with the hands, and the phrase "getting lost" is associated with motions that suggest searching for a destination. our proposed method therefore makes it possible to quantify and analyze specific relationships between lyrics and dance motions, revealing valuable correlations. in addition, we have developed a lyrics-to-dance motion retrieval method by utilizing our method to detect relationships between lyrics and dance motions. this method outperforms an existing text-tomotion retrieval method based on contrastive learning, which assumes that all text-motion pairs are inherently related. unlike this general method that focuses on text and non-dance motions, our method specifically targets lyrics and dance motions that frequently co-occur across multiple songs, indicating a meaningful relationship. our method over-comes the limitations observed in broader text-tomotion studies and improves retrieval performance, providing an effective solution in this area. ## time-synchronized lyrics and dance motion pair data our goal is to analyze and quantify the relationship between lyrics and dance motions. to achieve this, we need data where each frame of dance motion is associated with specific words or sentences from the corresponding lyrics. ## analyzing lyrics and dance motion relationships in this section we present a method for detecting frames that demonstrate meaningful relationships between lyrics and dance motions within our timesynchronized data. based on our hypothesis that the frequent co-occurrence of lyrics and dance motions across multiple songs indicates meaningful relationships, we use co-occurrence frequencies to quantitatively analyze these interactions. to analyze the co-occurrence relationships at each frame, we utilize pmi, a metric used in nlp. pmi is calculated with the formula high pmi values at a frame indicate a strong association between the lyrics and dance motions at that particular moment. to apply pmi, we convert lyrics and dance motions into discrete symbols x and y for each frame. this involves transforming feature vectors, derived from deep learning models, into a form suitable for pmi analysis. using a clustering method, we categorize similar motions and lyrical expressions into codebooks x and y, which serve as the basis for calculating pmi, enabling us to detect frames where lyrics and dance motions are closely related. the process begins with extracting feature vectors for lyrics and dance motions by using deep learning models. we then implement clustering to group similar lyrics and dance motions, facilitating the pmi calculations for our analysis. ## lyrics-to-dance motion retrieval we developed a method that allows input of a single musical bar of lyrics to retrieve the corresponding bar of dance motions. the input is a sequence of lyric words within a musical bar. the retrieved output is a ranked list of musical bars containing the corresponding motions. for the retrieval task, we use dynamic time warping (dtw) @xcite to measure the similarity between the input lyrics and available dance motions. dtw is ideal for handling time series data of varying lengths and for capturing partial frame similarities, thereby accounting for how lyrics and dance motions relate within a bar. in our dtw implementation, we derive the cost matrix from the normalized pointwise mutual information (npmi) between the lyric and dance motion codebooks. the substitution (match) cost between elements x and y is defined as this formula assigns lower costs when there is a stronger relationship between x and y (i.e., higher npmi values) and assigns higher costs when the relationship is weaker. for insertions and deletions, we assign a fixed cost of 1. this means that when aligning sequences, inserting or deleting an element incurs a constant penalty, regardless of the specific elements involved. this use of a fixed cost was based on preliminary experiments showing that using variable costs degraded performance in the retrieval task. we implemented two retrieval strategies: wordto-dance motion retrieval (w2d) based on the word-to-motion npmi(x w , y) and sentence-todance motion retrieval (s2d) based on the sentenceto-motion npmi(x s , y). in this section, our dataset comprised 868 songs and was divided into 78,875 bars (85%) for training, 5,237 bars (5%) for development, and 8,611 bars (10%) for testing. unlike section 4 where models were trained on the full dataset, here we specifically trained the motion encoder and decoder, and conducted k-means clustering from scratch using only the training subset. in training the motion encoder, we utilized early stopping based on the development subset, ensuring precision in model tuning. we confined k-means clustering and npmi calculations strictly to the training data, guaranteeing that these models were accurately calibrated for the specific tasks described in this section. additionally, we adjusted the codebook sizes for k-means clustering, with optimal sizes of 6000 for lyrics and 7000 for motions in the w2d method, and 3500 for lyrics and 7000 for motions in the s2d method. as there are no existing methods specifically designed for lyrics-to-dance motion retrieval, to evaluate our lyrics-to-dance motion retrieval method we devised a baseline using contrastive learning techniques adapted from text-to-motion retrieval studies @xcite @xcite . additionally, we included a basic random selection method that randomly selects dance motions from the test set. for the contrastive learning method, sentence vectors from a bar, generated by pre-trained sentence-bert, were first averaged into a 384-dimensional vector and then compressed to a 256-dimensional vector using an mlp. dance motion feature vectors from our pre-trained motion encoder were similarly processed into a sequence of 256-dimensional vectors by an mlp. those vectors were then averaged into a single 256-dimensional vector via mean pooling. we fixed the parameters for sentence-bert and the motion encoder, focusing training on the mlp parameters using a contrastive loss function to distinguish between matching and non-matching pairs. training employed the adamw optimizer with a mini-batch size of 16 over 200 epochs, with early stopping triggered after 10 epochs without improvement in development loss. the experimental results presented in ## conclusion and future work this paper introduces a novel method for quantifying the relationship between lyrics and dance motions that uses co-occurrence frequency. our method effectively detects where lyrics correlate with dance motions and was validated by identifying meaningful relationships, such as the association between "love" and heart-shaped motions, and outperforming a previous method in a lyricsto-dance motion retrieval task. the success of this method not only impacts fields such as lyrics information processing, dance information processing, music information retrieval, and computer vision but also opens promising avenues for interdisciplinary studies and enhances the integration of text content into choreographic design. future work will aim to integrate our method into existing audio-to-dance motion retrieval methods to improve their accuracy in matching dance motions with lyrics.