Synthetic Lyrics Detection Across Languages and Genres (2025)

Abstract: In recent years, the use of large language models (LLMs) to generate music content, particularly lyrics, has gained in popularity. These advances provide valuable tools for artists and enhance their creative processes, but they also raise concerns about copyright violations, consumer satisfaction, and content spamming. Previous research has explored content detection in various domains. However, no work has focused on the text modality, lyrics, in music. To address this gap, we curated a diverse dataset of real and synthetic lyrics from multiple languages, music genres, and artists. The generation pipeline was validated using both humans and automated methods. We performed a thorough evaluation of existing synthetic text detection approaches on lyrics, a previously unexplored data type. We also investigated methods to adapt the best-performing features to lyrics through unsupervised domain adaptation. Following both music and industrial constraints, we examined how well these approaches generalize across languages, scale with data availability, handle multilingual language content, and perform on novel genres in few-shot settings. Our findings show promising results that could inform policy decisions around AI-generated music and enhance transparency for users.

## introduction recent advancements in user-friendly tools, such as suno ai 1 , have significantly impacted the music field by introducing prompt-based interfaces that simplify music generation. in parallel, multiple research works have been exploring audio generation @xcite @xcite or lyrics generation @xcite @xcite with impressive results. llms such as @xcite , mistral 7b @xcite , gemma @xcite , or palm @xcite have demonstrated the ability to generate human-like text without adaptation, being 1 suno.com able to assist artists in tasks such as poem writing @xcite and song lyrics creation @xcite . nevertheless, the widespread use of llms for generating artistic content has raised concerns regarding authorship infringement @xcite , consumer satisfaction @xcite , and content spamming. these concerns outline the need to effectively detect synthetic content to regulate its distribution and prevent misuse. although many methods for synthetic text detection have been proposed and explored @xcite @xcite @xcite @xcite , their effectiveness in detecting ai-generated lyrics as a form of creative content remains unclear. lyrics differ significantly from other text types due to their unique semantics, rhythmic structures, and socio-cultural references @xcite . also, existing detection benchmarks predominantly focus on english, limiting their applicability across languages, and the synthetic text used in these evaluations is often not rigorously validated. to overcome these limitations, we propose the following contributions: data, pre-processing scripts, code, and models will be publicly accessible on github foot_0 under the apache 2.0 license and in compliance with the content copyrights. the detection of machine-generated content has emerged as a well-established research domain @xcite @xcite @xcite @xcite . traditionally, efforts have focused on identifying generated text in areas like news @xcite , scientific writing @xcite , or voice spoofing in audio @xcite . however, recent advances in generative models in terms of quality and creativity have underscored the need for detectors capable of identifying more complex forms of machine-generated text, such as creative content. in music, multiple modalities are vulnerable to ai-generated content, but current efforts have mainly targeted audio detection @xcite @xcite . detection of machine-generated text is typically framed as a binary classification task distinguishing between human-written and synthetic content @xcite . one way of solving it relies on supervised learning, where classifiers are trained based on textual encoders like roberta or longformer @xcite @xcite @xcite or llms @xcite @xcite . this approach requires a sufficiently large training corpus, which is not always available, and may encounter overfitting issues on unseen data, including new authorial styles or generative models @xcite . another line of research has focused on distinguishing between machine-generated and humanwritten texts using various metrics derived from output probabilities of generative models or stylistic features @xcite @xcite @xcite . these methods have been proven effective, while sometimes shown to yield lower performance than the supervised ones depending on the generative model and data @xcite . parallel research has explored watermark-based detection methods @xcite @xcite , but these approaches are limited by the requirement to access model logits, which is not feasible for models accessible only via apis, such as @xcite . as discussed above, previous research has explored content detection across various domains, yet no work has exclusively focused on the text modality, lyrics, in music. moreover, prior benchmarks have primarily targeted english text and often lacked a rigorous validation of the synthetic text used in experiments, raising concerns about the findings' reliability and generalization. these gaps highlight the need for a validated pipeline to generate and refine lyrics, the release of synthetic data that is realistic, musically diverse, and multilingual, and more targeted generalization experiments that explore various factors, including generative models, languages, and writing styles. ## data creation and validation as no prior public studies have addressed the detection of machine-generated lyrics, there is a lack of data reflecting the inherent diversity of song lyrics. to address this gap, we introduce and document the creation of the first lyrics dataset specifically designed for synthetic lyrics detection. this data encompasses a wide variety of artistic styles, music genres, and languages. for generation, we chose to focus on textual input only, excluding lyrics generators that use multiple modalities, such as melody or audio @xcite . likewise, we align with the most widely used tools among content creators, such as suno and chat-gpt, which produce lyrics based entirely on text. ## lyrics detection experiments we approached the detection task as a few-shot prediction using a k-nearest neighbors (k-nn) algorithm on a pre-computed lyrics features space. this method, which works with a limited set of lyrics, supports continuous updates as new synthetic content, including human-flagged material, becomes available. the vector space is constructed using both human-written and machine-generated lyrics, corresponding to our binary classification setup, incorporating multiple features commonly used in text detection (as detailed in section 4.2). during evaluation, we applied a distance-based metric (minkowski) to find the k closest points to the input and assign the most frequent label (with @xmath0. this approach also allowed for better control and explainability by understanding the influence of individual features foot_4 . ## results in the following, we report macro-recall as the primary metric, following @xcite ; @xcite . this ensures a realistic evaluation of the detectors, particularly since black-box models such as human predictors cannot be evaluated using au-roc. the focus is on minimizing false negatives for human-written lyrics and maximizing true positives for synthetic ones to prevent mislabeling. ## conclusion in this work, we presented a diverse dataset of lyrics to evaluate detectors' generalization capabilities. we then conducted a quantitative evaluation over various scenarios to assess detectors' robustness, capabilities to scale, and generalizability across languages and new genres. the results show that our generation pipeline produces lyrics that are very difficult to distinguish by humans from real ones, thus validating it. using automated methods, the detection performance varies greatly depending on the llm used for lyrics generation as well as the type of feature and artistic styles used when building the embedding space. increasing the amount of training data only marginally improves detection performance, whereas expanding the number of languages has a more potent impact; cross-lingual performance of detectors is highly dependent on the source language. we adapted the best-performing features, based on llm2vec, to the distinct features of the lyrics domain via novel unsupervised means, indicating that mntpbased unsupervised domain adaptation improves detection performance. overall, our dataset and detection experiments pave the way for more robust detection of ai-generated music, thereby enabling improved fairness in the music industry.