O pen S ep: Leveraging Large Language Models with Textual Inversion for Open World Audio Separation (2024)

Abstract: Audio separation in real-world scenarios, where mixtures contain a variable number of sources, presents significant challenges due to limitations of existing models, such as over-separation, under-separation, and dependence on predefined training sources. We propose OpenSep, a novel framework that leverages large language models (LLMs) for automated audio separation, eliminating the need for manual intervention and overcoming source limitations. OpenSep uses textual inversion to generate captions from audio mixtures with off-the-shelf audio captioning models, effectively parsing the sound sources present. It then employs few-shot LLM prompting to extract detailed audio properties of each parsed source, facilitating separation in unseen mixtures. Additionally, we introduce a multi-level extension of the mix-and-separate training framework to enhance modality alignment by separating single source sounds and mixtures simultaneously. Extensive experiments demonstrate OpenSep’s superiority in precisely separating new, unseen, and variable sources in challenging mixtures, outperforming SOTA baseline methods. Code is released at https://github.com/tanvir-utexas/OpenSep.git.

## introduction audio and music mostly appear in real-world mixtures containing various audio sources and background noise @xcite . separating clean sources from noisy mixtures have numerous applications in audio processing @xcite . precise separation of clean sound sources require their complete semantic understanding, as well as extensive training on natural mixtures. however, in open world scenarios, audio mixtures may contain a variable number of sources, as well as new, unseen, and possibly noisy sources. hence, gathering clean sounds from a variety of sources, and modeling exhaustive mixture combi-figure 1 : unconditional audio separators suffer from both over-separation and under-separation in noisy mixtures, and cannot parse audio entities without additional classifiers. furthermore, conditional separators rely on manual text prompts for source separation, limiting their use in practice. in contrast, opensep fully automates the source parsing and separation flow, even with varying number of unseen and noisy sources in open world. prior work on audio separation mostly focused on two approaches: unconditional and conditional source separation. unconditional separators @xcite ) mostly attempt to disentangle the mixture into a fixed set of output predictions, and later rely on post-processing to select/process the separated sources. however, this approach is largely limited to training sources and mixture combinations, which, in practice, results in over-/under-separation @xcite . in addition, unconditional separators cannot provide the corresponding class entities of separated sources. @xcite introduced source classifiers with audio separators, but their method is limited by training sources only. conditional source separation simplifies the problem by relying on conditional prompts from other modalities/signals, such as text, vision, and sometimes, clean audio representing the target sound @xcite . however, these methods are limited by the availability of target prompts for separation, which are often difficult to gather in practice. moreover, the conditioning signal usually contains simple class instances as prompts which cannot explicitly describe the target audio source. hence, these conditional audio separators are mostly limited to seen sources, often under-performing on unseen, noisy source mixtures in open world. to overcome these limitations, we introduce opensep, a novel framework to automatically separate and parse unseen audio from noisy mixtures with a variable number of sources in open world (fig 1 ). in particular, we propose textual inversion by using an off-the-shelf audio captioning model to extract text representations of noisy mixtures. we also leverage the world knowledge of audio sources in large language models (llms) to automatically parse, disentangle, and extract enriched representations of audio properties of each source present in the mixture. finally, we train a text-conditioned audio separator to extract audio sources from noisy mixtures. for enhancing the modality alignment between conditional prompts and separated sources, we propose a multi-order separation objective by extending the baseline mix-and-separate framework. extensive experiments demonstrates significant performance improvements of opensep over prior work, e.g., opensep achieves +64% and +180% sdr improvements on unseen sources in music and vggsound datasets, respectively, over baseline state-of-the-art models. our contributions are summarized as follows: ## methodology opensep addresses two critical challenges of audio separation: handling a variable number of sources without manual intervention and enhancing performance on unseen sources during inference. as illustrated in fig 2 , the opensep architecture comafterwards, a pre-trained instruction-tuned llm is used to parse audio sources from the caption, followed by the extraction of detailed audio properties of each source. finally, a text-conditional audio-separator is used for separating each audio source from the noisy mixture using the enriched text prompts. here, the audio separator is trained for leveraging detailed audio properties in textual representation. the following sections detail each component of the opensep framework to achieve robust and scalable audio separation. ## limitations opensep performance is limited by the precise detection of sound sources in noisy mixtures, which mostly stems from the challenge in having a suitable audio captioning model. nevertheless, opensep framework can potentially integrate any superior audio captioning approach to scale-up on real world cases. finally, given its use of multiple building blocks, opensep is computationally expensive in general. however, by further optimizing the architecture, such as using mobile llms (e.g., phi-3-mini, gemma-2b), opensep computational cost can be largely reduced, which we leave for future study. ## ethics statement we only use publily available dataset for this study.