Paper: Typing to Listen at the Cocktail Party: Text-Guided Target Speaker Extraction (ICLR-2024)

Reviewer comment (Novelty):
Almost all the previous methods rely on feature embeddings for representing speaker properties to be extracted, which indicates that we do not mind which modality should be used for speaker representations. More specifically, universal sound selector [Ochiai+ Interspeech2022] has almost the same structure as the proposed model presented in Figure 3. This type of integration seems to be natural if we try to feed text prompts into target speaker extraction.

Author response:
Thank you for suggesting the two relevant works. We have added them to the manuscript. 
We appreciate your feedback and acknowledge the challenges in identifying an appropriate baseline for comparison. 

The work [Ochiai] centers on audio event extraction, utilizing labels of each audio event as extraction cues. As previously stated, the foundational framework for target speaker/audio event extraction is well established. This is also grounded in the SpeakerBeam model. The innovation of this research primarily lies in its use of acoustic event classes to extract sound events. However, this model is not directly transferrable to speech. In sound events, all speakers are classified into a single category. In addition, this approach cannot focus on the content of the speech, such as through descriptions embedded in transcription snippets. It also does not accommodate general natural language queries. To establish a comparison with this approach, which employs one-hot representation, we removed data in our dataset associated with extraction through transcription snippets. We only kept the data can be distilled as a one-hot representation to demonstrate the discriminative capabilities of LLMs. The updated experimental results are shown here. 

| Entry   | Input (Audio) | Input (Text) | Transcription Snippet (50%, 80%, 100%) | Gender | Language | Far-near | Loudness |
| ------- | ------------- | ------------ | -------------------------------------- | ------ | -------- | -------- | -------- |
| Universal Sound Selector (Ochiai et al.)       | N             | One-Hot      | No Support                             | 10.54  | 8.88     | 10.25    | 8.96     |
| LLM-TSE | Y             | N            | 7.30                                   | 10. 17 | 8.87     | 9.77     | 7.75     |
| LLM-TSE        | N             | Y            | 2.70, 3.97,7.48                        | 10.40  | 9.38     | 10.57    | 8.89     |
| LLM-TSE        | **Y**             | **Y**            | **7.96,9.81,10.05**                        | **10.87**  | **9.72**     | **10.66**    | **9.41**     |

Noteworthy, however, using one-hot representations introduces the limitations:
- One-hot representations can only express attributes with clear classifications, e.g., language/gender/loudness. However, if we want to use other cues, like transcription snippets, one-hot representations are unable to accomplish this
- One-hot representations lack flexibility. Large language models (LLMs) can assist the target speaker extraction system understand user text inputs, allowing for more generic and diverse expressions. For example, the input of LLM-TSE can be easily extended to open-ended questions, such as "separating the speaker based on the 3-4 second segment in the mixed speech," something that one-hot representations are completely incapable of.

Reviewer comment (Novelty):
ConceptBeam [Ohishi+ ACMMM2022] presented a different network architecture for target speech extraction. Although this work mainly focuses on the use of images for representing the speech property, it can be easily applied to other modalities such as texts.

Author response:
We appreciate that you pointed out ConceptBeam, a system designed to operate in situations where multiple speakers are discussing various semantic concepts or topics within a mixed audio setting. The system leverages related images to the discussed topic to aid speech extraction. While the architecture of this system, which incorporates an encoder, a fusion module, and an extractor, does bear some similarity to our design, the operational conditions differ significantly. It's important to highlight that the functionality of ConceptBeam is contingent upon the availability of pair-matched images that correspond with the semantic content of the speech. This condition isn't typically satisfied in many practical applications. This unique prerequisite significantly constrains the adaptability of the ConceptBeam system when compared with methods that employ registered voices or our proposed text-based approach.

Reviewer comment (Novelty):
Mask generation using embeddings of auxiliary information is a common practice today such as (Liu et al. 2022; Oishi et al. 2023).

Author response:
We appreciate your insightful comments and the opportunity to clarify our work's scope. As our first venture into text-guided target speaker extraction, we aimed to showcase the potential of our proposed framework by examining a wide range of application scenarios and auditory cues. These include absolute attributes (like language and gender), relative attributes (such as loudness and distance), and semantic cues (i.e., transcription snippets). These cues are extensively used by the human auditory system to tackle the cocktail party problem [1].  Text-based systems open up new possibilities and address issues inherent in traditional target speaker extraction systems:

1. Resolving privacy concerns by using text instead of voiceprints.
2. Reducing dependency on the length of registered voices by using text as an extraction cue.
3. Use text can serve as a valuable auxiliary tool to improve the efficacy of target speaker extraction systems, overcoming intra-speaker variability by informing the model about the speaker's current state.

While one-hot representations can handle some of these scenarios, they have notable limitations when dealing with text-based cues. Specifically, one-hot representations are constrained to express attributes with distinct classifications, such as language, gender, and loudness. However, they fall short when encoding more nuanced cues, such as transcription snippets that remain less disrupted in an audio sample.

In contrast, Large Language Models (LLMs) can help the target speaker extraction system understand user text inputs, thereby enabling more generic and diverse text-based cues. For instance, under our proposed framework, the input of LLM-TSE can be easily extended to open-ended questions like "separate the speaker based on the 3-4 second segment in the mixed speech," a task beyond the capabilities of one-hot representations. Similarly, It has the potential to handle complex requests like "I want to extract the speaker who initially spoke loudly about basketball, but later his voice softened, and his speech speed slowed down as he moved further from the microphone." Thus, the proposed LLM-based target speaker extraction framework offers a more adaptable and nuanced approach to speaker extraction, addressing the limitations of existing methods. We believe that our work could have a substantial impact on the field.

Following your question, we added this part (distilling the text prompts into one-hot representations) of the experiment to confirm the LLM's ability to understand the text prompts related to speaker extraction. The integration of the one-hot representation is referenced from the work of Universal Sound Selector (Ochiai et al.) [2]. The performance of the text-based system is comparable to that of the one-hot-based system, but it supports more tasks.

| Entry   | Input (Audio) | Input (Text) | Transcription Snippet (50%, 80%, 100%) | Gender | Language | Far-near | Loudness |
| ------- | ------------- | ------------ | -------------------------------------- | ------ | -------- | -------- | -------- |
| Universal Sound Selector (Ochiai et al.) [2]       | N             | One-Hot      | No Support                             | 10.54  | 8.88     | 10.25    | 8.96     |
| LLM-TSE | Y             | N            | 7.30                                   | 10. 17 | 8.87     | 9.77     | 7.75     |
| LLM-TSE        | N             | Y            | 2.70, 3.97,7.48                        | 10.40  | 9.38     | 10.57    | 8.89     |
| LLM-TSE        | **Y**             | **Y**            | **7.96,9.81,10.05**                        | **10.87**  | **9.72**     | **10.66**    | **9.41**     |

Reviewer comment (Novelty):
What has become fundamentally possible given the power of LLMs?

Author response:
In contrast, Large Language Models (LLMs) can help the target speaker extraction system understand user text inputs, thereby enabling more generic and diverse text-based cues. For instance, under our proposed framework, the input of LLM-TSE can be easily extended to open-ended questions like "separate the speaker based on the 3-4 second segment in the mixed speech," a task beyond the capabilities of one-hot representations. Similarly, It has the potential to handle complex requests like "I want to extract the speaker who initially spoke loudly about basketball, but later his voice softened, and his speech speed slowed down as he moved further from the microphone." Thus, the proposed LLM-based target speaker extraction framework offers a more adaptable and nuanced approach to speaker extraction, addressing the limitations of existing methods.

Reviewer comment (Experiments):
If we restrict some form of templates of prompts to feed into the LLM for a handful of separation scenarios to specify gender/language/loudness etc., such information may be provided as one-hot representations.

Author response:
While one-hot representations can handle some of these scenarios, they have notable limitations when dealing with text-based cues. Specifically, one-hot representations are constrained to express attributes with distinct classifications, such as language, gender, and loudness. However, they fall short when encoding more nuanced cues, such as transcription snippets that remain less disrupted in an audio sample.

In contrast, Large Language Models (LLMs) can help the target speaker extraction system understand user text inputs, thereby enabling more generic and diverse text-based cues. For instance, under our proposed framework, the input of LLM-TSE can be easily extended to open-ended questions like "separate the speaker based on the 3-4 second segment in the mixed speech," a task beyond the capabilities of one-hot representations. Similarly, It has the potential to handle complex requests like "I want to extract the speaker who initially spoke loudly about basketball, but later his voice softened, and his speech speed slowed down as he moved further from the microphone." Thus, the proposed LLM-based target speaker extraction framework offers a more adaptable and nuanced approach to speaker extraction, addressing the limitations of existing methods.

Following your question, we added this part (distilling the text prompts into one-hot representations) of the experiment to confirm the LLM's ability to understand the text prompts related to speaker extraction. The integration of the one-hot representation is referenced from the work of Universal Sound Selector (Ochiai et al.) [2]. The performance of the text-based system is comparable to that of the one-hot-based system, but it supports more tasks.

Reviewer comment (Experiments):
The ablation study in Table 1 shows some variations of LLM-TSE: audio + text, text-only and audio-only. Can we train the model with audio + text and switch to text-only or audio-only in inference time? It seems the embedding vectors are concatenated and used for mask generation. In this case, I am wondering how the lack of either modality is handled.

Author response:
Thank you for your question. Our model can handle the scenarios where either modality is missing. Specifically, our proposed LLM-TSE model uses a combination of audio+text (50%), text-only (20%), and audio-only (30%) data for training. After the model had been trained, we tested it across three scenarios: audio-only, text-only, and audio+text, as shown in Table 1.  This presents a unique advantage of our system, as the quality of the registered audio might be poor or missing due to privacy concerns. In these cases, the proposed LLM-TSE model can provide additional text-based cues to augment the performance of target speaker extraction.

Reviewer comment (Novelty):
How does audio-only LLM-TSE differ from TD-SpeakerBeam?

Author response:
For a fair comparison, our audio-only LLM-TSE is the same as TD-SpeakerBeam, and we have its open-sourced code from https://github.com/BUTSpeechFIT/speakerbeam. We would like to highlight that instead of improving the audio TSE model, our main objective in this work lies in incorporating text-based cues to address the issues of the target speaker extraction system.

Reviewer comment (Experiments):
The experiment design is limited in the sense that these text prompt such as gender, language, far-near and loudness, are not typical cues which can be widely used for target speaker extraction. For example, if all the speakers are of the same gender in a conversation, which target speaker will be extracted? If all speakers speak the same language, how to extract the target speaker? For these scenarios, speakers are of different genders or speaking different languages, the target speaker extraction problem is indeed an easy problem. A more practical design scheme is needed for the input text.

Author response:
Thank you for your insightful question, which allows us to clarify further the scope and potential of our proposed framework for real-world applications. As an initial venture into text-guided target speaker extraction, we aimed for a broad examination of application scenarios and auditory cues. Specifically, we scrutinized absolute attributes (like language and gender), relative attributes (such as loudness and distance), and semantic content (via transcription snippets). These are all extensively utilized by the human auditory system to tackle the cocktail party problem [4, 5].

In real-world scenarios, when one type of auditory cue is insufficient to discern a sound source, humans naturally shift their focus to other cues. Our LLM-TSE model is motivated by this adaptive behavior. It is designed as a versatile text-based speaker extraction framework, capable of flexibly choosing and smoothly transitioning between auxiliary cues based on different situations.

However, we agree with your assertion that refining the text prompts to cover more practical scenarios carries great significance. This indeed calls for substantial research efforts to design comprehensive evaluation benchmarks. As stated in our conclusion, we are actively integrating additional cues such as timbre, emotional state, sentiment scoring, speech velocity (calculated using Whisper timestamps), pitch, and energy (evaluated with Librosa). These improvements aim to enhance the applicability of our framework further. We look forward to sharing our progress in future work.

Reviewer comment (Experiments):
For the audio cue encoder, have you tried a pre-trained speaker embedding module, such as d-vector model used for speaker recognition?

Author response:
Thank you for your valuable question. We have not yet experimented with pre-trained speaker embedding modules, such as d-vector or x-vector, as our audio cue encoder. While neural network-based embeddings like x-vectors or d-vectors are designed and trained specifically for speaker classification tasks and indeed contain speaker-specific information, it remains uncertain whether these representations are optimal for TSE tasks [6]. In our work, we've chosen a more common setting. Our audio cue encoder, which performs speaker embedding extraction, takes an enrollment utterance as input. It typically includes a pooling layer that converts frame-level features into a single vector, mirroring the functionality of the aforementioned embedding extractors. This neural network is trained concurrently with the main network using a shared objective function. The benefit of this approach is that the embeddings are trained explicitly for TSE, thus gathering crucial information specifically for this task. Contrastingly, pre-trained embedding extractors like d/x-vectors are often trained on larger corpora, potentially offering greater robustness. A potential compromise might involve utilizing a pre-trained embedding extractor and fine-tuning it with the TSE task. To our knowledge, this approach has not been explored yet. It would be interesting to explore these pre-trained speaker embeddings, which can potentially offer greater robustness as they have been trained on larger corpora. Due to the time limitation during the rebuttal, we will leave this study as a future work.