Paper: WavJourney: Compositional Audio Creation with Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The script writer acts like an agent. There are numerous works and existing methods working on connecting LLM to perform various tasks, e.g., AutoGPT. But the authors didn’t compare against these agent-based system design.

Author response:
We appreciate the reviewer's insightful comments. Existing works, such as AutoGPT, focus on using LLMs as agents to perform various tasks, these typically involve using LLMs as deterministic task solvers (e.g., calling APIs, question answering). To the best of our knowledge, the use of LLM agents as creators of engaging, realistic media content remains largely unexplored. WavJourney specifically addresses this gap. WavJourney is designed for compositional audio creation, engaging LLMs in a specific and complex process. As WavJourney asks LLMs to create scripts with structures *(creative)* that serve for Python code generation *(deterministic)*, which makes direct comparisons between WavJourney and AutoGPT challenging. 

However, in WavJourney, we propose and design a hand-crafted script compiler to generate Python code from audio scripts. This approach is similar to previous work using LLMs to generate code for complex vision-language deterministic tasks via simple in-context prompting [1, 2]. To further validate our design, we conducted additional comparative experiments between a GPT-4-based script compiler and our proposed hand-crafted script compiler (see Appendix A.7.2 in the revision). The results demonstrate that our approach mitigates instability in script compilation and significantly improves computational efficiency. We believe this experiment provides additional insights into system-level design ablation studies.

Additionally, we acknowledge the importance of comparing our approach with several very recent LLM-based video creation agent designs, such as [3, 4]. We plan to explore this in our future research.

[1] Surís, Dídac, Sachit Menon, and Carl Vondrick. "Vipergpt: Visual inference via python execution for reasoning." *arXiv preprint arXiv:2303.08128* (2023).

[2] Gupta, Tanmay, and Aniruddha Kembhavi. "Visual programming: Compositional visual reasoning without training." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.

[3] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." *arXiv preprint arXiv:2309.15091* (2023).

[4] Lian, Long, et al. "LLM-grounded Video Diffusion Models." *arXiv preprint arXiv:2309.17444* (2023).

Reviewer comment (Evaluation):
Based on the text, it is not clear what the basis of the metrics proposed in Audio Storytelling benchmark is. There are no references to any studies from the relevant communities which justify the choice of the specific criteria. I would expect the authors to at least add some information, either in the paper, or an appendix, going into more detail regarding the justification being choosing these metrics.

Author response:
The proposed metrics is inspired by expert-driven metrics design process [1]. Specifically, we develop these metrics with input from audio and product experts. These five metrics (creativity, relevance, emotional resonance, engaging, pace & tempo) allow for an assessment of audio storytelling that goes beyond traditional evaluations (e.g., Mean Opinion Score) to consider narrative complexity, music and sound design, and vocal delivery in a coherent and comprehensive manner. We have added these texts in the Section 4 in the revision.

[1] Shah, J.J., Smith, S.M. and Vargas-Hernandez, N., 2003. Metrics for measuring ideation effectiveness. Design studies, 24(2), pp.111-134.

Reviewer comment (Evaluation):
The result showing WavJourney outputs to be better than AudioCaps GT audio seems a little off to me. It’s not clear what the preference test is focusing on: the overall quality, or the adherence to the caption.

Author response:
The preference test is designed to evaluate the overall impression of the generated audio. This evaluation is supported by evidence from A/B testing [2, 3] a widely accepted method for comparing two versions of a variable to determine which performs better. In our case, the A/B tests consistently showed that listeners preferred audio that not only had high-quality sound but also closely matched the provided caption. 

[2] Siroker, D., & Koomen, P. (2013). A/B testing: the most powerful way to turn clicks into customers. John Wiley & Sons.

[3] A/B Testing: https://en.wikipedia.org/wiki/A/B_testing

Reviewer comment (Evaluation):
The result showing WavJourney outputs to be better than AudioCaps GT audio seems a little off to me. I wonder if the authors ran the GT audio through the same audio quality enhancement model for evaluation.

Author response:
We didn’t run VoiceFixer for GT audios as it will suppress all non-speech sounds.

Reviewer comment (Presentation):
The result showing WavJourney outputs to be better than AudioCaps GT audio seems a little off to me. In table 3, the 6th column seems to have the wrong entry bolded. GT audio has a higher REL score than WavJourney audio.

Author response:
Thanks for your suggestion! We have fixed it in the revision.

Reviewer comment (Evaluation):
The result showing WavJourney outputs to be better than AudioCaps GT audio seems a little off to me. The fact that the results are so one-sided for Clotho dataset gives the impression that the result is not too significant (even though the statement is true), or that there is something wrong in one of the two datasets.

Author response:
We would like to highlight that the improvements achieved by WavJourney over AudioGen are considerable (WavJourney uses AudioGen), as evidenced by both subjective and objective results on the Clotho dataset, and subjective results on the AudioCaps dataset, including the additional subjective tests (ablation studies, Turing test) we conducted, as reported in the Appendix A.5 and A.6 in the revision. 

WavJourney achieved negative objective results on AudioCaps dataset compared with AudioGen. However, we argue that objective evaluation metrics, such as FAD, may not always effectively capture the performance of text-to-audio generation systems. This viewpoint aligns with conclusions drawn in previous work [4], which suggests that systems with good subjective evaluation results may not always achieve high FAD scores.

[4] Choi, Keunwoo, et al. "Foley sound synthesis at the dcase 2023 challenge." *arXiv preprint arXiv:2304.12521* (2023).

Reviewer comment (Evaluation):
Based on the samples shared, I felt all the GT audio to be more natural sounding than WavJourney. I would encourage the authors to also showcase examples where WavJourney was rated higher than GT by the listeners.

Author response:
Thanks for your suggestions! We have uploaded more results to the project page (https://anonymous.4open.science/w/WavJourney_Anonymous/).

Reviewer comment (Evaluation):
I would also be interested in seeing a Turing-test type question asking users if they can tell whether the audio is generated or not.

Author response:
Thanks for your constructive suggestions! It’s interesting to see if WavJourney can generate audio that is indistinguishable from real-world audio in human perception, considering its good results achieved on text-to-audio generation benchmarks. To do so, we conduct both Turing test on AudioCaps and Clotho dataset. To mitigate any potential bias that might skew listeners’ perceptions, we provide human participants with audio clips only (without texts). The experimental setup and results are detailed in the Appendix A.6 in the revision. The conclusion we made is, WavJourney outperforms AudioGen and AudioLDM systems in Turing test score, but can not pass the Turing test, indicating that there is a gap between WavJourney generated synthetic and real audio as judged by human listeners. The discrepancy between the results of the Turing test and other subjective metrics (e.g., OVL, REL, preference-based test) on AudioCaps may be attributed to text-to-speech generated speech. Although this generated speech can positively influence subjective metrics, it may be easily to distinguish due to human's sensitive perception of speech.

Reviewer comment (Experiments):
As an AI audio creation tool, it would be nice to enable the editing of existing audio content. It happens a lot in movie or TV creation. In this case, the framework needs to take the audio as an input as well. However, it seems like the current framework cannot support that.

Author response:
We appreciate your suggestion to enable the editing of existing audio content, a feature that would indeed be valuable in applications such as movie or TV creation. This work focuses on the compositional audio creation from text, which is a novel and relatively unexplored task to the best of our knowledge. We will study the compositional audio editing task in the future.

Reviewer comment (Evaluation):
What is the exact time cost to produce an audio sample?

Author response:
To analyze the time cost, we randomly selected 20 text captions from the AudioCaps test set and calculated the average inference time required by WavJourney and AudioGen to generate 10-second audio clips. For WavJourney, we report the time costs associated with script writing and audio generation. The testing was conducted on a machine equipped with a single NVIDIA GeForce RTX 2080 Ti GPU. The results are shown in Table below and added to the revision. In practice, by carefully optimizing the use of computational resources (e.g., parallel inference) for WavJourney, it is possible to achieve high-quality synthetic audio generation while minimizing the time required for inference, which we leave as future work.

| Model | Script Writing Time (s) | Audio Generation Time (s) |
| --- | --- | --- |
| AudioGen | - | 23.0 |
| WavJourney | 8.1 | 45.3 |

Reviewer comment (Evaluation):
The author points out that WavJourney outperforms previous state of the art methods in both subjective and objective evaluations. However, on the objective indicators of the Audiocaps dataset, WavJourney's performance is lower than AudioGen's.

Author response:
As we described in Section 5.2, objective evaluation metrics (e.g., FAD) may not always be an effective way to evaluate the performance of text-to-audio generation systems. This is consistent with conclusions drawn from previous work, which suggests that some systems with good subjective evaluation results may have poor FAD scores [1]. We believe that the subjective evaluation results detailed in Section 5.2 (OVL, REL, and Preference-based test), as well as the additional results provided in Appendix A.5 and A.6 of the revision (ablation studies and Turing tests), are sufficient to demonstrate that WavJourney outperforms AudioGen on the AudioCaps dataset. Additionally, more examples can be found on the project page at https://anonymous.4open.science/w/WavJourney_Anonymous/.  

[1] Choi, Keunwoo, et al. "Foley sound synthesis at the dcase 2023 challenge." arXiv preprint arXiv:2304.12521 (2023).

Reviewer comment (Experiments):
The experimental part of the article is rather inadequate. The analysis of composite audio generation capabilities of the method is limited to the scene of storytelling, and the analysis of interactive audio creation in multi-round dialogues is also oversimplified, making an inadequate demonstration of the core capabilities of the model.

Author response:
We would like to clarify that both the AudioCaps and Clotho datasets consist of all speech, music, and sound effects elements. Comparative experiments with state-of-the-art end-to-end generative models such as AudioGen and AudioLDM (as well as ground truth audios) on text-to-audio generation tasks also provide valuable insights into the advancement of our proposed compositional approach.  
We have also included additional results (e.g., Turing tests on text-to-audio and system-level ablation studies) in revision. We hope that these additional results and analysis will address your concerns regarding the adequacy of our experiments.

Reviewer comment (Experiments):
More ablation study need to be conducted. Considering that WavJourney uses AudioGen for text-to-audio synthesis, the phenomenon of superiority over AudioGen on objective metrics of the clotho dataset requires further analysis. Is this due to the introduction of a text-to-speech synthesis model by WavJourney, or is it due to WavJourney's use of LLM to improve its understanding of long captions?

Author response:
We would like to thank the reviewer for pointing out the importance of conducting ablation studies to explain the superiority of WavJourney over AudioGen. We have conducted ablation studies on **both the AudioCaps and Clotho datasets**. We manually categorized the test audio clips into two splits: 1) Speech-Inclusive, which includes speech and sounds, and 2) Non-Speech Exclusive, which consists only of non-speech elements. The results from the Non-Speech Exclusive evaluation can tell us if the improvements come from the LLM, while the results from the Speech-Inclusive category can show the combined effectiveness of the LLM and TTS models.  

Detailed results and analysis are provided in Appendix A.5 of the revision. In conclusion, **WavJourney consistently outperforms AudioGen in OVL and REL metrics for both Speech-Inclusive and Non-Speech Exclusive categories on the AudioCaps and Clotho datasets**. More significant enhancements were observed in the Speech-Inclusive category. These results suggest that WavJourney's advancements are derived from **both the LLM and TTS models**.

Reviewer comment (Experiments):
More examples of speech content control should be provided. Whether users can control the speech content in the story? It’s important if you want to involve text-to-speech function. I didn't see any relevant examples of speech content control.

Author response:
We have provided an additional study on a radio-play-based storytelling case focusing on voice content control, as shown in Appendix A.9 in revision. In the example, we can control the content of the speech by "topic" or "specific sentences" and the type of language in the multi-turn conversation.