Paper: Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR-2024)

Reviewer comment (Evaluation):
This model is trained on a very small amount of data, just 909 songs. Despite this, subjectively speaking, the results from the proposed model are quite good. It seems quite plausible that the model is overfit to the training data and producing copies or near-copies. Can the authors provide any evidence that this is not happening?

Author response:
Thank you for pointing out this issue! Since this is really important and is also mentioned by other reviewers, we write a response to all reviewers to answer the question. Please kindly check the **response to all reviewers** for more detail. In short, the current results show that our model does not suffer from plagiarism problem.

Reviewer comment (Evaluation):
Can the authors provide quantitative or qualitative evidence that the model is not overfit to the training data? If so, I would consider raising my score from a 6 to an 8.

Author response:
Thank you for pointing out this issue! Since this is really important and is also mentioned by other reviewers, we write a response to all reviewers to answer the question. Please kindly check the **response to all reviewers** for more detail. In short, the current results show that our model does not suffer from plagiarism problem.

Reviewer comment (Experiments):
I wonder how this method applies to a more general MIDI-like dataset. Or does it rely heavily on the POP909 dataset?

Author response:
### **Generalizability to other MIDI datasets**

Currently, we use POP909 to study whole-song generation because our current focus is on pop songs and the dataset contains relatively accurate annotation. With the help of current MIR algorithms, it is possible to annotate larger MIDI datasets and train our model on these datasets, which is actually our ongoing project. We expect the performance will be better with more training data.

We admit there is a limitation for our current methods that our hierarchical language only works for pop songs (e.g., not for classical) and does not directly work for multi-track music. In our future work, we hope to define more general hierarchical languages to model those types of music.

Reviewer comment (Evaluation):
One thing missing in the evaluation is some nearest neighbor analysis to check if the model is returning part of the training data directly.

Author response:
We indeed see the evaluation of overfitting is crucial. We are conducting a systematic evaluation on the issue. Please see the **response to all reviewers** for more detail. In short, the current results show that our model does not suffer from plagiarism problem.

Reviewer comment (Evaluation):
The evaluation doesn't really measure the capability of whole-song generation, but there is no proper baselines as far as I know, so it's fine.

Author response:
Directly evaluating whole-song generation is hard because 1) the evaluation of longer segments is more subjective; and 2) long music segments take a longer time to listen to and is hard to be memorized. In this paper, we propose to effectively evaluate whole-song generation by decomposing the evaluation problem into structure and quality evaluation:

- Structure: a good structure usually implies clear phrase boundary, similarity between, e.g., two verses, and dissimilarity between, e.g., verse and chorus. These are evaluated by ILS in objective evaluation (Table 2) and “Boundary clarity” & “Phrase similarity” in subjective evaluation (Figure 3c&d).
- Quality: we consider music quality in both short-term (8 measures) and long-term (32 measures) using criteria “Naturalness”, “Creativity” and “Overall musicality” (see Figure 3).

Also, in the evaluation, whole songs are generated under several condensed music form such that the length of the piece is balanced: on one hand, the generated pieces won’t be too long (i.e., within 1’30’’); on the other hand, the music contains all necessary sections with repetition of verses or choruses.

Reviewer comment (Writing):
(Section 1) "and therefore we need to organize various music representations in a structured way." -> I cannot understand this sentence. Why do you mean by "organize representations"?

Author response:
The common sense of music hierarchies are ambiguous. There are a lot of possible music hierarchies (e.g., pitch contour, tonal structure or metrical structure), but these hierarchies are usually not compatible with one another and do not serve for generation purpose. In this paper, we rely on these concepts to define music compositional hierarchy that can directly used in music generation.

Reviewer comment (Writing):
(Section 3.2) "continuous" -> I'm not sure what "continuous" means here. Are you using binary or real-valued piano rolls?

Author response:
The data are represented as real-valued piano-rolls and the generated real-valued piano-rolls will be quantized. The process is the same as Polyffusion [1], where the authors show such quantization operation has almost zero syntax error in practice.

Reviewer comment (Reproducibility):
(Section 3.2) "Both melody reduction and simplified chord progression ..." -> How were these achieved? A pointer to the Appendix would be helpful here.

Author response:
These are computed using Tonal Reduction Algorithm discussed in A. 1. We will add the pointer here.

Reviewer comment (Theory):
(Section 3.2) "13 times" -> Why 13 times? Isn't it 128/12 = 11 times?

Author response:
It should be 11 times. We will fix the typo in our revision.

Reviewer comment (Writing):
(Section 3.3) "We select Sk relevant music segments prior to t based on a defined similarity metric on X<k." -> Is this X^k or X^<k? The descriptions in this paragraph are somewhat confusing. From Figure 1, it seems like it's both X^k and X^<k. Please clarify this.

Author response:
Sorry for the confusion. The term $X^{<k}$ here is correct. We use $X^{<k}$ as the criteria to select segments but embed the corresponding $X^{\leq k}$ as condition. This is not clearly addressed in the submission and we will discuss the procedure in detail in our revision.

Reviewer comment (Reproducibility):
(Algorithm 1) Isn't the song length M also an input?

Author response:
No, it’s not. In the quantization step discussed above, we quantize the generated real-valued piano-rolls. In the training of stage one, the samples of *Form* are zero-padded to 256 measures. So in inference, the model outputs *Form* containing 256 measures and length M is determined by finding the first time step that contains all-zero entries. We will make it clearer in the revision.

Reviewer comment (Experiments):
(Section 5.1) "... and segment them into 8-bar musical segments with a 1-bar hopping size." -> What is this segmentation step for?

Author response:
We apologize for the unclarity. It should be “We segment pieces into 32-bar segments to train the second stage of the model, and segment pieces into 8-bar segments to train the third and forth stage of the model”. We will revise the paper accordingly.

Reviewer comment (Reproducibility):
(Section 5.3) "Using pre-trained VAEs from Yang et al. (2019) and Wang et al. (2020)." -> Are these the models you used to extract autoregressive controls?

Author response:
No. Unlike external condition, for autoregressive condition, we use a trainable encoder consisting of convolutional layers. We will make these details clearer in the revision.

Reviewer comment (Experiments):
(Section 5.3) How did you select the test inputs for the evaluation?

Author response:
All the samples are randomly selected. Samples that have obvious failure will be discarded for both proposed and baseline methods. Such failure happens rarely, e.g., blank for one measure). For 8-measure subjective evaluation, the first two measures are given, and for 32-measure subjective evaluation, the first 4 measures are given. This setting reduce the possible variance in unconditional generation evaluation.

The reviewer also asks why not include ground truth samples in 32-measure subjective evaluation. The reason is that we regard 32-measure as a condensed pop song form (see response to Weakness 3). Such ground truth does not exist in the dataset. In the future, we should consider rewriting a ground truth piece into shorter piece for comparison.

Reviewer comment (Presentation):
(Section 3.1) The musical compositional hierarchy discussed in this paper seems to be constrained to pop music. It would be help to discuss this limitation somewhere in the main text.

Author response:
Thank you for pointing out our current limitation. We admit the current work is limited to pop music and we will put it in our revision. 

On the other hand, high-level form does not need to be given. In the training of stage one, the samples of *Form* are zero-padded to 256 measures. So in inference, the model can generate *Form* containing 256 measures and we derive the song length M by finding the first time step that contains all-zero entries. In section 4, we show music piece of 40 measures under given form for better visualization and space-saving purpose.

Reviewer comment (Presentation):
(Figure 3) It would be great to also include the ground truth for 32-measure generation.

Author response:
All the samples are randomly selected. Samples that have obvious failure will be discarded for both proposed and baseline methods. Such failure happens rarely, e.g., blank for one measure). For 8-measure subjective evaluation, the first two measures are given, and for 32-measure subjective evaluation, the first 4 measures are given. This setting reduce the possible variance in unconditional generation evaluation.

The reviewer also asks why not include ground truth samples in 32-measure subjective evaluation. The reason is that we regard 32-measure as a condensed pop song form (see response to Weakness 3). Such ground truth does not exist in the dataset. In the future, we should consider rewriting a ground truth piece into shorter piece for comparison.