Paper: MuseCoco: Generating Symbolic Music from Text (ICLR-2024)

Reviewer comment (Writing):
The paper writing can be improved. The passage is usually wordy and the word choice is not precise. Also, long sentences are often used which contains a lot of unnecessary logic twists. (For example, the “three advantages” part presented in the abstract section is wordy and includes long sentences.) Word spelling is another problem, for example “two-stage design”. The overall logic flow is okay to help readers capture the idea of the proposed method. A better organization at local level can help readers to catch the insight in a smoother way.

Author response:
Thank you for your guidance. We appreciate your input and we enhance the wording for the abstract part in the updated version for conciseness. Regarding the term "two-stage design," we have confirmed its grammatical correctness through dictionary research. However, we are open to specific suggestions on how to improve its usage. Your additional insights would be valuable. Thank you.

Reviewer comment (Theory):
In our current standing, the problem of text-to-music is unsolved and one probably cannot conclude text-to-music can be perfectly decomposed into two stages with “attributes” as the intermediate representation. This is because sometimes word meanings are vague and the implication to attributes might not be this direct. The proposed method does not solve this problem scientifically but proposes a practical solution for this. To this end, the paper possibly overclaim the contribution in the understanding/modeling of the text-to-music problem.

Author response:
In the formulation of attribute conditions, we meticulously incorporated a spectrum of explicit and nuanced descriptors to accommodate the varied preferences of users. For proficient musicians, the inclusion of precise technical music terms optimizes efficiency, ensuring the model generates music precisely in line with their articulated preferences. Conversely, users lacking a musical background often articulate their ideas using more abstract language to convey emotions or stylistic preferences. Consequently, we employ objective attributes to streamline the music creation process for musicians, supplementing them with subjective elements like emotion, artistic style, and genres. This approach facilitates expressive music generation, catering to users with diverse backgrounds and preferences.  
As you highlighted, acknowledging the crucial role of subjective elements in music generation involves describing a more abstract representation, we incorporate artist, genre, and emotion—elements commonly featured in music retrieval projects [2,3] and previous controlled music generation endeavors that incorporated subjective attributes [4-7].  

However, your insight into the exploration of more obscure descriptions adds an intriguing dimension to our approach. Our framework is intentionally designed for expansibility, allowing for the integration of captivating and diverse descriptions, as per your suggestion. In response to your recommendations, we undertook the creation of an additional dataset by employing ChatGPT to refine the text-to-attribute understanding stage. During this phase, we directed ChatGPT to emulate the perspective of a musician, randomly selecting attributes and their corresponding values (which could also be user-provided). Following this, we tasked ChatGPT, now assuming the role of a user without a musical background, to utilize these chosen values in crafting descriptions enriched with more abstract language. To ensure coherence, we instructed ChatGPT to provide an explanation of how each description aligns with the selected values. This augmentation and examples are presented comprehensively in Appendix D.  

[1] https://jmir.sourceforge.net/jSymbolic.html  

[2] https://mubert.com/render/themes  

[3] https://open.spotify.com/search  

[4] Hung, Hsiao-Tzu, et al. "Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation." arXiv preprint arXiv:2108.01374 (2021).  

[5] L. N. Ferreira, L. Mou, J. Whitehead, and L. H. Lelis, “Controlling perceived emotion in symbolic music generation with monte carlo tree search,” in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 18, no. 1, 2022, pp. 163– 371 170.  

[6] C. Bao and Q. Sun, “Generating music with emotions,” IEEE Transactions on Multimedia, 2022.  

[7] H. H. Mao, T. Shin, and G. Cottrell, “Deepj: Style-specific music generation,” in 2018 IEEE 12th International Conference on Semantic Computing (ICSC). IEEE, 2018, pp. 377–382.

Reviewer comment (Experiments):
Based on the previous point, the comparison with other end-to-end text-to-music models can be questionable. The two-stage model is likely to be superior in the cases where a given text prompt indicates clear attribute assignment. But what about the cases where the text prompt is abstract? The paper will be stronger if some demo of abstract text prompt is presented.

Author response:
In the formulation of attribute conditions, we meticulously incorporated a spectrum of explicit and nuanced descriptors to accommodate the varied preferences of users. For proficient musicians, the inclusion of precise technical music terms optimizes efficiency, ensuring the model generates music precisely in line with their articulated preferences. Conversely, users lacking a musical background often articulate their ideas using more abstract language to convey emotions or stylistic preferences. Consequently, we employ objective attributes to streamline the music creation process for musicians, supplementing them with subjective elements like emotion, artistic style, and genres. This approach facilitates expressive music generation, catering to users with diverse backgrounds and preferences.  
As you highlighted, acknowledging the crucial role of subjective elements in music generation involves describing a more abstract representation, we incorporate artist, genre, and emotion—elements commonly featured in music retrieval projects [2,3] and previous controlled music generation endeavors that incorporated subjective attributes [4-7].  

However, your insight into the exploration of more obscure descriptions adds an intriguing dimension to our approach. Our framework is intentionally designed for expansibility, allowing for the integration of captivating and diverse descriptions, as per your suggestion. In response to your recommendations, we undertook the creation of an additional dataset by employing ChatGPT to refine the text-to-attribute understanding stage. During this phase, we directed ChatGPT to emulate the perspective of a musician, randomly selecting attributes and their corresponding values (which could also be user-provided). Following this, we tasked ChatGPT, now assuming the role of a user without a musical background, to utilize these chosen values in crafting descriptions enriched with more abstract language. To ensure coherence, we instructed ChatGPT to provide an explanation of how each description aligns with the selected values. This augmentation and examples are presented comprehensively in Appendix D.  

[1] https://jmir.sourceforge.net/jSymbolic.html  

[2] https://mubert.com/render/themes  

[3] https://open.spotify.com/search  

[4] Hung, Hsiao-Tzu, et al. "Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation." arXiv preprint arXiv:2108.01374 (2021).  

[5] L. N. Ferreira, L. Mou, J. Whitehead, and L. H. Lelis, “Controlling perceived emotion in symbolic music generation with monte carlo tree search,” in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 18, no. 1, 2022, pp. 163– 371 170.  

[6] C. Bao and Q. Sun, “Generating music with emotions,” IEEE Transactions on Multimedia, 2022.  

[7] H. H. Mao, T. Shin, and G. Cottrell, “Deepj: Style-specific music generation,” in 2018 IEEE 12th International Conference on Semantic Computing (ICSC). IEEE, 2018, pp. 377–382.

Reviewer comment (Theory):
In section 3.1, does position encoding not included for the m classes? Under the bi-directional BERT design, the model cannot differentiate the m classes if no position encoding is provided. For example, [cls1][cls2][x1]…[xn] and [cls2][cls1][x1]…[xn] yield the same output.

Author response:
In BERT pre-training, the first position is the first token of the input text rather than the $[CLS]$prepended to the input text. Therefore, to unleash the power of BERT by aligning with pre-training, we also assigned the first text token as the first position and did **not** compute the position embedding of $[CLS_i]^m_{i=1}$.  We prepend $[CLS_i]^m_{i=1}$ to the text after the text is inputted to BERT, instead of concatenating them before the input action. We separately calculate the embedding of $[CLS_i]^m_{i=1}$ (the word embedding and token type embedding) and the text (the word embedding, position embedding and token type embedding), and then **concatenate** the $[CLS_i]^m_{i=1}$ embedding and the text embedding. This implementation makes sure the order of $[CLS_i]^m_{i=1}$ is unchangeable. We provide the code about this in the line 244-253 of `code_submit/1-text2attribute_model/text-attribute_understanding/bert/modeling_bert.py`.

Reviewer comment (Evaluation):
When the text prompt is vague, how does the model behave?

Author response:
In the formulation of attribute conditions, we meticulously incorporated a spectrum of explicit and nuanced descriptors to accommodate the varied preferences of users. For proficient musicians, the inclusion of precise technical music terms optimizes efficiency, ensuring the model generates music precisely in line with their articulated preferences. Conversely, users lacking a musical background often articulate their ideas using more abstract language to convey emotions or stylistic preferences. Consequently, we employ objective attributes to streamline the music creation process for musicians, supplementing them with subjective elements like emotion, artistic style, and genres. This approach facilitates expressive music generation, catering to users with diverse backgrounds and preferences.  
As you highlighted, acknowledging the crucial role of subjective elements in music generation involves describing a more abstract representation, we incorporate artist, genre, and emotion—elements commonly featured in music retrieval projects [2,3] and previous controlled music generation endeavors that incorporated subjective attributes [4-7].  

However, your insight into the exploration of more obscure descriptions adds an intriguing dimension to our approach. Our framework is intentionally designed for expansibility, allowing for the integration of captivating and diverse descriptions, as per your suggestion. In response to your recommendations, we undertook the creation of an additional dataset by employing ChatGPT to refine the text-to-attribute understanding stage. During this phase, we directed ChatGPT to emulate the perspective of a musician, randomly selecting attributes and their corresponding values (which could also be user-provided). Following this, we tasked ChatGPT, now assuming the role of a user without a musical background, to utilize these chosen values in crafting descriptions enriched with more abstract language. To ensure coherence, we instructed ChatGPT to provide an explanation of how each description aligns with the selected values. This augmentation and examples are presented comprehensively in Appendix D.  

[1] https://jmir.sourceforge.net/jSymbolic.html  

[2] https://mubert.com/render/themes  

[3] https://open.spotify.com/search  

[4] Hung, Hsiao-Tzu, et al. "Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation." arXiv preprint arXiv:2108.01374 (2021).  

[5] L. N. Ferreira, L. Mou, J. Whitehead, and L. H. Lelis, “Controlling perceived emotion in symbolic music generation with monte carlo tree search,” in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 18, no. 1, 2022, pp. 163– 371 170.  

[6] C. Bao and Q. Sun, “Generating music with emotions,” IEEE Transactions on Multimedia, 2022.  

[7] H. H. Mao, T. Shin, and G. Cottrell, “Deepj: Style-specific music generation,” in 2018 IEEE 12th International Conference on Semantic Computing (ICSC). IEEE, 2018, pp. 377–382.

Reviewer comment (Novelty):
While text-to-music generation in the symbolic domain is correctly assessed as a relatively young field of inquiry, by breaking down the task into text-to-attribute and attribute-to-music generation, the authors omit a large body of work in the generative music space that has dealt with attribute-conditioned generation, albeit in a more limited capacity. Of relevance:  
[1] Rhythmic intensity, polyphonic density  
[2] Harmonic information, melodic information  
[3] Instrument information  
[4] Harmonic information, tempo

Author response:
Thank you for your valuable suggestions. Firstly, MuseCoco significantly focuses on overcoming challenges in generating symbolic music from text descriptions. Consequently, a thorough comparison with methods specifically designed for this scenario is essential, and we have already undertaken this in Table 4.

Secondly, as attribute-to-music generation constitutes a crucial module in our system, it becomes imperative to conduct an ablation study to validate its effectiveness, as depicted in Table 5. In our analysis, we compare our control method labeled as "Prefix Token" with other widely used methods such as "Embedding" and "Layer Norm." The results reveal that in a multi-attribute control scenario, the "Prefix Token" method demonstrates significantly superior controllability, manifesting at least a 20% improvement.

The paper you referenced is indeed valuable for a thorough comparison to further validate our method. However, it's crucial to note that it falls within the category of methods we are already comparing. For instance, in [1], the conversion of segment-level conditions into an embedding vector, followed by concatenation with token embeddings at each time step, aligns with what we denoted as the "Embedding" method in Table 5.

Additionally, three other Works[2-4], namely "Compose & Embellish," "Multitrack Music Transformer," and "Compound Word Transformer," share similarities with the "Prefix Tokens" method employed in our approach. This similarity lies in the explicit representation of condition tokens in the input sequence instead of condensing them into a vector. Notably, this architecture is widely employed for decoder-only generation tasks, akin to Prefix language modeling [5].

We appreciate your observation regarding missing references and will promptly address this in Section 3.2 and Section 4.3.2 in the updated version. Thank you for bringing it to our attention.

[1] Wu, S. L., & Yang, Y. H. (2021). MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE.

[2] Wu, S. L., & Yang, Y. H. (2023). Compose & Embellish: Well-structured piano performance generation via a two-stage approach.

[3] Dong, H. W., Chen, K., Dubnov, S., McAuley, J., & Berg-Kirkpatrick, T. (2022). Multitrack Music Transformer.

[4] Hsiao, W.-Y., Liu, J.-Y., Yeh, Y.-C., & Yang, Y.-H. (2021). Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs.

[5] Wang, Thomas, et al. "What language model architecture and pretraining objective works best for zero-shot generalization?." International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Novelty):
While the proposed system encapsulates multiple control signals that the aforementioned works only are able to use a subset of, the fact that none of these systems were used for even targeted comparisons over the control methods they share calls into question the novelty of the present work. It would be useful to contrast how the present work is sufficiently different from these previous papers, and to show whether the proposed method can beat prior works on the attributes they can condition on.

Author response:
Thank you for your valuable suggestions. Firstly, MuseCoco significantly focuses on overcoming challenges in generating symbolic music from text descriptions. Consequently, a thorough comparison with methods specifically designed for this scenario is essential, and we have already undertaken this in Table 4.

Secondly, as attribute-to-music generation constitutes a crucial module in our system, it becomes imperative to conduct an ablation study to validate its effectiveness, as depicted in Table 5. In our analysis, we compare our control method labeled as "Prefix Token" with other widely used methods such as "Embedding" and "Layer Norm." The results reveal that in a multi-attribute control scenario, the "Prefix Token" method demonstrates significantly superior controllability, manifesting at least a 20% improvement.

The paper you referenced is indeed valuable for a thorough comparison to further validate our method. However, it's crucial to note that it falls within the category of methods we are already comparing. For instance, in [1], the conversion of segment-level conditions into an embedding vector, followed by concatenation with token embeddings at each time step, aligns with what we denoted as the "Embedding" method in Table 5.

Additionally, three other Works[2-4], namely "Compose & Embellish," "Multitrack Music Transformer," and "Compound Word Transformer," share similarities with the "Prefix Tokens" method employed in our approach. This similarity lies in the explicit representation of condition tokens in the input sequence instead of condensing them into a vector. Notably, this architecture is widely employed for decoder-only generation tasks, akin to Prefix language modeling [5].

We appreciate your observation regarding missing references and will promptly address this in Section 3.2 and Section 4.3.2 in the updated version. Thank you for bringing it to our attention.

[1] Wu, S. L., & Yang, Y. H. (2021). MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE.

[2] Wu, S. L., & Yang, Y. H. (2023). Compose & Embellish: Well-structured piano performance generation via a two-stage approach.

[3] Dong, H. W., Chen, K., Dubnov, S., McAuley, J., & Berg-Kirkpatrick, T. (2022). Multitrack Music Transformer.

[4] Hsiao, W.-Y., Liu, J.-Y., Yeh, Y.-C., & Yang, Y.-H. (2021). Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs.

[5] Wang, Thomas, et al. "What language model architecture and pretraining objective works best for zero-shot generalization?." International Conference on Machine Learning. PMLR, 2022.

Reviewer comment (Experiments):
The text-to-attribute results seem inflated, as the data generation technique seems to always place the ground truth attribute value in the description. While this is a valid type of description, it is hard to believe that human descriptions, conditional on wanting to address specific attributes, would always directly express that in natural language as explicitly as is done in the paper. I think an extension that could significantly improve the paper is to perform the ChatGPT-based data augmentation on the already-filled-in captions themselves and allow ChatGPT to obfuscate the ground-truth attributes, as the task then actually becomes text-to-attribute understanding (rather than just key word extraction).

Author response:
Thanks for pointing out this interesting topic that we can discuss. 

Initially, in the process of selecting attributes as conditions, we carefully considered the inclusion of explicit and obscure descriptions to cater to diverse user preferences. For professional musicians, efficiency is enhanced by providing technical music terms, enabling the model to generate music explicitly aligned with their descriptions. Conversely, users without a music background express their ideas using more obscure language to convey emotions or preferred styles. Consequently, we utilize objective attributes to streamline the music creation process for musicians and introduce subjective attributes such as emotion, artistic style, and genres to facilitate expressive music generation for users of varying backgrounds and preferences.

Furthermore, our framework is designed to be expansible, allowing for the inclusion of more intriguing and diverse descriptions, as you suggested. In response to your recommendations, we endeavored to construct an additional dataset by leveraging ChatGPT to fine-tune the text-to-attribute understanding stage. In this phase, we instructed ChatGPT to simulate the role of a musician, randomly selecting attributes and their corresponding values (which can also be user-provided). Subsequently, we tasked ChatGPT, assuming the role of a user without a music background, to employ these chosen values in generating descriptions enriched with more abstract language. To ensure logical alignment, we let it provide an explanation of how the description corresponds to the selected values. This augmentation is presented in Appendix D in the updated version.

Reviewer comment (Evaluation):
I have serious concerns about the evaluation section of the work. Namely, ASA and AvgAttrCtrlAcc are metrics I have never seen before in prior work on controllable music generation, which makes it exceedingly hard to compare this work to prior works.

Author response:
The metrics ASA (Attribute Specific Accuracy) and AvgAttrCtrlAcc (Average Attribute Control Accuracy) have been specifically introduced to gauge the control performance of the proposed system. Given the objective of guiding the music generation process through attributes specified in textual descriptions, it is imperative to assess whether the generated samples adhere to the attribute values outlined in the text. 

ASA quantifies the correctness of predicted attributes within each sample and subsequently averages the accuracy across the entire test set. This metric is proposed for this text-to-symbolic music generation framework, which provides insight into the alignment between the generated sample and the textual descriptions, ensuring that the generated content aligns appropriately with the specified attributes.
On the other hand, AvgAttrCtrlAcc measures the accuracy of each controlled attribute individually for the attribute-to-music generation stage. By evaluating the precision of attribute control for each specific parameter, this metric offers a nuanced understanding of the system's ability to accurately manipulate individual attributes.

In contrast to earlier methodologies [1,2] that employ end-to-end approaches for synthesizing symbolic music from textual descriptions, a significant challenge arises in assessing the alignment between the generated music and the specified attributes in the text. This difficulty arises because the prior methods do not explicitly identify these attributes. In contrast, MuseCoco takes a distinctive approach by explicitly defining objective attributes within the text descriptions. These defined attributes can be easily extracted from the generated music, facilitating a straightforward evaluation of alignment through direct accuracy calculations. This departure from traditional end-to-end methods not only enables a more precise assessment but also establishes a clear and robust relationship between textual descriptions and the resulting musical output. 


[1] Shangda Wu and Maosong Sun. Exploring the efficacy of pre-trained checkpoints in text-to-music generation task. arXiv preprint arXiv:2211.11216, 2022.

[2] Yixiao Zhang, Ziyu Wang, Dingsu Wang, and Gus Xia. BUTTER: A representation learning framework for bi-directional music-sentence retrieval and generation. In Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA), pp. 54–58. Association for Computational Linguistics, 16 October 2020.

Reviewer comment (Evaluation):
It is unclear how ASA is calculated (from within the paper and as well from the Appendix). If ASA is calculated based on the text-to-attribute model, then how does that actually evaluate the music generation itself? Conversely, if ASA is calculated based on the attribute-to-music model, how is each attribute value being calculated deterministically from the output music?

Author response:
ASA plays a crucial role in the Main Results section, specifically for comparing text-to-music generation against various baselines. Specifically, we produce samples for distinct baselines using identical text descriptions as inputs. These text descriptions are synthesized following the methods outlined in Section 3.3. Given that the text descriptions are generated based on the attributes defined in Table 8, we can easily assess the alignment of the generated music with the text descriptions. This evaluation involves extracting objective attributes from the generated music and comparing them with the attributes outlined in the text descriptions. For subjective attributes, we conducted a listening test to gauge control accuracy. Subsequently, we determine the proportion of accurately predicted attributes in each sample and compute the average accuracy across the entire test set.

To facilitate understanding, consider the following example: 

“Text descriptions: 3/4 is the time signature of the music. This song is unmistakably classical in style. This music is low-tempo. The use of piano is vital to the music.”

Within these descriptions, three attributes—time signature, tempo, and instrumentation—are objective, while one, genre, is subjective. Objective attribute values are extracted from generated samples, and a listening test is conducted to categorize the subjective attribute (genre). Assume that correct predictions are made for time signature, tempo, and genre, while instrumentation predictions fall short. The resulting sample-wise accuracy is 75%. By aggregating these accuracies across the entire testing set, the averaged sample-wise accuracy (ASA) is determined.

Reviewer comment (Evaluation):
It is never defined what AvgAttrCtrlAcc is in the main paper, how it is calculated, or how it is any different from ASA.

Author response:
AvgAttrCtrlAcc serves as a key metric for evaluating control accuracy during the attribute-to-music generation stage. It specifically gauges the accuracy of predicting each attribute by calculating the proportion of correctly predicted instances across all testing samples. The resulting average accuracy for each specific attribute is presented in Table 11. It's important to distinguish AvgAttrCtrlAcc from ASA in two fundamental aspects: 1) AvgAttrCtrlAcc is computed by averaging accuracy over a particular attribute across the entire testing set, whereas ASA is an average calculated over samples containing multiple attributes within the testing set. 2) AvgAttrCtrlAcc is designed to evaluate the attribute-controlling performance during the attribute-to-music generation stage, focusing on the precision of individual attributes. In contrast, ASA is formulated to assess the overall alignment between generated samples and provided text descriptions. 

Utilizing abbreviations may lead to a misperception that we are introducing an entirely novel metric. In reality, both metrics fundamentally measure accuracy in distinct aspects, which is not groundbreaking. To prevent any misunderstanding, we have decided to rename them as **"Text-controlled Accuracy"** and **"Attribute-controlled Accuracy"** in the updated version. Thanks for your advice.

Reviewer comment (Experiments):
In Appendix B.2, it is mentioned that ASA values are calculated over different sample sizes and batch sizes for each method. While the costs for the GPT-4 baseline are understandable, I do not understand why the BART-base baseline could not have the exact same sample size as the proposed method.

Author response:
Thank you for bringing this to our attention. The variance in sample size between MuseCoco and the BART-based baseline results from our listening test methodology. In the listening test, we typically sample three generated responses for each prompt. To manage time constraints, we opted to generate only five clips per prompt for the BART-based baseline. Your valuable feedback prompted us to reconsider, and in light of your input, we have increased the sample size for the BART-based baseline to 10 samples per prompt, aligning it with MuseCoco's sample size. 

Consequently, we recalculated the ASA values, and the updated result for the BART-based baseline is now 32.47%, compared to the previous value of 31.98%. We will reflect this correction in the main paper. Your timely and thoughtful feedback has contributed significantly to the accuracy and completeness of our study.

Reviewer comment (Experiments):
It is unclear whether the BART-base model was retrained on the same data or used directly from pretrained checkpoints (though I think it is implied the latter). In this case, without training on the same data it is hard to tell whether any of the differences are due to model design or just differences in data quality / size, aside from the fact that BART-base has 50% of the number of parameters that the proposed model does.

Author response:
Achieving a truly equitable comparison proves to be a formidable task, particularly when confronted with the challenge of aligning the model size of GPT-4 while ensuring parity with a BART-based model.

Several notable works deviate from the conventional approach of comparing baselines solely based on identical training data sizes. For instance, in the case of MusicLM [2] and Noise2Music [3], comparisons were conducted by directly querying APIs or running inference steps using provided checkpoints, without strictly adhering to matching data sizes.

In our pursuit of a fair assessment, we advocate for evaluating models at their pinnacle performance levels, enabling the results to stand on their own merit. The BART-based model, chosen for its exceptional performance among various pretrained checkpoints as detailed in [1], is employed directly from pretrained checkpoints. Despite this, our system distinguishes itself by leveraging a substantial volume of unlabeled data within the proposed framework, thereby showcasing optimal performance. Renowned for its formidable language modeling capabilities, GPT-4 owes its prowess to an extensive dataset and robust model parameters.

The results unequivocally demonstrate MuseCoco's superiority over the two mentioned baselines in terms of controllability and musicality. It's noteworthy that the version of MuseCoco used for this comparison, as indicated in Table 6, does not represent its maximum potential with a larger model size. Nevertheless, even in this configuration, MuseCoco outperforms the baselines, attesting to its impressive capabilities.

[1] Shangda Wu and Maosong Sun. Exploring the efficacy of pre-trained checkpoints in text-to-music generation task. arXiv preprint arXiv:2211.11216, 2022.

[2] Agostinelli, Andrea, et al. "Musiclm: Generating music from text." arXiv preprint arXiv:2301.11325 (2023).

[3] Huang, Qingqing, et al. "Noise2music: Text-conditioned music generation with diffusion models." arXiv preprint arXiv:2302.03917 (2023).

Reviewer comment (Experiments):
Claims that the model "closely resembles human performances in musicality" are not substantiated, since no comparison with human performance was done.

Author response:
Thank you for bringing this out. Comparing the cost-effectiveness of our results with human compositions, where employing musicians to compose based on provided text descriptions incurs significant expenses, we have opted for a subjective listening test. This test is designed to evaluate musicality by gauging the similarity between the generated samples and human compositions.  
To measure this similarity, we have designed a scoring system ranging from 1 to 5 as illustrated in Section 4.1, with higher scores indicating a closer resemblance to human compositions, which is explicitly stated in questionnaire. To ensure the professionalism of the feedback, we only consider responses from participants with a proficiency level of at least 3, as defined in Table 9. This ensures that participants can provide expert evaluations on the gap between human compositions and the generated samples.  
MuseCoco attains a score of 4.06, significantly closer to 5 than the baselines. This scoring approach aligns with the methodology employed in numerous prior studies [1-4].  


[1] Dong, Hao-Wen, et al. "Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.  

[2] Yu, Botao, et al. "Museformer: Transformer with fine-and coarse-grained attention for music generation." Advances in Neural Information Processing Systems 35 (2022): 1376-1388.  

[3] Schneider, Flavio, Zhijing Jin, and Bernhard Schölkopf. "Mo\^ usai: Text-to-Music Generation with Long-Context Latent Diffusion." arXiv preprint arXiv:2301.11757 (2023).  

[4] Shih, Yi-Jen, et al. "Theme transformer: Symbolic music generation with theme-conditioned transformer." IEEE Transactions on Multimedia (2022).

Reviewer comment (Reproducibility):
Training was performed on private datasets so the result is not reproducible - it would be excellent if a comparison with publicly available data could be done.

Author response:
We are committed to ensuring reproducibility, and therefore, we provide codes, and will release trained checkpoints later in the future, for transparency in our results. This will not only facilitate the replication of our findings but also enable the re-training of the model using publicly available data. While the released code supports re-training, it's worth noting that due to the limited availability of public data, utilizing the checkpoints directly is recommended for more robust reproducibility.  

It's important to highlight that our past experiments indicate that larger datasets contribute to enhanced performance. To illustrate, we present the comparative results below:  

|                                 | Musicality    | Controllability | Overall       | ASA(%) |
|---------------------------------|---------------|-----------------|---------------|--------|
| MuseCoco (6 layers, 20w data)   | 3.55$\pm$0.81 | 3.55$\pm$1.11   | 3.56$\pm$0.80 | 77.45  |
| MuseCoco (16 layers, 100w data) | 4.06$\pm$0.82 | 4.15$\pm$0.78   | 4.13$\pm$0.75 | 77.59  |
|                                 |               |                 |               |        |

In examining this table, it becomes evident that as both model size and data size increase, there is a corresponding improvement in musicality and control accuracy. Consequently, leveraging pretrained checkpoints becomes imperative to attain a relatively superior result.

Reviewer comment (Experiments):
There was no commentary on how this set of attributes were arrived at - although it is mentioned that this list could be expanded easily, it would require a complete re-train of the model. Is there any justification that this is a good base set of attributes?

Author response:
We transform the ABC notation music generated by the baselines into MIDI music using the music21 library [1]. After getting the midi files, the same procedure for the extraction of attributes follows the rules outlined in Table 8. Elements such as instrument, bar, time signature, and tempo are directly derived from MIDI sequences. The pitch range is computed by subtracting the minimum pitch from the maximum pitch within a music sequence. Rhythm danceability is determined by calculating the ratio of downbeats, while rhythm density is measured as the average note density. Key estimation is based on note distribution using musical rules. Additionally, time is derived from the time signature and the total number of bars. We will add the detail to Appendix C in the updated version, thanks for your question.  

[1] http://web.mit.edu/music21/

Reviewer comment (Experiments):
Is there a reason that the Lakh dataset was not used too for training (I know you use the MMD but this is not a superset)?

Author response:
Actually, Lakh dataset is included in MMD corpus when we got from the authors, so we don’t mention it additionally.

Reviewer comment (Experiments):
How would you expand this model for multi-instrument music?

Author response:
The model inherently possesses the capability to handle multi-instrument music, as demonstrated on the demo page (e.g., Sample 3, Sample 4, and Sample 5 in Generation Diversity section). As illustrated in Table 7, users can select different instruments collectively as conditions for generating music, so that users can specify a diverse range of instrumentation through text descriptions.

Reviewer comment (Presentation):
Some readers may possibly criticize the results as "cherry-picking." Although objective evaluations such as Table 4 and comments by professional musicians make the results convincing, the lack of examples in the body of the paper gives an impression that something is missing. Simply posting multiple sheets or piano rolls would probably improve the impression of the paper.

Author response:
When you suggested including the sheet in the "main body" of the paper, we opted to focus more on the general idea and results due to page limitations. However, if it's acceptable to add this information in the Appendix, we have done so in the updated version (i.e., Appendix E). We want to clarify that our argument is not an attempt to conceal inconvenient information; otherwise, we could have omitted the sheets from the demo page.

It's worth noting that not all papers of this nature necessarily include examples; it depends on the specific topics. For instance, works [1,2] also don't include examples, perhaps because it's challenging to assess effectiveness by examining only a few bars. In our case, explaining how the generated music aligns well with the text inputs is complex with just a few bars shown in the lead sheets. Some objective attributes need to be calculated over the entire sequence, and subjective attributes should be evaluated through listening experiences. We appreciate your proactive response and understanding, contributing to the enhancement and persuasiveness of our work.

[1] von Rütte, Dimitri, et al. "FIGARO: Controllable Music Generation using Learned and Expert Features." The Eleventh International Conference on Learning Representations. 2022.

[2] Wu, Shih-Lun, and Yi-Hsuan Yang. "MuseMorphose: Full-song and fine-grained piano music style transfer with one transformer VAE." IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023): 1953-1967.

Reviewer comment (Presentation):
Even though the example in Figure 4 is the baseline GPT-4 result to which the proposed method is being compared, the paper would still benefit from including specific input examples and output results of the proposed method in the main body.

Author response:
Yes, it is the baseline GPT-4 result that may not be controlled well. Given the limited page length, we showcase input examples and output results of our proposed method on the demo page, prioritizing the communication of the core idea within the main paper. We plan to release checkpoints in the future so that users can directly get the lead sheets from their given descriptions. Thanks for your attention.