From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: This paper proposes a sequence-to-sequence model for music generation from discrete codes. The discrete codes can be human annotations or learned from vector quantization. Experiments show superior performance compared to the state of the art.

The paper has strong experimental results. The novelty is incremental, as 3 out of 4 reviewers commented on this. The value of the paper lies in the execution of the experiments.
justification_for_why_not_higher_score: The novelty is limited.
justification_for_why_not_lower_score: There is still merit in the execution of the experiments, and building of the system itself.
summary_of_AC-reviewer_meeting: Only two of the reviewers engaged in the conversation. Both were ambivalent about the results. Both commented that the novelty is limited.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper5189/Authors
title: Revised Version of the Paper
comment: We have uploaded a revised version of the manuscript, implementing all minor comments that were pointed out by the reviewers. In particular, the following changes have been made:
- Fixed spelling mistakes ("GithHub" -> "GitHub", "MOA" -> "MAO")
- Made links appear as text (in addition to being clickable)
- We elaborate on "random guessing" of the baseline in Section 6
- Changed mention of $F_\mathrm{learned}$ to $F_\mathrm{latent}$
- Added missing citation about mode collapse
- Added clarification about training of the description-to-sequence models in Section 4

We encourage the reviewers to point out any further comments or suggestions they may have. We are happy to implement their feedback in another revision before the deadline or in the final version of the paper.

From: ICLR.cc/2023/Conference/Paper5189/Reviewer_itPu
title: response to the authors
comment: Thank you for your response. 

I do not have anything additional to add. It was a pleasure to read the manuscript.

All the best,

From: ICLR.cc/2023/Conference/Paper5189/Authors
title: General Comment to all Reviewers
comment: We thank all the reviewers for their constructive feedback and for the time spent to write the reviews. We are glad to see that all reviewers have acknowledged the contributions of our work and recognized its value. Besides, we are pleased to see that the manuscript is considered well-written and easy to follow by all reviewers. We address the individual concerns of each reviewer below their report.

From: ICLR.cc/2023/Conference/Paper5189/Authors
title: Response to Reviewer g33b (2/2)
comment: #### **Clarifying questions**
- Re: Description-to-Sequence models. There exist three models which are trained independently: One is trained only on expert descriptions, one is trained only on learned descriptions and one is trained on both descriptions at the same time. All hyperparameters are left constant but the input during training is different, yielding three distinct models.
- Re: “random guessing”. We agree that this phrasing is not the most precise, considering that we do not specify what kind of distribution is meant by “random”. It should be more clear by adapting the phrasing you propose, so we’ll do that in the revised version of the paper.
- Re: Creating learned descriptions. As hinted at before, the latent description requires existing music to extract features from, so to create a latent description from scratch, one would need to compose at least one bar of music. The extracted features then could be repeated or otherwise modified to yield a complete learned description.
- Re: Mention of $F_\mathrm{latent}$. Correct, this should indeed read $F_\mathrm{learned}$, thank you for pointing this out.
- Re: Style transfer. This is a good observation and has been a topic of discussion among the authors. We are aware that this terminology is not entirely consistent with the literature, where style transfer often refers to applying the style of one sample to the content of another one. Ultimately, we decided to use the term “style transfer” as we felt this was very descriptive of what is actually happening: New pieces of music are generated in a way such that their style matches the predefined condition, so by extracting the condition from a given piece, we are transferring its style to the generated one. This terminology is consistent with related work (Choi et al. ‘20; Wu & Yang ‘21), which is ultimately why we chose to use it. *This being said, if we’re able to minimize confusion by adopting more generally used terminology, we’re happy to do that in the revised version of the paper.*

We hope we were able to address your main concerns and are looking forward to a constructive conversation. In the case that we were able to resolve any issues regarding the use case, which you mentioned to be the main weakness, we would also like to ask you to consider whether this can improve your final recommendation.

---

References:
- Choi et al. '20: https://proceedings.mlr.press/v119/choi20b.html
- Wu & Yang '21: https://arxiv.org/abs/2105.04090

From: ICLR.cc/2023/Conference/Paper5189/Authors
title: Response to Reviewer g33b (1/2)
comment: Thank you for your thorough and insightful review. We are happy to hear that you found our experiments convincing and extensive, that our listening study provides further evidence that the generated music is of an overall high quality and that our manuscript is clearly written. 
We appreciate the feedback and would like to engage in a conversation about some of the points raised.

#### **Usability of the proposed model**
Let us preface with summarizing the use case we imagine: Some user has some high-level ideas about a piece they wish to compose. They have two tools at their disposal: 1) expert description; 2) take the template of certain bars of some music piece and use them as a condition for the bar at hand. We would like to remark that creating a new expert description is significantly easier and less time consuming than composing a piece (note-by-note) of music of the same length from scratch. FIGARO shows that providing the user with high-level features is promising and it paves the way for new choices, such as features that better align with human emotions. As acknowledged, the process of creating descriptions could be streamlined and made much easier by developing additional tools to help in interacting with the model more efficiently. For example, the selection of chord progressions could be made more accessible even to people without domain knowledge by letting the user choose from a selection of well-known songs and use its chord progression as a template. Similarly, using sliders to select values for note density, mean pitch/velocity/duration will be much more intuitive than typing values between 1 and 32. Indeed, making a user-friendly application based on the proposed model might prove to be an interesting HCI challenge in itself. In the end, it boils down to the time and resources available within the scope of the project. As it’s intended to be a machine learning research project, we decided not to focus on this aspect too much and instead leave it up to future work. Still, we didn’t want to completely neglect the usability side, which is why we provide an (admittedly cumbersome) interface in the form of the interactive demo. Learned features are originally conceived for two reasons: a) increase music quality by filling the gap between the true piece and what the expert description can’t straightforwardly model; b) provide an additional conditioning tool, allowing the user to generate bars characterized by certain properties of some template bars. This is one way FIGARO could be used in practice and our experiments positively support this idea. 

#### **Motivation behind the learned description**
The learned description is intended as a way to “let the model decide what’s important”. This is in contrast to the expert description, where we extract features that we as authors deem important from a compositional standpoint and throw away the rest of the information. It is almost certain that the set of expert features does not capture the underlying music in its entirety and that many “information gaps” are left for the model to fill in. Therefore the model might be able to better reconstruct the underlying sequence by just using learned features, which are optimized to encode all information about each respective partition. Based on the experimental results, however, this is not the case (given that the models are well-trained). In fact, the best model is the one combining expert and learned features, which is able to rely on expert features for some information and use the learned features for the rest.
As far as human input is concerned, in order for the user to condition on latent features, they need to provide at least one bar of music to extract features from. Once some features have been extracted, they can be replicated, spliced and mixed to form a desirable conditioning sequence (see Section 6.3).

In conclusion: The motivation behind our choice to use learned features is an important point and we’re ready to acknowledge that this should have been addressed more thoroughly in the paper. We hope that our motivation is more clear now, and we’re open to including a more detailed explanation such that confusion is avoided as much as possible.

From: ICLR.cc/2023/Conference/Paper5189/Authors
title: Response to Reviewer W9Fj
comment: Thank you for your review and constructive feedback. We are happy to see that many aspects of our work were positively received. In particular, it was recognized that the main goal of the paper was achieved, which is controllable music generation on a fine-grained level. It was also noted that the work was original, that the proposed method was elegant, simple yet effective and that the writing was clear. Thank you for the positive comments. We also confirm that *the code (and trained models) will be publicly available via GitHub*, although to keep the authors anonymous the link had to be removed from the submission.

We would also like to engage in a conversation on the extent of our contribution, as this was raised as the main weakness of the paper. As we understand, the concern is twofold: Firstly, the proposed method has many parallels to Jukebox, hence could be seen as applying the same techniques to a different modality (symbolic music as opposed to audio music) and secondly, the addition of a prior model could have strengthened the contribution significantly, as this would enable unconditional generation. We see your point and do agree to some extent, but beg to differ from a qualitative standpoint.

#### **Regarding Similarity to Jukebox**
Jukebox was a breakthrough in music generation and *we have the utmost respect for what has been achieved*. We indeed build upon the findings of Dhariwal et al., as they were able to show that VQ-VAE is an effective approach for learning sequence representations for conditional generation. However, the two models are trained on different modalities, which is of course the most obvious difference, and there are other key distinctions beyond that. First and foremost, we use the VQ-VAE learned features in conjunction with expert ones in the downstream model. The expert description is an integral part of our method and crucial for enabling controllability on a fine-grained level, which is the goal of the paper. In a similar vein, Jukebox allows for lyrics-conditioning, but does so through the prior model, which generates the VQ-VAE codes for reconstruction rather than conditioning the downstream model directly. 
Going into further detail, in our work we train a bar-level VQ-VAE entirely independently from the downstream model and only use the resulting encoder for feature extraction, whereas Jukebox trains a hierarchical VQ-VAE on reconstruction and only uses the decoders in the downstream model. Note that our VQ-VAE only ever sees a single bar and therefore extracts features temporally independently, whereas Jukebox operates on the entire sequence through a sliding window. In our approach, temporal dependence is only introduced in the downstream description-to-sequence model, where learned features are combined with expert ones. Our approach is also simpler as it doesn’t require a hierarchical VQ-VAE to establish long-range dependencies, since the downstream sequence-to-sequence model can see the entire conditioning sequence at once.

#### **Training a Prior Model**
The point about training a prior model is an observant one, as this has been a topic of discussion among the authors. A prior model in our case would have been able to generate expert descriptions and learned descriptions unconditionally or conditionally on the other description respectively. We did not anticipate this to be particularly challenging from a technical standpoint (*nor a particularly novel contribution*), as these kinds of sequence modeling problems have been solved to great success in the recent past (Jukebox, language models s.a. GPT, VQGAN, machine translation, …) and are very similar to training FIGARO. However, we ended up deciding against it as this would not contribute to our main goals of achieving fine-grained controllability and human-interpretability. In fact, controllability could be diminished in a setting where music is generated unconditionally. Ultimately, we had to prioritize our efforts within the time and resource limits that were imposed on the project. While a prior model is something we feel would be “nice to have”, it is not essential and can be left as future work.

--- 

In conclusion, we regret to hear that the novelty of our contributions is considered incremental and we hope that we were able to address your concerns adequately. We invite you to a constructive conversation and to challenge us on any points you might disagree with, as we strive to use this opportunity to make the paper the best it can be. As authors, we believe the simplicity of our approach to be a strength, as a simpler solution to the same problem is generally to be preferred over a more complicated one.

If we were indeed able to address your main concerns, we would like to ask you to consider whether or not this can increase your final score.

From: ICLR.cc/2023/Conference/Paper5189/Authors
title: Response to Reviewer GHu3
comment: Thank you for your extensive review and insightful comments. We appreciate the positive feedback: It was noted that the approach was original and a valid research contribution, that the paper was well written and that it achieves what it claims. We would like to engage in a conversation and have a constructive discussion about the weaknesses that were mentioned.

#### **Human-Machine Interaction**
You bring up an important point that we as authors also feel passionate about: namely the user-interaction aspect of AI models. As mentioned in the paper (Section 1 and 7), we believe that human-AI interaction is an important component of generative models. Having shown that the proposed approach can successfully achieve controllable generation and is able to “execute” instructions provided to it by a user, the next step will be to improve them. The paper at hand is focused on the machine learning aspect of the problem, so gathering user feedback will be crucial for making an informed decision on which parts need to be improved.
While the set of expert features is an important choice, there are no constraints on which features are included. We show that this particular set of features, the choice of which is informed by previous work, *works well in terms of interpretability, controllability and sample quality, which was the main goal*. Future work might want to further refine this choice in order to optimize some additional objectives.

#### **UI Interface**
With regards to inpainting and missing controls, we agree that the real-world applicability of the model in its current form might be somewhat limited, but we believe that adding a suitable user interface would already make it really quite useful. As mentioned in our reply to Reviewer W9Fj, a UI that lets the user pick chord progressions based on templates, adds sliders for real values like note density and mean pitch/velocity/duration and maybe also lets the user set attributes globally (e.g. time signature) will make the experience much less cumbersome. We did not implement such an interface, but anyone can do so using the checkpoints we provide for download. We believe that both features you mention could be implemented with an appropriate interface and *our current model makes this possible*.

#### **Minor Comments**
Thank you for pointing out that the links are invisible. In our version, all links had colorful bounding boxes which apparently got lost in translation. This (and the spelling error) will be fixed in the revised version. Here’s the link to the interactive demo:
https://colab.research.google.com/drive/1UAKFkbPQTfkYMq1GxXfGZOJXOXU_svo6

From: ICLR.cc/2023/Conference/Paper5189/Authors
title: Response to Reviewer itPu
comment: We would like to thank the reviewer for their report. In particular, we were happy to see that the main contributions of the paper – the introduction of a novel objective, the possibility to handle multi-track music, the combination of expert and learned features, and our state-of-the-art quantitative results – have been recognized and appreciated by the reviewer.  We are also grateful to hear that the manuscript is well-written and easy to follow and that our contribution is sufficiently novel.

With regards to the minor issues you pointed out, all of them will be addressed in the revised version of the paper. If there are any remaining issues or clarifications, we invite you to point them out in order to make the best version of the paper possible.

From: ICLR.cc/2023/Conference/Paper5189/Reviewer_itPu
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper presents a transformer-based conditional model to generate multitrack symbolic music. The model utilizes a "description to sequence" learning scheme. The authors show that the model produces more desirable musical sequences when human-interpretable features are incorporated during training.


strength_and_weaknesses: Strengths:

- Novel training objective
- The model can handle multi-track, multi-signature symbolic music
- Comparative experiments between strong baselines in different settings with respect to numerous evaluation measures

There are a few cosmetic issues, which are explained below.


clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and very easy to follow. The approach, experiments and results are clearly presented, and the appendices are very well prepared. 
- The literature review is comprehensive.
- The model is built on past work, yet it's sufficiently novel
- The code and model weights are shared. The implementation details (Appendix A) and extended data representing (REMI+) provide the readers with enough detail to reproduce the model.

Minor issues:
- At the start of page six: "macro overlapping area (MAO)" -> should have been MOA
- Hyperlinks to websites (e.g. SoundCloud, Google Colab) should be more explicit, e.g. as shortened URL
- Please cite a work describing "mode collapse"
summary_of_the_review: The work advances the state-of-the-art. The usage of learned and expert descriptions is sound. The experiments are comprehensive, and the dataset is large and representative enough (for music information research). The authors also extend the existing data representation in existing datasets. Given these strengths, I would like to recommend the paper for acceptance.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper5189/Reviewer_W9Fj
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper describes a music generation approach which aims to extend the type of conditioning variables typically used in this area of research. The authors describe this as so-called description-to-sequence learning. Inspired by recent progress in text-conditioned image generation, the authors propose to derive descriptions of music in terms of musical attributes which can be easily extracted from the data itself. These descriptions are termed as expert descriptions. In addition, to fill in the gaps in the expert descriptions per bar, they also propose to learn descriptions using a VQ-VAE. Combining the two: expert and learned descriptions, the authors learn a sequence to sequence model transforming said descriptions to musical sequences. The proposed method is compared against competing methods using both objective and subjective evaluations. In all cases, FIGARO outperforms the baseline approaches. The authors also carry out various ablation studies showing the benefit of different types of expert descriptions.
strength_and_weaknesses: Strengths:
- The proposed method is elegant in combining domain knowledge of music to extract attributes along with a VQ-VAE to learn latent descriptors of measures which are fed to a transformer encoder-decoder architecture to generate musical sequences.
- The approach extends the fine-grained control of previous approaches to more musical dimensions thus affording more control in the output.
- The results, both qualitative and quantitative look and sound good.

Weakness:
- The contributions seem a bit incremental. While it's true that this has not been done before in symbolic music, the setup is pretty similar to Jukebox with the addition of some extra musical attributes as conditioning minus the prior to generate conditioning latents/descriptions. In fact, the absence of the prior feels like a big weakness, but this is left as future work by the authors.

clarity,_quality,_novelty_and_reproducibility: The paper quality is good and writing is very clear. The work is also original, although I feel it lacks novelty a bit.
The authors have shared the code in the supplementary material and it seems like they plan to share it online as well.
summary_of_the_review: My overall impression of this paper is that it presents a simple yet effective method for controllable music generation. My only issue is that it feels a little incomplete and does not extend existing literature to the extent I would have hoped. Despite that, the results sound convincing and the user study shows that this method outperforms some competing approaches. Thus I am rating it as a weak accept.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5189/Reviewer_GHu3
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper describes a system for training a multi-instrument symbolic music generation model conditioned on two types of features: human-interpretable attributes that can be extracted programmatically, and latent VQ-VAE encodings that are learned.  Both feature types can be used to exert fine-grained control over a piece of generated music, as demonstrated in several experiments.
strength_and_weaknesses: Strengths
-----------
1) The paper does exactly what it claims: proposes a system for controlling multi-instrument music generation with human-interpretable and learned latent features, and evaluates the technical aspects of the system.

2) The samples on SoundCloud sound quite reasonable.

Weaknesses
--------------
1) Far and away the biggest thing missing here, even though the authors may consider it out of scope for ICLR, is some kind of user study demonstrating that this method of controlling a music generation system is helpful to musicians or composers, expert or novice.  The experiments in the paper demonstrate that the system *obeys* the controls and produces plausible music, but not that it is usable.  And based on the baseline models, it isn't surprising that the model would obey the conditioning.  Much of the effectiveness of such a system seems like it would depend on exactly what attributes are used for expert conditioning, and this question isn't explored beyond measuring the impact of dropping out individual attributes, nor is there discussion of why these attributes were chosen.

2) Two major weaknesses of the system that I suspect limit its real-world utility are its lack of support for a) infilling and b) missing controls.  Even though the conditioning signal is temporally fine-grained, a change to the conditioning at time T1 may have effects at time T2 distant from T1, which is likely not what users want.  And having to provide two types of conditioning values (one with multiple attributes) at all positions in a piece of music is likely to be very cumbersome for users.

3) Minor issues:
    * The paper mentions that an online demo is available; however no link is provided.
    * "GitHub" is misspelled in the footnote on page 2.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, original if somewhat incremental, seemingly experimentally sound, and I believe I could reproduce it with some effort.
summary_of_the_review: The paper seems correct and is a valid research contribution to the area of music generation.  I sincerely hope the authors (or someone else) writes another paper that explores the HCI questions relevant to the system presented in this paper.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5189/Reviewer_g33b
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: FIGARO is a model trained on a seq2seq task, where the input is either a human-interpretable high-level description of the music or a learned sequence of VQ-VAE tokens. Output is a symbolic music composition in the proposed REMI+ format, an extension to REMI that adds multi-instrument and multi-time signature capabilities. The authors show via several experiments and metrics that the produced music reflects the inputs given to the model.
strength_and_weaknesses: **Strengths**

- The experiments and metrics are convincing that the model does utilize the inputs when constructing the output music.
- The listening study indicates that the output music is of an overall high quality (or at least as high as the training dataset).
- Code and weights are available for download.

**Weaknesses**

My main concern is that the motivation for the model is somewhat unclear. The authors say that providing the expert description is easier than actually writing the music, but it seems to be only marginally easier. You still have to annotate every bar and include attributes that aren't necessarily musically intuitive, like note density and mean pitch. I could perhaps see constructing an UI that makes these attributes more intuitive. The authors do mention some of these issues in the Discussion section, but it would have been nice to see a more clear path toward direct usability.

I also don't understand the motivation for the learned representation at all. It doesn't seem to be suitable for human input, so I guess another model would be necessary to translate between another representation and the learned tokens?
clarity,_quality,_novelty_and_reproducibility: **Novelty**

This work is most closely related to MMM (Ens & Pasquier, 2020) in that it allows for multi-track, multi-instrument symbolic generation. This paper expands on those ideas by adding more conditioning controls.

**Quality**

Extensive experiments are done to ensure model output is based on input and that overall quality is close to the training dataset.

**Reproducibility**

Code and weights are available for download. A colab demo provides the ability to interact with the model.

**Clarity**

Overall the paper is well written, but I had a few clarifying questions:

- Section 4, Description-to-Sequence Model: How is training done so that the model supports any combination of expert or learned annotations? Is a combination selected randomly for each input?

- Section 6: I thought calling unconditional generation "random guessing" was a little misleading. Perhaps you could say something about sampling a sequence based on the training distribution prior.

- Section 6.1: "both $F_{expert}$ and especially $F_{learned}$ require time and domain knowledge to create" How would someone go about creating $F_{learned}$, even with domain knowledge?

- Section 6.1: There's a mention of $F_{latent}$ that I think was supposed to be $F_{learned}$

- Section 6.2: This doesn't sound like style transfer to me, this is more of a reconstruction metric. Style transfer would usually imply that some semantic quality of the music remains constant while underlying details change, but here you're just looking for as close of a reconstruction as possible. The "Description mixing" test sounds closer to style transfer, but from listening to the outputs, it's not clear to me that it's successful as a style transfer process.

- Links throughout the paper are not clear. It was only by accident that I realized I could click on the Colab link. I would suggest highlighting the links or just including the URL to make it more obvious.
summary_of_the_review: FIGARO provides a way to condition symbolic music generation on several per-bar controls, and experiments show that these controls are successful. However, the use case is not completely clear.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold