Original: FIGARO: C ONTROLLABLE M USIC G ENERATION USING E XPERT AND L EARNED F EATURES
Revised: FIGARO: C ONTROLLABLE M USIC G ENERATION USING E XPERT AND L EARNED F EATURES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent symbolic music generative models have achieved significant improvements in the quality of the generated samples.
Revised: Recent symbolic music generative models have achieved significant improvements in the quality of the generated samples.

Original: Nevertheless, it remains hard for users to control the output in such a way that it matches their expectation.
Revised: Nevertheless, it remains hard for users to control the output in such a way that it matches their expectation.

Original: To address this limitation, high-level, human-interpretable conditioning is essential.
Revised: To address this limitation, high-level, human-interpretable conditioning is essential.

Original: In this work, we release FIGARO, a Transformer-based conditional model trained to generate symbolic music based on a sequence of high-level control codes.
Revised: In this work, we release FIGARO, a Transformer-based conditional model trained to generate symbolic music based on a sequence of high-level control codes.

Original: To this end, we propose description-to-sequence learning, which consists of automatically extracting fine-grained, human-interpretable features (the description ) and training a sequence-to-sequence model to reconstruct the original sequence given only the description as input.
Revised: To this end, we propose description-to-sequence learning, which consists of automatically extracting fine-grained, human-interpretable features (the description ) and training a sequence-to-sequence model to reconstruct the original sequence given only the description as input.

Original: FIGARO achieves state-of-the-art performance in multi-track symbolic music generation both in terms of style transfer and sample quality.
Revised: FIGARO achieves state-of-the-art performance in multi-track symbolic music generation both in terms of style transfer and sample quality.

Original: We show that performance can be further improved by combining human-interpretable with learned features.
Revised: We show that performance can be further improved by combining human-interpretable with learned features.

Original: Our extensive experimental evaluation shows that FIGARO is able to generate samples that closely adhere to the content of the input descriptions, even when they deviate significantly from the training distribution.
Revised: Our extensive experimental evaluation shows that FIGARO is able to generate samples that closely adhere to the content of the input descriptions, even when they deviate significantly from the training distribution.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Music is a fascinating subject that surrounds us constantly, being a source of inspiration and canvas for imagination to many.
Revised: Music is a fascinating subject that surrounds us constantly, being a source of inspiration and canvas for imagination to many.

Original: To some, creating music is a topic worthy of dedicating one’s life to, which is a testament to the artistry and mastery involved.
Revised: To some, creating music is a topic worthy of dedicating one’s life to, which is a testament to the artistry and mastery involved.

Original: While composition is an intricate form of art that requires a deep understanding of the human experience and domain knowledge, the idea of devising a systematic or algorithmic approach to music creation has been around for centuries (Nierhaus, 2009).
Revised: While composition is an intricate form of art that requires a deep understanding of the human experience and domain knowledge, the idea of devising a systematic or algorithmic approach to music creation has been around for centuries (Nierhaus, 2009).

Original: With the advent of deep learning, automatic music generation has witnessed renewed interest (Hernandez-Olivan & Beltran, 2021).
Revised: With the advent of deep learning, automatic music generation has witnessed renewed interest (Hernandez-Olivan & Beltran, 2021).

Original: Especially the Transformer architecture (Vaswani et al., 2017), which has seen applications to many Machine Learning domains (Brown et al., 2020; Dosovitskiy et al., 2021; Lample & Charton, 2019; Biggio et al., 2021), has proven to be a powerful tool for musical sequence modelling.
Revised: Especially the Transformer architecture (Vaswani et al., 2017), which has seen applications to many Machine Learning domains (Brown et al., 2020; Dosovitskiy et al., 2021; Lample & Charton, 2019; Biggio et al., 2021), has proven to be a powerful tool for musical sequence modelling.

Original: Initial breakthroughs by Huang et al.
Revised: Initial breakthroughs by Huang et al.

Original: and
Revised: and

Original: Payne (2019) applied language modelling techniques to symbolic music to achieve state-of-the-art music generation.
Revised: Payne (2019) applied language modelling techniques to symbolic music to achieve state-of-the-art music generation.

Original: Though these models were capable of some form of conditional generation (e.g. melody or genre conditioning), other conditioning mechanisms and different types of control have since been proposed (Ens & Pasquier, 2020; Choi et al., 2020; Wu & Yang, 2021).
Revised: Though these models were capable of some form of conditional generation (e.g. melody or genre conditioning), other conditioning mechanisms and different types of control have since been proposed (Ens & Pasquier, 2020; Choi et al., 2020; Wu & Yang, 2021).

Original: As deep generative models are improving and producing more and more realistic samples, it remains an area of active research how humans can interact with these models and steer them to generate a desirable result.
Revised: As deep generative models are improving and producing more and more realistic samples, it remains an area of active research how humans can interact with these models and steer them to generate a desirable result.

Original: Recent efforts in text-to-image generation (Ramesh et al., 2022; Saharia et al., 2022) have shown the potential in usability and artistic applications of human-interpretable controllable generative models.
Revised: Recent efforts in text-to-image generation (Ramesh et al., 2022; Saharia et al., 2022) have shown the potential in usability and artistic applications of human-interpretable controllable generative models.

Original: Whereas text-based conditioning has yielded human-interpretable control for image generation, the same conditioning mechanisms are not easily applicable to music generation.
Revised: Whereas text-based conditioning has yielded human-interpretable control for image generation, the same conditioning mechanisms are not easily applicable to music generation.

Original: We aim to extend this kind of control to other domains, in this case to music generation.
Revised: We aim to extend this kind of control to other domains, in this case to music generation.

Original: As scale has proven to be key for achieving capable models, we cannot rely on scarce annotated data and instead propose a self-supervised objective, which we call description-to-sequence learning.
Revised: As scale has proven to be key for achieving capable models, we cannot rely on scarce annotated data and instead propose a self-supervised objective, which we call description-to-sequence learning.

Original: We take inspiration from recent text-to-image approaches, but instead of a natural language description of the target, we automatically extract a sequence of high-level features (the description ).
Revised: We take inspiration from recent text-to-image approaches, but instead of a natural language description of the target, we automatically extract a sequence of high-level features (the description ).

Original: These can  either be hand-crafted using domain knowledge or learned.
Revised: These can [Figure] either be hand-crafted using domain knowledge or learned.

Original: The description then serves as the input to a conditional model to reconstruct the original sequence.
Revised: The description then serves as the input to a conditional model to reconstruct the original sequence.

Original: To this end, we define a description function which extracts said features from a given piece of music.
Revised: To this end, we define a description function which extracts said features from a given piece of music.

Original: The choice of description function[Figure] determines the characteristics of the resulting model and serves as an inductive bias, allowing us to emphasize desirable properties such as human-interpretability or fine-grained control over instruments and chord progression.
Revised: The choice of description function determines the characteristics of the resulting model and serves as an inductive bias, allowing us to emphasize desirable properties such as human-interpretability or fine-grained control over instruments and chord progression.

Original: Note that the general nature of the proposed framework allows for applications to other domains despite our focus on symbolic music.
Revised: Note that the general nature of the proposed framework allows for applications to other domains despite our focus on symbolic music.

Original: Our main contribution is FIGARO (FIne-grained music Generation via Attention-based, RObust control), a model trained on the proposed description-to-sequence objective by combining two separate description functions: 1)
Revised: Our main contribution is FIGARO (FIne-grained music Generation via Attention-based, RObust control), a model trained on the proposed description-to-sequence objective by combining two separate description functions: 1)

Original: The hand-crafted expert description, which provides global context in the form of a high-level, human-interpretable sequence and 2) the learned description, where we use representation learning to extract salient features from the source sequence.
Revised: The hand-crafted expert description, which provides global context in the form of a high-level, human-interpretable sequence and 2) the learned description, where we use representation learning to extract salient features from the source sequence.

Original: The learned description is intended to amend the expert description with high-fidelity information in places where the latter might be incomplete, albeit at the cost of human-interpretability.
Revised: The learned description is intended to amend the expert description with high-fidelity information in places where the latter might be incomplete, albeit at the cost of human-interpretability.

Original: The model is trained on conditional generation, mapping descriptions to music.
Revised: The model is trained on conditional generation, mapping descriptions to music.

Original: An illustrated overview of the model is given in Figure 1.
Revised: An illustrated overview of the model is given in Figure 1.

Original: At inference time, users may interact with the model in human-interpretable description space and we provide a simple interface in the form of an online demo of our model.
Revised: At inference time, users may interact with the model in human-interpretable description space. We provide a simple interface in the form of an online demo of our model.

Original: 1 We also release the source code and model weights for anyone to download and use freely.
Revised: 1 We also release the source code and model weights for anyone to download and use freely.

Original: 2 Our secondary contribution is REMI+, an extension to the REMI input representation (Huang & Yang, 2020) which makes it applicable to multi-track, multi-time-signature music.
Revised: 2 Our secondary contribution is REMI+, an extension to the REMI input representation (Huang & Yang, 2020) which opens the way to multi-track, multi-time-signature music.

Original: We evaluate FIGARO on its ability to adhere to the prescribed condition by comparing it to state-ofthe-art methods for controllable symbolic music generation (Choi et al., 2020; Wu & Yang, 2021).
Revised: We evaluate FIGARO on its ability to adhere to the prescribed condition by comparing it to state-ofthe-art methods for controllable symbolic music generation (Choi et al., 2020; Wu & Yang, 2021).

Original: We demonstrate empirically that our technique outperforms the state-of-the-art in controllable generation and sample quality.
Revised: We demonstrate empirically that our technique outperforms the state-of-the-art in controllable generation and sample quality.

Original: To evaluate sample quality, we employ subjective evaluation in the form of a listening study.
Revised: To evaluate sample quality, we employ subjective evaluation in the form of a listening study.

Original: We further demonstrate that FIGARO is robust with respect to distributional shifts in description space and performs well on constructed samples outside the training distribution, indicating that the proposed objective is effective at learning generalized concepts about the data.
Revised: We further demonstrate that FIGARO is robust with respect to distributional shifts in description space and performs well on constructed samples outside the training distribution, indicating that the proposed objective is effective at learning generalized concepts about the data.

Original:  C ONTROLLABLE S YMBOLIC M USIC G ENERATION
Revised:  C ONTROLLABLE S YMBOLIC M USIC G ENERATION

Original: Music emerges as an expression of human emotions, which is why, it is especially important for the user to be able to steer the generation process in a desired direction, i.e. to exert creative control, in the case of music modelling.
Revised: In the context of generative modelling, controllability is an important issue, as such models only become useful if the user is able to steer the generation process in a desired direction. This has recently been observed for text-to-image models and we intend to take a closer look at controllable music generation.

Original: We identify two different levels of controllability over the generation process.
Revised: We identify two different levels of controllability: global and fine-grained control.

Original: The most prevalent form of control is global conditioning of the model, where the generation is guided by a constant set of attributes which do not change during the generation process.
Revised: Global conditioning, where the generation is guided by a constant set of attributes that do not change during the generation process, is the most prevalent form of control.

Original: Examples of global control include prompt-based conditioning (Payne, 2019) or conditional decoding of latent representations(Brunner et al., 2018).
Revised: Examples of global control include prompt-based conditioning (Payne, 2019) or conditional decoding of latent representations

Original: Fine-grained control is achieved when the generation process can be guided at any point in time, i.e. if the control attributes can be arbitrarily varied over time.

Original: Consider controlling what instruments are playing in the generated sequence as an example of global control in contrast to controlling what instruments are playing at any point in time as an example of

Original: Method
Revised: Method

Original: Input Rep.
Revised: Input Rep.

Original: Multi-Track
Revised: Multi [Table] Track

Original: Multi-Sig.
Revised: Multi

Original:  GlobalCtrl.
Revised: [Table] Global

Original: Fine-Grained Ctrl.
Revised: [Table] Fine

Original: MIDI-VAE Brunner et al.
Revised: [Table]

Original:  Pianoroll ✓ ✓ ✓ - MuseNet (Payne, 2019)
Revised: [Table] Pianoroll ✓ ✓ ✓ [Table]

Original: MIDI-like ✓ ✓ ( ✓ )- MMM (Ens & Pasquier, 2020)
Revised: MIDI [Table] ✓ ✓ [Table]

Original: MIDI-like ✓ ✓ ✓ ( ✓ )
Revised: [Table] MIDI- [Table] ✓ ✓ ✓ ( ✓ )

Original: Choi et al.
Revised: [Table] [Table]

Original: MIDI-like - ✓ ✓ ( ✓ )
Revised: MIDI-like - ✓ ✓ ( ✓ )

Original: MuseMorphose (Wu & Yang, 2021)
Revised: MuseMorphose [Table]

Original: REMI - - ✓ ✓
Revised: REMI - - ✓ ✓

Original: FIGARO (ours)
Revised: FIGARO (ours)

Original: REMI+ ✓ ✓ ✓ ✓ fine-grained control.
Revised: REMI+ ✓ ✓ ✓ ✓ (Brunner et al., 2018).

Revised: Fine-grained control is achieved when the generation process can be guided at any point in time, i.e. if the control attributes can be arbitrarily varied over time.

Revised: Consider controlling what instruments are playing in the generated sequence as an example of global control in contrast to controlling what instruments are playing at any point in time as an example of fine-grained control.

Original: Note that fine-grained control also implies global control, as global control can be achieved by fixing the control attributes to some constant value.
Revised: Note that fine-grained control also implies global control, as global control can be achieved by fixing the control attributes to some constant value.

Original: Fine-grained control is therefore a strictly more powerful property and seemingly harder to obtain, as is highlighted in Table 1.
Revised: Fine-grained control is therefore a strictly more powerful property and seemingly harder to obtain, as is highlighted in Table 1.

Original: 2. R ELATED W ORK
Revised: 2. R ELATED W ORK

Original: The capabilities of symbolic music generative models have been steadily improving with many recent contributions (Huang et al., 2018; Payne, 2019; Huang & Yang, 2020; Hsiao et al., 2021; Wu et al., 2022).
Revised: The capabilities of symbolic music generative models have been steadily improving with many recent contributions (Huang et al., 2018; Payne, 2019; Huang & Yang, 2020; Hsiao et al., 2021; Wu et al., 2022).

Original: This line of work focuses on improving the quality of generated samples but does not contribute substantially toward controllability.
Revised: This line of work focuses on improving the quality of generated samples but does not contribute substantially toward controllability.

Original: An exception to that is MuseNet (Payne, 2019), which allows some control through prompt-based conditioning with control tokens.
Revised: An exception to that is MuseNet (Payne, 2019), which allows some control through prompt-based conditioning with control tokens.

Original: Even still, prompt-based control is very limited, as control tokens are “forgotten” by the model once the generation advances beyond the initial context size.
Revised: Even still, prompt-based control is very limited, as control tokens are “forgotten” by the model once the generation advances beyond the initial context size.

Original: Another line of work focuses on finding ways of controlling the generation process.
Revised: Another line of work focuses on finding ways of controlling the generation process.

Original: VAE-based methods often achieve global control through latent conditioning vectors, achieving genre transfer (Brunner et al., 2018)  or control over arousal (Tan & Herremans, 2020).
Revised: VAE-based methods often achieve global control through latent conditioning vectors, enabling genre transfer (Brunner et al., 2018) capabilities or control over arousal (Tan & Herremans, 2020).

Original: Transformer auto-encoders have been used for melody conditioning (Huang et al., 2018; Choi et al., 2020) and encoding of musical style Choi et al.
Revised: Transformer autoencoders have been used for melody conditioning (Huang et al., 2018; Choi et al., 2020) and encoding of musical style Choi et al.

Original: Ens & Pasquier (2020) present MMM which is capable of bar-level and track-level symbolic music inpainting.
Revised: Ens & Pasquier (2020) present MMM which is capable of bar-level and track-level symbolic music inpainting.

Original: Finally, Wu & Yang (2021) introduce MuseMorphose for fine-grained attribute conditioning and latent space style editing.
Revised: Finally, Wu & Yang (2021) introduce MuseMorphose for fine-grained attribute conditioning and latent space style editing.

Original: All of these approaches have some limitations that are highlighted in Table 1.
Revised: All of these approaches have some limitations that are highlighted in Table 1.

Original: Recent work tackling the problem of controllable generation is often either limited to single-track, single-time-signature music or to global control.
Revised: Recent work tackling the problem of controllable generation is often either limited to single-track, single-time-signature music or to global control.

Original: Fine-grained control has been a topic of interest in the recent literature (Choi et al., 2020; Wu & Yang, 2021; Di et al., 2021; Ferreira & Whitehead, 2021) and is an essential property when considering user-directed applications.
Revised: Fine-grained control has been a topic of interest in the recent literature (Choi et al., 2020; Wu & Yang, 2021; Di et al., 2021; Ferreira & Whitehead, 2021) and is an essential property when considering user-directed applications.

Original: In essence, fine-grained control is necessary to allow control over salient features in the generation, as saliency in music at least partly lies in how it changes over time.
Revised: In essence, fine-grained control is necessary to allow control over salient features in the generation, as saliency in music at least partly lies in how it changes over time.

Original: In addition, salient features may be impossible or prohibitively expensive to quantify directly (Choi et al., 2020; Ferreira & Whitehead, 2021), emphasizing the need for un- or self-supervised fine-grained control.
Revised: In addition, salient features may be impossible or prohibitively expensive to quantify directly (Choi et al., 2020; Ferreira & Whitehead, 2021), emphasizing the need for un- or self-supervised fine-grained control.

Original: Our proposed method provides both global and fine-grained control through description-to-sequence learning.
Revised: Our proposed method provides both global and fine-grained control through description-to-sequence learning.

Original:  D ESCRIPTION - TO -S EQUENCE L EARNING
Revised:  D ESCRIPTION - TO -S EQUENCE L EARNING

Original: Description-to-sequence learning is a self-supervised objective based on the reconstruction of a sequence given automatically extracted features of that same sequence (the description ) as input.
Revised: Description-to-sequence learning is a self-supervised objective based on the reconstruction of a sequence given automatically extracted features of that same sequence (the description ) as input.

Original: Conceptually, an information bottleneck is applied by mapping a sample to the description space and the model is trained to learn a probabilistic map from description space to sample space.
Revised: Conceptually, an information bottleneck is applied by mapping a sample to the description space and the model is trained to learn a probabilistic map from description space to sample space

Revised: [Figure]

Original: Description-to-Sequence Objective.
Revised: Description-to-Sequence Objective.

Original: Let V seq denote a sequence vocabulary and V desc a description vocabulary.
Revised: Let V seq denote a sequence vocabulary and V desc a description vocabulary.

Original: Let x ∈ V ∗ seq be an input sequence and F : V ∗ seq → V ∗ desc be a partition-wise feature extraction function that extracts descriptive features given a partition x 1 , . . .
Revised: Let x ∈ V ∗ seq be an input sequence and F : V ∗ seq → V ∗ desc be a partition-wise feature extraction function that extracts descriptive features given a partition x 1 , . . .

Original: , x n of x such that[Figure] x = concat( x 1 , . .. , x n ) .
Revised: , x n of x such that x = concat( x 1 , . . , x n ) .

Original: We then define the description of x to be:
Revised: We then define the description of x to be:

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we write d = F ( x ) in the following.
Revised: For simplicity, we write d = F ( x ) in the following.

Original: The description-to-sequence objective is then given by the cross-entropy reconstruction loss of x given F ( x ) :
Revised: The description-to-sequence objective is then given by the cross-entropy reconstruction loss of x given F ( x ) :

Original: [Equation]
Revised: [Equation]

Original: Here X is the true distribution of sequences and ϕ denotes the parameters of a sequence-to-sequence model.
Revised: Here X is the true distribution of sequences and ϕ denotes the parameters of a sequence-to-sequence model.

Original: An illustrated overview of this objective is given in Figure [Figure]
Revised: An illustrated overview of this objective is given in Figure 3.

Original: Note that some level of fine-grained control is guaranteed by the way we partition x prior to feature extraction, as the length of partitions puts an upper bound on the receptive field of the description function.
Revised: Note that some level of fine-grained control is guaranteed by the way we partition x prior to feature extraction, as the length of partitions puts an upper bound on the receptive field of the description function.

Original: 3. E XPERT D ESCRIPTION
Revised: 3. E XPERT D ESCRIPTION

Original: [Figure]
Revised: [Figure]

Original: We aim for a human-interpretable, fine-grained conditioning mechanism by using domain knowledge to automatically extract high-level features that are easy to understand for human experts and relevant from a compositional standpoint.
Revised: We aim for a human-interpretable, fine-grained conditioning mechanism by using domain knowledge to automatically extract high-level features that are easy to understand for human experts and relevant from a compositional standpoint.

Original: We identify time signature, chords and instruments as such features and also consider note density, mean pitch, mean velocity and mean duration as auxiliary features quantifying musical style (Choi et al., 2020).
Revised: We identify time signature, chords and instruments as such features and also consider note density, mean pitch, mean velocity and mean duration as auxiliary features quantifying musical style (Choi et al., 2020).

Original: All of these features are computed bar-per-bar and quantized according to a special-purpose vocabulary.
Revised: All of these features are computed bar-per-bar and quantized according to a special-purpose vocabulary.

Original: Finally, all resulting tokens are concatenated into a sequence, which yields the expert description function
Revised: Finally, all resulting tokens are concatenated into a sequence, which yields the expert description function

Original: F expert ( x ) , of which an example is given in Figure 2.
Revised: F expert ( x ) , of which an example is given in Figure 2.

Original: More details on the feature extraction algorithm and quantization can be found in Appendix C.
Revised: More details on the feature extraction algorithm and quantization can be found in Appendix C.

Original: 3. L EARNED D ESCRIPTION
Revised: 3. L EARNED D ESCRIPTION

Original: While the expert description serves as a human-interpretable inductive bias, it is only able to capture fairly low-fidelity features by design and suffers from non-injectivity (i.e. there can be many sequences that map to one description).
Revised: While the expert description serves as a human-interpretable inductive bias, it is only able to capture fairly low-fidelity features by design and suffers from non-injectivity (i.e. there can be many sequences that map to the same description).

Original: In an attempt to alleviate some of these limitations, we use representation learning to extract features that maximize the information content about the underlying sequence.
Revised: In an attempt to alleviate some of these limitations, we use representation learning to extract features that maximize the information content about the underlying sequence.

Original: We choose the VQ-VAE framework as the basis for our learned description function as it produces discrete codes that allow for learning a prior sequence model and has been shown to be effective for various tasks such music generation in particular (Dhariwal et al., 2020).
Revised: We choose the VQ-VAE framework as the basis for our learned description function as it produces discrete codes that allow for learning a prior sequence model and has been shown to be effective for various tasks such music generation in particular (Dhariwal et al., 2020).

Original: The latent representations given by this model is used as the output of the learned description function F learned ( x ) .
Revised: The latent representations given by this model is used as the output of the learned description function F learned ( x ) .

Original: This way, we add high-fidelity information and reduce non-injectivity, albeit at the cost of human interpretability.
Revised: This way, we add high-fidelity information and reduce non-injectivity, albeit at the cost of human interpretability.

Original: We later show that combining both descriptions yields better performance than using either one on its own (Section 6.2) and that the resulting model is fairly robust (Section 6.3).
Revised: We later show that combining both descriptions yields better performance than using either one on its own (Section 6.2) and that the resulting model is fairly robust (Section 6.3).

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: REMI+ Input Representation.
Revised: REMI+ Input Representation.

Original: The Revamped MIDI (REMI) representation (Huang & Yang, 2020) is a beat-based input representation, which has recently been shown to give better sample quality than traditional time-based representation.
Revised: The Revamped MIDI (REMI) representation (Huang & Yang, 2020) is a beat-based input representation, which has recently been shown to give better sample quality than traditional time-based representation.

Original: REMI represents notes with four consecutive tokens encoding note position, pitch, velocity and duration and also includes chord and tempo events.
Revised: REMI represents notes with four consecutive tokens encoding note position, pitch, velocity and duration and also includes chord and tempo events.

Original: In order to make it applicable to any general MIDI piece, we extend REMI by adding instrument and time-signature information, yielding REMI+.
Revised: In order to make it applicable to any general MIDI piece, we extend REMI by adding instrument and time-signature information, yielding REMI+.

Original: We add tokens encoding the time signature of a given bar at the beginning of each bar and instrument tokens at the beginning of each note, thus making it suitable for multi-time-signature, multi-track modelling tasks.
Revised: We add tokens encoding the time signature of a given bar at the beginning of each bar and instrument tokens at the beginning of each note, thus making it suitable for multi-time-signature, multi-track modelling tasks.

Original: We also increase the temporal resolution of note positions from 4 to 12 sub-beats per quarter note.
Revised: We also increase the temporal resolution of note positions from 4 to 12 sub-beats per quarter note.

Original: This allows for precise quantization of triplets in addition to sixteenths, which was previously not possible.
Revised: This allows for precise quantization of triplets in addition to sixteenths, which was previously not possible.

Original: For exact details on quantization and tokenization, we refer to Appendix B. A limitation left unaddressed by REMI+ is the inability to encode expressive performances, which requires frequent shifts in tempo or small time offsets.
Revised: For exact details on quantization and tokenization, we refer to Appendix B. A limitation left unaddressed by REMI+ is the inability to encode expressive performances, which requires frequent shifts in tempo or small time offsets.

Original: Time-based representations are still preferred for that setting.
Revised: Time-based representations are still preferred for that setting.

Original: In our experiments, we tokenize MIDI files using the REMI+ input representation and train FIGARO on the proposed objective using the combination of the proposed expert and latent description functions.
Revised: In our experiments, we tokenize MIDI files using the REMI+ input representation and train FIGARO on the proposed objective using the combination of the proposed expert and latent description functions.

Original: Description-to-Sequence Model.
Revised: Description-to-Sequence Model.

Original: We use the original Transformer auto-encoder (Vaswani et al., 2017) architecture with 4 encoder and 6 decoder layers as the sequence-to-sequence backbone.
Revised: We use the original Transformer auto-encoder (Vaswani et al., 2017) architecture with 4 encoder and 6 decoder layers as the sequence-to-sequence backbone.

Original: Three separate models are trained using the proposed description-to-sequence objective (Equation 2) using either F expert ( x ) , F learned ( x ) or a combination of both descriptions.
Revised: Three separate models are trained using the proposed description-to-sequence objective (Equation 2) using either F expert ( x ) , F learned ( x ) or a combination of both descriptions.

Original: Combining descriptions is done by simply summing their respective learned token embeddings.
Revised: Combining descriptions is done by simply summing their respective learned token embeddings.

Original: The model architecture is kept deliberately simple, so as to be able to accurately assess the effectiveness of the various description functions.
Revised: The model architecture is kept deliberately simple, so as to be able to accurately assess the effectiveness of the various description functions.

Original: Further details and hyperparameters are given in Appendix A.
Revised: Further details and hyperparameters are given in Appendix A.

Original: VQ-VAE Model.
Revised: VQ-VAE Model.

Original: For our bar-level VQ-VAE model, we use the same Transformer auto-encoder backbone but with an additional quantization bottleneck consisting of 16 latent codes with a shared codebook of size 2048 between the encoder and decoder.
Revised: For our bar-level VQ-VAE model, we use the same Transformer auto-encoder backbone but with an additional quantization bottleneck consisting of 16 latent codes with a shared codebook of size 2048 between the encoder and decoder.

Original: The model is trained on reconstruction for individual bars.
Revised: The model is trained on reconstruction for individual bars.

Original: Note that training FIGARO is a two-stage process: In the first step, we train the
Revised: Note that training FIGARO is a two-stage process: In the first step, we train the

Original: VQ-VAE for feature extraction, which is later used to compute F learned ( x ) .
Revised: VQ-VAE for feature extraction, which is later used to compute F learned ( x ) .

Original: The VQ-VAE model is frozen during training of the description-to-sequence model.
Revised: The VQ-VAE model is frozen during training of the description-to-sequence model.

Original: Training Setup.
Revised: Training Setup.

Original: We use the LakhMIDI dataset (Raffel, 2016) as training data in all of our experiments, which to the best of our knowledge is the largest publicly available symbolic music dataset.
Revised: We use the LakhMIDI dataset (Raffel, 2016) as training data in all of our experiments, which to the best of our knowledge is the largest publicly available symbolic music dataset.

Original: We use a 80%-10%-10% training-validation-test split.
Revised: We use a 80%-10%-10% training-validation-test split.

Original: Each model is trained for 24 hours on 4 Nvidia GTX 2080 Ti GPUs.
Revised: Each model is trained for 24 hours on 4 Nvidia GTX 2080 Ti GPUs.

Original: For evaluation, we generate samples conditioned on descriptions sampled from the test set, generating 32 bars for each sample.
Revised: For evaluation, we generate samples conditioned on descriptions sampled from the test set, generating 32 bars for each sample.

Original: For training details, we refer to Appendix A.
Revised: For training details, we refer to Appendix A.

Original:  P ERFORMANCE E VALUATION 5. F LUENCY
Revised:  P ERFORMANCE E VALUATION 5. F LUENCY

Original: We use perplexity (PPL) as a way to measure fluency and to compare the likelihood of different models in addition to task-specific metrics.
Revised: We use perplexity (PPL) as a way to measure fluency and to compare the likelihood of different models in addition to task-specific metrics.

Original: The perplexity measures the likelihood of sequences while normalizing over the sequence length, which makes it better suited to comparing sequences of different lengths than the negative log-likelihood.
Revised: The perplexity measures the likelihood of sequences while normalizing over the sequence length, which makes it better suited to comparing sequences of different lengths than the negative log-likelihood.

Original: 5. D ESCRIPTION F IDELITY
Revised: 5. D ESCRIPTION F IDELITY

Original: We also quantitatively evaluate the fidelity of generated sequences to the given condition.
Revised: We also quantitatively evaluate the fidelity of generated sequences to the given condition.

Original: Let x denote a test sample and F ( x ) its description.
Revised: Let x denote a test sample and F ( x ) its description.

Original: Then we generate ˆ x by sampling the model conditioned on F ( x ) and examine x and ˆ x for similarity.
Revised: Then we generate ˆ x by sampling the model conditioned on F ( x ) and examine x and ˆ x for similarity.

Original: Metrics are computed as an empirical estimate over the test distribution.
Revised: Metrics are computed as an empirical estimate over the test distribution.

Original: More details and exact formulas are given in Appendix D.
Revised: More details and exact formulas are given in Appendix D.

Original: Accuracy.
Revised: Accuracy.

Original: We compute accuracy metrics for categorical values, namely for instruments, chords and time signature.
Revised: We compute accuracy metrics for categorical values, namely for instruments, chords and time signature.

Original: Instruments and chord are multi-label features for which we compute the mean Fscore.
Revised: Instruments and chord are multi-label features for which we compute the mean Fscore.

Original: As there is only one time signature per bar, we compute the standard accuracy score.
Revised: As there is only one time signature per bar, we compute the standard accuracy score.

Original: Macro Overlapping Area.
Revised: Macro Overlapping Area.

Original: Previous work has used the overlapping area (OA) metric to quantify similarity between two musical sequences for a given feature (Choi et al., 2020; Wu & Yang, 2021).
Revised: Previous work has used the overlapping area (OA) metric to quantify similarity between two musical sequences for a given feature (Choi et al., 2020; Wu & Yang, 2021).

Original: However, we find that the standard OA metric fails to take the order of the sequences into account, as feature histograms are computed over the entire sequence.
Revised: However, we find that the standard OA metric fails to take the order of the sequences into account, as feature histograms are computed over the entire sequence.

Original: For example, a sequence has maximal overlapping area with itself in reverse order, even though the resulting sequences will sound completely different.
Revised: For example, a sequence has maximal overlapping area with itself in reverse order, even though the resulting sequences will sound completely different.

Original: To alleviate this limitation, we propose the macro overlapping area (MOA), which partition-wise computes the overlap in the distributions of a given feature, taking sequential order into account.
Revised: To alleviate this limitation, we propose the macro overlapping area (MOA), which partition-wise computes the overlap in the distributions of a given feature, taking sequential order into account.

Original: We use the MOA metric to compute similarity in pitch, velocity and duration between ground truth and reconstruction.
Revised: We use the MOA metric to compute similarity in pitch, velocity and duration between ground truth and reconstruction.

Original: The exact definition of our MOA metric is given in Appendix D.
Revised: The exact definition of our MOA metric is given in Appendix D.

Original: Normalized Root-Mean-Square Error.
Revised: Normalized Root-Mean-Square Error.

Original: We compute the normalized RMSE (NRMSE) for bar-wise note density.
Revised: We compute the normalized RMSE (NRMSE) for bar-wise note density.

Original: This helps compare similarity accross different feature magnitudes.
Revised: This helps compare similarity accross different feature magnitudes.

Original: Cosine Similarity.
Revised: Cosine Similarity.

Original: We also evaluate chroma and grooving similarity as a way to quantify similarity in sound and rhythm as proposed by Wu & Yang (2021).
Revised: We also evaluate chroma and grooving similarity as a way to quantify similarity in sound and rhythm as proposed by Wu & Yang (2021).

Original: We compute bar-wise cosine similarity for the chroma vectors (Fujishima, 1999) and grooving vectors (Dixon et al., 2004).
Revised: We compute bar-wise cosine similarity for the chroma vectors (Fujishima, 1999) and grooving vectors (Dixon et al., 2004).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Recent work on controllable generation has largely focused on single-track symbolic music and to the best of our knowledge, there are no competitive baselines for multi-track music generation with fine-grained control.
Revised: Recent work on controllable generation has largely focused on single-track symbolic music and to the best of our knowledge, there are no competitive baselines for multi-track music generation with fine-grained control.

Original: To make due, we train two state-of-the-art methods for single-track controllable generation (Choi et al., 2020; Wu & Yang, 2021) on the REMI+ input representation, which allows us to apply them to a multi-track setting.
Revised: To make due, we train two state-of-the-art methods for single-track controllable generation (Choi et al., 2020; Wu & Yang, 2021) on the REMI+ input representation, which allows us to apply them to a multi-track setting.

Original: We also train an unconditional baseline based on Music
Revised: We also train an unconditional baseline based on Music

Original: Transformer (Huang et al., 2018), which acts as a sanity check: It has no additional information about the target sequence and essentially performs “random guessing”, i.e. it uniformly samples a sequence from the training distribution.
Revised: Transformer (Huang et al., 2018), which acts as a sanity check: It has no additional information about the target sequence and essentially performs “random guessing”, i.e. it uniformly samples a sequence from the training distribution.

Original: If any conditional model performs worse than unconditional on similarity metrics, it is an indication for mode collapse (Dieng et al., 2018).
Revised: If any conditional model performs worse than unconditional on similarity metrics, it is an indication for mode collapse (Dieng et al., 2018).

Original: Indeed, although the training of MuseMorphose (Wu & Yang, 2021) initially appeared to be successful, quantitative analysis revealed that the model had suffered mode collapse, as evidenced by the small chord and instrument entropy in Table 2 and the worse-than-unconditional performance on some of the style transfer metrics in Table 3a.
Revised: Indeed, although the training of MuseMorphose (Wu & Yang, 2021) initially appeared to be successful, quantitative analysis revealed that the model had suffered mode collapse, as evidenced by the small chord and instrument entropy in Table 2 and the worse-than-unconditional performance on some of the style transfer metrics in Table 3a.

Original: As this model was originally designed for modelling single-track piano music and the LakhMIDI dataset constitutes a quite drastic domain shift, we instead focus on comparison to Choi et al.
Revised: As this model was originally designed for modelling single-track piano music and the LakhMIDI dataset constitutes a quite drastic domain shift, we instead focus on comparison to Choi et al.

Original: Details on baseline training are given in Appendix E.
Revised: Details on baseline training are given in Appendix E.

Original: 6. H UMAN I NPUT
Revised: 6. H UMAN I NPUT

Original: The question of how well FIGARO can handle human-generated descriptions is both important and tricky to evaluate quantitatively, as both F expert and especially F learned require time and domain knowledge to create (albeit substantially less than composing a piece of music from scratch).
Revised: The question of how well FIGARO can handle human-generated descriptions is both important and tricky to evaluate quantitatively, as both F expert and especially F learned require time and domain knowledge to create (albeit substantially less than composing a piece of music from scratch).

Original: We therefore demonstrate the generative capabilities of our model in an interactive demo 6 based on
Revised: We therefore demonstrate the generative capabilities of our model in an interactive demo 6 based on

Original: F expert .
Revised: F expert .

Original: We provide four hand-crafted examples, namely a simple four-bar piano progression, an 8-bar description containing uncommon instruments, a 9-bar description based on a well-known Jazz standard and a longer 20-bar description demonstrating control over multiple features in parallel with fine-grained instructions over instruments, chords, note density and mean pitch.
Revised: We provide four hand-crafted examples, namely a simple four-bar piano progression, an 8-bar description containing uncommon instruments, a 9-bar description based on a well-known Jazz standard and a longer 20-bar description demonstrating control over multiple features in parallel with fine-grained instructions over instruments, chords, note density and mean pitch.

Original: These example are meant to show several attributes of FIGARO, namely its level of controllability and its ability to generate meaningful outputs even when input descriptions are human-made and present features that rarely appear in the training set.
Revised: These example are meant to show several attributes of FIGARO, namely its level of controllability and its ability to generate meaningful outputs even when input descriptions are human-made and present features that rarely appear in the training set.

Original: The reader is invited to generate samples based on these descriptions or to come up with their own creations.
Revised: The reader is invited to generate samples based on these descriptions or to come up with their own creations.

Original: Demonstrating the use of F learned on human input is less straight-forward as in the absence of a prior model, latent codes need to be extracted from an existing piece of music.
Revised: Demonstrating the use of F learned on human input is less straight-forward as in the absence of a prior model, latent codes need to be extracted from an existing piece of music.

Original: We approximate this and the previous setting by using pieces of music from the test set to extract descriptions and perform quantitative evaluation on style transfer and controllable generation in Section 6.2 and 6.3 respectively.
Revised: We approximate this and the previous setting by using pieces of music from the test set to extract descriptions and perform quantitative evaluation on style transfer and controllable generation in Section 6.2 and 6.3 respectively.

Original: 6. S TYLE T RANSFER
Revised: 6. S TYLE T RANSFER

Original: In this experiment, we extract conditioning sequences by feeding samples drawn from the test set uniformly at random through neural bottlenecks in the case of MuseMorphose, Choi et al.
Revised: In this experiment, we extract conditioning sequences by feeding samples drawn from the test set uniformly at random through neural bottlenecks in the case of MuseMorphose, Choi et al.

Original: (2020) and
Revised: (2020) and

Original: F learned or a symbolic bottleneck in the case of F expert .
Revised: F learned or a symbolic bottleneck in the case of F expert .

Original: New samples are then generated based on the condition and closeness in style to the original sequence is evaluated using the metrics from Section 5.
Revised: New samples are then generated based on the condition and closeness in style to the original sequence is evaluated using the metrics from Section 5.

Original: Note that many of these metrics are directly present in F expert and FIGARO (expert) expectedly does well in that regard.
Revised: Note that many of these metrics are directly present in F expert and FIGARO (expert) expectedly does well in that regard.

Original: The fact that the results obtained with the expert description adhere to the content

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: ↓

Original: I ↑

Original: C ↑

Original: TS ↑

Original: ND ↓

Original: P ↑

Original: V ↑

Original: D ↑ s c ↑ s g ↑ [Table] [Table] of the description itself is a highly desirable feature for controllable music generation.
Revised: The fact that the results obtained with the expert description adhere to the content of the description itself is a highly desirable feature for controllable music generation.

Original: However, it also performs well on chroma and grooving similarity, both of which are not directly present [Table] Model [Table]
Revised: However, it also performs well on chroma and grooving similarity, both of which are not directly present.

Original: [Table]

Original: [Table]

Original: Unconditional −

Original: [Table]

Original: −

Original: [Table]

Original: Choi et al.

Original: −

Original: [Table]

Original: .− 0 .

Original: Wu & Yang (2021) − 1 .−

Original: [Table]
Revised: [Table]

Original: This shows that the expert description can act as a strong inductive bias and helps guide the generation process even beyond what information is directly represented.
Revised: This shows that the expert description can act as a strong inductive bias and helps guide the generation process even beyond what information is directly represented.

Original: By using this description, the degree of control exerted by the user is enhanced and the results match expectations.
Revised: By using this description, the degree of control exerted by the user is enhanced and the results match expectations.

Original: We also observe that adding learned features further improves performance across the board, with FIGARO beating FIGARO (expert) in every category.
Revised: We also observe that adding learned features further improves performance across the board, with FIGARO beating FIGARO (expert) in every category.

Original: The success of this hybrid approach means that we can preserve the interpretability and inductive bias, yet increase the fidelity of the generated music by exploiting black-box AI.
Revised: The success of this hybrid approach means that we can preserve the interpretability and inductive bias, yet increase the fidelity of the generated music by exploiting black-box AI.

Original: Our expert description and hybrid models significantly outperform all baselines, and the learned description model outperforms
Revised: Our expert description and hybrid models significantly outperform all baselines, and the learned description model outperforms

Original: Choi et al.
Revised: Choi et al.

Original: (2020) on most metrics by a slight margin.
Revised: (2020) on most metrics by a slight margin.

Original: The difference in performance is explained by the conditioning used for Choi et al.
Revised: The difference in performance is explained by the conditioning used for Choi et al.

Original: (2020), where the conditioning vector is temporally aggregated over the entire sample with any style progression throughout the sequence being lost.
Revised: (2020), where the conditioning vector is temporally aggregated over the entire sample with any style progression throughout the sequence being lost.

Original: We provide a non-cherrypicked collection of samples and encourage the reader to get an impression of the quality and diversity of the music by listening to some of them on SoundCloud ( https://tinyurl.com/bdfds8f5 ).
Revised: We provide a non-cherrypicked collection of samples and encourage the reader to get an impression of the quality and diversity of the music by listening to some of them on SoundCloud.

Original: The full list of results is provided in Table 3a.
Revised: The full list of results is provided in Table 3a.

Original: 6. C ONTROLLABLE G ENERATION
Revised: 6. C ONTROLLABLE G ENERATION

Original: To evaluate the robustness of our model with respect to out-of-distribution conditions, we combine two sequences x (1) and x (2) (either splicing or mixing the descriptions) and generate samples conditioned on the resulting descriptions.
Revised: To evaluate the robustness of our model with respect to out-of-distribution conditions, we combine two sequences x (1) and x (2) (either splicing or mixing the descriptions) and generate samples conditioned on the resulting descriptions.

Original: This way, we generate samples that have unusual transitions (in the case of splicing) ordescriptions with conflicting information (in the case of mixing) and are not part of the training distribution in general.
Revised: This way, we create examples that have unusual transitions (in the case of splicing) or conflicting information (in the case of mixing) and are not part of the training distribution in general.

Original: Description splicing.
Revised: Description splicing.

Original: For this experiment, we take 16 bars of each sequence and concatenate them into a novel description ˜ x
Revised: For this experiment, we take 16 bars of each sequence and concatenate them into a novel description ˜ x

Original: = b (1)  ∥ · · · ∥b (1)16 ∥ b (2)17 ∥ · · · ∥ b (2)32 .
Revised: = b (1) [Equation] [Equation] 17 ∥ · · · ∥

Revised: [Equation]

Revised: [Equation]

Original: We then input the description
Revised: We then input the description

Original: F (˜ x ) to the model and sample the output distribution to generate a “medley” of the input sequences.
Revised: F (˜ x ) to the model and sample the output distribution to generate a “medley” of the input sequences.

Original: Importantly, none of the models are finetuned for this specific task.
Revised: Importantly, none of the models are finetuned for this specific task.

Original: We see a drop in performance in all evaluation metrics for every model compared to the style transfer task, which is expected due to distributional shifts in the data.
Revised: We see a drop in performance in all evaluation metrics for every model compared to the style transfer task, which is expected due to distributional shifts in the data.

Original: However, FIGARO and FIGARO (expert) drop significantly less than FIGARO (learned) and Choi et al.
Revised: However, FIGARO and FIGARO (expert) drop significantly less than FIGARO (learned) and Choi et al.

Original: (2020), showing that the expert description provides a strong inductive bias and is robust with respect to distribution shifts.
Revised: (2020), showing that the expert description provides a strong inductive bias and is robust with respect to distribution shifts.

Original: We sometimes even observe FIGARO turning the hard cut-off points from bar-splicing into smooth transitions, which can be attributed to its capacity to pay attention to future bars and anticipate significant changes in style.
Revised: We sometimes even observe FIGARO turning the hard cut-off points from bar-splicing into smooth transitions, which can be attributed to its capacity to pay attention to future bars and anticipate significant changes in style.

Original: The complete list of results is available in Table 3b.
Revised: The complete list of results is available in Table 3b.

Original: We omit Wu & Yang (2021) from these experiments as we do not expect competitive performance due to mode collapse.
Revised: We omit Wu & Yang (2021) from these experiments as we do not expect competitive performance due to mode collapse.

Original: Description mixing.
Revised: Description mixing.

Original: Description mixing is done by combining F expert ( x (1) ) with F learned ( x (2) ) and thus is a scenario that is only applicable to FIGARO, the only model that takes both description types as input.
Revised: Description mixing is done by combining F expert ( x (1) ) with F learned ( x (2) ) and thus is a scenario that is only applicable to FIGARO, the only model that takes both description types as input.

Original: Note that mixing descriptions in this way potentially provides conflicting information to the model as the two sequences might not agree with each other on certain features which might decrease sample quality or lead to a collapse of the model’s performance in the worst case.
Revised: Note that mixing descriptions in this way potentially provides conflicting information to the model as the two sequences might not agree with each other on certain features which might decrease sample quality or lead to a collapse of the model’s performance in the worst case.

Original: Conflicting information also makes quantitative evaluation more difficult, which is why we focus on qualitative evaluation.
Revised: Conflicting information also makes quantitative evaluation more difficult, which is why we focus on qualitative evaluation.

Original: We find that mixing descriptions in this way is not detrimental to the sample quality and that the model relies on different descriptions for different attributes of the generated sample.
Revised: We find that mixing descriptions in this way is not detrimental to the sample quality and that the model relies on different descriptions for different attributes of the generated sample.

Original: In general terms, the model seems to rely on the expert description for attributes including instruments, harmonics and time signature (one might call these “low-fidelity” attributes) and on the learned description for attributes including texture and rhythmic intensity (“high-fidelity” attributes).
Revised: In general terms, the model seems to rely on the expert description for attributes including instruments, harmonics and time signature (one might call these “low-fidelity” attributes) and on the learned [Table] [Table] [Table]

Revised: [Table]

Original: This is not surprising as the expert description is designed to make this kind of low-fidelity information readily available and the learned description is intended to “fill in the gaps” of the expert description.
Revised: ↓ I ↑ C ↑ TS ↑

Revised: ND ↓

Revised: P ↑

Revised: V ↑

Revised: D ↑ s c ↑ s g ↑

Revised: [Table]

Revised: [Table]

Revised: description for attributes including texture and rhythmic intensity (“high-fidelity” attributes).

Revised: This is not surprising as the expert description is designed to make this kind of low-fidelity information readily available and the learned description is intended to “fill in the gaps” of the expert description.

Original: We invite the reader to examine how the models combines two pieces of music into one by listening to some of the samples on SoundCloud ( https://tinyurl.com/4tksvsub ).
Revised: We invite the reader to examine how the models combines two pieces of music into one by listening to some of the samples on SoundCloud.

Original: We also provide evaluation metrics with respect to the two source samples in Appendix F.
Revised: We also provide evaluation metrics with respect to the two source samples in Appendix F.

Original: 6. A BLATION S TUDY
Revised: 6. A BLATION S TUDY

Original: To evaluate which parts of the expert description are essential, we group it into three components: instruments, chords and meta-information.
Revised: To evaluate which parts of the expert description are essential, we group it into three components: instruments, chords and meta-information.

Original: Instruments and chords include all tokens with information about instruments and chords respectively while all other tokens (time signature, note density and mean pitch, velocity and duration) are classified as meta-tokens.
Revised: Instruments and chords include all tokens with information about instruments and chords respectively while all other tokens (time signature, note density and mean pitch, velocity and duration) are classified as meta-tokens.

Original: We train separate models with one part of the description removed and compare the performance to FIGARO (expert), which receives the full expert description as input.
Revised: We train separate models with one part of the description removed and compare the performance to FIGARO (expert), which receives the full expert description as input.

Original: As one would expect, removing each component reduces the performance significantly in the respective metrics, indicating that each component carries useful information not entirely inferable through the remaining components.
Revised: As one would expect, removing each component reduces the performance significantly in the respective metrics, indicating that each component carries useful information not entirely inferable through the remaining components.

Original: Interestingly, our experiments show that removing any component slightly decreases the over-all performance even in metrics that we would not necessarily expect to be affected.
Revised: Interestingly, our experiments show that removing any component slightly decreases the over-all performance even in metrics that we would not necessarily expect to be affected.

Original: Removing instrument information, for example, increases the error for note density, pitch, velocity and duration, indicating that the instruments also carry implied information about those features.
Revised: Removing instrument information, for example, increases the error for note density, pitch, velocity and duration, indicating that the instruments also carry mutual information about those features.

Original: This seems plausible considering the fact that different styles (or genres) of music usually identifies a set of instruments that is common for said style.
Revised: This seems plausible considering the fact that different styles (or genres) of music usually identifies a set of instruments that is common for said style.

Original: Similar arguments can be made for the other two components.
Revised: Similar arguments can be made for the other two components.

Original: The full list of results is available in Table 3c.
Revised: The full list of results is available in Table 3c.

Original: 6. S UBJECTIVE E VALUATION
Revised: 6. S UBJECTIVE E VALUATION

Original: Finally, we evaluate the subjective quality of generated samples through a listening study, comparing our best model to the baselines.
Revised: Finally, we evaluate the subjective quality of generated samples through a listening study, comparing our best model to the baselines.

Original: In this experiment we are not interested in how closely the generated samples follows the prescribed condition, but instead try to quantify the perceived quality of samples by recording pairwise preferences.
Revised: In this experiment we are not interested in how closely the generated samples follows the prescribed condition, but instead try to quantify the perceived quality of samples by recording pairwise preferences.

Original: To this end, we have conducted a survey where participants were asked to indicate their preference between 20 second excerpts of two samples chosen uniformly at random.
Revised: To this end, we have conducted a survey where participants were asked to indicate their preference between 20 second excerpts of two samples chosen uniformly at random.

Original: In total, we gathered 7569 comparisons by 691 unique participants, whose expertise ranged everything between non-musicians and professional musicians, although no prior musical knowledge was required.
Revised: In total, we gathered 7569 comparisons by 691 unique participants, whose expertise ranged everything between non-musicians and professional musicians, although no prior musical knowledge was required.

Original: In two types of questions, participants had to choose 1) between a real sample and a generated sample or 2) between two generated samples.
Revised: In two types of questions, participants had to choose 1) between a real sample and a generated sample or 2) between two generated samples.

Original: Generated samples are taken from the style transfer task.
Revised: Generated samples are taken from the style transfer task.

Original: Question type 1) ranks the different methods on how good generated samples are compared to real, human-composed music.
Revised: Question type 1) ranks the different methods on how good generated samples are compared to real, human-composed music.

Original: In this respect, FIGARO beats all other baseline with a win rate of 39 .
Revised: In this respect, FIGARO beats all other baseline with a win rate of 39 .

Original: 3% compared to the next best model by Choi et al.
Revised: 3% compared to the next best model by Choi et al.

Original: (2020), which has a win rate of 33 . 2% .
Revised: (2020), which has a win rate of 33 . 2% .

Original: As evidenced by the fact that the unconditional baseline (Huang et al., 2018) and
Revised: As evidenced by the fact that the unconditional baseline (Huang et al., 2018) and

Original: Choi et al.
Revised: Choi et al.

Original: (2020) are very close in terms of win rates, sample quality and reconstruction accuracy are not necessarily correlated, highlighting the need for a dedicated evaluation of sample quality.
Revised: (2020) are very close in terms of win rates, sample quality and reconstruction accuracy are not necessarily correlated, highlighting the need for a dedicated evaluation of sample quality.

Original: Win rates of the different models are displayed in Figure 4 with 90% confidence intervals obtained through normal approximation.
Revised: Win rates of the different models are displayed in Figure 4 with 90% confidence intervals obtained through normal approximation.

Original: WhileChoi et al.
Revised: While

Original: (2020) is able to adhere to the prescribed condition as shown in Section 6.2, the quality of generated samples is approximately on par with the unconditional model.
Revised: (2020) is able to adhere to the prescribed condition as shown in Section 6.2, the quality of generated samples is approximately on par with the unconditional model.

Original: FIGARO on the other hand is able to surpass the unconditional baseline in sample quality, while also providing controllable generation capabilities.
Revised: FIGARO on the other hand is able to surpass the unconditional baseline in sample quality, while also providing controllable generation capabilities.

Original: Question type 2) is used to construct a pairwise ranking of models by applying the Wilcoxon signed-rank test on the study results.
Revised: Question type 2) is used to construct a pairwise ranking of models by applying the Wilcoxon signed-rank test on the study results.

Original: In this ranking, our model beats each of the baselines with a p -value of < 10 − 7 .
Revised: In this ranking, our model beats each of the baselines with a p -value of < 10 − 7 .

Original: The complete ranking and test results can be found in Appendix G.
Revised: The complete ranking and test results can be found in Appendix G.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Future Work.
Revised: Future Work.

Original: While our work represents a step towards high-quality symbolic music generation with human-interpretable control, we recognize three main avenues for future work.
Revised: While our work represents a step towards high-quality symbolic music generation with human-interpretable control, we recognize three main avenues for future work.

Original: Firstly, the proposed model is strictly speaking not a generative model as creative input in the form of a description is still required.
Revised: Firstly, the proposed model is strictly speaking not a generative model as creative input in the form of a description is still required.

Original: Training a prior model to generate such descriptions unconditionally could allow for faster generation with less human input required while still retaining the ability to influence the generation process in description space.
Revised: Training a prior model to generate such descriptions unconditionally could allow for faster generation with less human input required while still retaining the ability to influence the generation process in description space.

Original: Secondly, adding new description functions or extending the proposed ones could lead to further improved controllability and sample quality.
Revised: Secondly, adding new description functions or extending the proposed ones could lead to further improved controllability and sample quality.

Original: Melody conditioning stands out as a useful attribute still lacking control in FIGARO.
Revised: Melody conditioning stands out as a useful attribute still lacking control in FIGARO.

Original: Lastly, applying the description-to-sequence objective to other domains such as natural language could potentially enable human-interpretable control in a way that is not currently possible.
Revised: Lastly, applying the description-to-sequence objective to other domains such as natural language could potentially enable human-interpretable control in a way that is not currently possible.

Original: For example, the long-range conditioning ability could be interesting in the setting of story generation.
Revised: For example, the long-range conditioning ability could be interesting in the setting of story generation.

Original: Conclusion.
Revised: Conclusion.

Original: We present the self-supervised description-to-sequence objective and apply it in the context of symbolic music generation using expert and learned features.
Revised: We present the self-supervised description-to-sequence objective and apply it in the context of symbolic music generation using expert and learned features.

Original: We train and release FIGARO, which combines two description functions in a model that achieves human-interpretable, fine-grained control on multi-track, multi-time-signature music while beating state-of-the-art controllable generation models both in terms of sample quality and style transfer.
Revised: We train and release FIGARO, which combines two description functions in a model that achieves human-interpretable, fine-grained control on multi-track, multi-time-signature music while beating state-of-the-art controllable generation models both in terms of sample quality and style transfer.

Original: The proposed description functions complement each other: The expert description is a human-interpretable sequence that is easy to create and acts as a strong inductive bias to the model.
Revised: The proposed description functions complement each other: The expert description is a human-interpretable sequence that is easy to create and acts as a strong inductive bias to the model.

Original: The learned description is able to mitigate non-injectivity present in the expert description and amend it with high-fidelity, detailed information.
Revised: The learned description is able to mitigate non-injectivity present in the expert description and amend it with high-fidelity, detailed information.

Original: In the case of conflicting descriptions, the model still produces plausible samples, thus preserving human-interpretability even in the presence of learned features.
Revised: In the case of conflicting descriptions, the model still produces plausible samples, thus preserving human-interpretability even in the presence of learned features.

Original: We can combine both descriptions to achieve state-of-the-art symbolic music generation with human-interpretable, fine-grained control.
Revised: We can combine both descriptions to achieve state-of-the-art symbolic music generation with human-interpretable, fine-grained control.

Original: On a broader perspective, we hope that the proposed method is a step toward facilitating artists in their creative process as well as enabling amateurs to express themselves by lowering the barrier of entry to music creation and making the process faster and easier overall.
Revised: On a broader perspective, we hope that the proposed method is a step toward facilitating artists in their creative process as well as enabling amateurs to express themselves by lowering the barrier of entry to music creation and making the process faster and easier overall.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Automatic music generation may raise ethical concerns similar to those of large language models.
Revised: Automatic music generation may raise ethical concerns similar to those of large language models.

Original: The models may exhibit biases toward a certain style of music and may not represent the music of marginalized cultures accurately.
Revised: The models may exhibit biases toward a certain style of music and may not represent the music of marginalized cultures accurately.

Original: In our case, the majority of training samples are western music, which is not necessarily representative of music at large.
Revised: In our case, the majority of training samples are western music, which is not necessarily representative of music at large.

Original: The model might also reproduce copyrighted material that is present in the training data and potentially generate samples that infringe on copyright law.
Revised: The model might also reproduce copyrighted material that is present in the training data and potentially generate samples that infringe on copyright law.

Original: Additionally, powerful music model capable of competing with human composers have the potential to create a big impact on the music industry.
Revised: Additionally, powerful music model capable of competing with human composers have the potential to create a big impact on the music industry.

Original: In this regard, we believe that our contribution is a step toward human-AI collaboration rather than competition
Revised: In this regard, we believe that our contribution is a step toward human-AI collaboration rather than competition

Original: A I MPLEMENTATION D ETAILS A. D ESCRIPTION - TO -S EQUENCE M ODEL
Revised: A I MPLEMENTATION D ETAILS A. D ESCRIPTION - TO -S EQUENCE M ODEL

Original: We largely follow the original paper for both of our Transformer auto-encoders.
Revised: We largely follow the original paper for both of our Transformer auto-encoders.

Original: Unless specified differently, hyperparameters are left unchanged.
Revised: Unless specified differently, hyperparameters are left unchanged.

Original: One exception is the reduced context size oftokens for the sake of training time.
Revised: One exception is the reduced context size oftokens for the sake of training time.

Original: In the same spirit, we also reduce the number of encoder layers to 4 but leave the number of decoder layers at 6.
Revised: In the same spirit, we also reduce the number of encoder layers to 4 but leave the number of decoder layers at 6.

Original: We use relative positional embeddings (Huang et al., 2020) as this has been shown to be beneficial for symbolic music generation (Huang et al., 2018).
Revised: We use relative positional embeddings (Huang et al., 2020) as this has been shown to be beneficial for symbolic music generation (Huang et al., 2018).

Original: In addition to the positional embeddings, we also add a learned bar embedding and a learned beat-position embedding.
Revised: In addition to the positional embeddings, we also add a learned bar embedding and a learned beat-position embedding.

Original: This puts a limit on the maximum number of bars (512 in our case).
Revised: This puts a limit on the maximum number of bars (512 in our case).

Original: With a 4 / 4 time signature at 120 BPM this equates to 17 minutes of music, which we deem an acceptable limitation.
Revised: With a 4 / 4 time signature at 120 BPM this equates to 17 minutes of music, which we deem an acceptable limitation.

Original: In the case of using both descriptions, we simply add their token embeddings before passing it to the encoder.
Revised: In the case of using both descriptions, we simply add their token embeddings before passing it to the encoder.

Original: In total, the description-to-sequence model has 44.6 M trainable parameters.
Revised: In total, the description-to-sequence model has 44.6 M trainable parameters.

Original: In the presence of the learned description F learned , the discrete latent codes from the VQ-VAE feature extractor are embedded using the frozen codebook from the VQ-VAE instead of re-learning new embeddings.
Revised: In the presence of the learned description F learned , the discrete latent codes from the VQ-VAE feature extractor are embedded using the frozen codebook from the VQ-VAE instead of re-learning new embeddings.

Original: This is done to ensure stable training and reduce the number of trainable parameters.
Revised: This is done to ensure stable training and reduce the number of trainable parameters.

Original: A. VQ-VAE M ODEL
Revised: A. VQ-VAE M ODEL

Original: For our bar-level VQ-VAE model, we use the same Transformer auto-encoder backbone as for the description-to-sequence model.
Revised: For our bar-level VQ-VAE model, we use the same Transformer auto-encoder backbone as for the description-to-sequence model.

Original: The final layer of the encoder is pooled as proposed by Devlin et al.
Revised: The final layer of the encoder is pooled as proposed by Devlin et al.

Original: and then fed through a vector-quantization bottleneck, quantizing it to 16 latent codes from a codebook of size 2048.
Revised: and then fed through a vector-quantization bottleneck, quantizing it to 16 latent codes from a codebook of size 2048.

Original: We use a modified version of the sliced vector quantization scheme proposed by Kaiser et al.
Revised: We use a modified version of the sliced vector quantization scheme proposed by Kaiser et al.

Original: as our discretization bottleneck.
Revised: as our discretization bottleneck.

Original: Specifically, we decompose the latent representation z into 16 slices z 1 , . . .
Revised: Specifically, we decompose the latent representation z into 16 slices z 1 , . . .

Original: , z 16 and discretize each of them to a shared codebook
Revised: , z 16 and discretize each of them to a shared codebook

Original: C with |C| = 2048 using the k -means discretization technique from the original paper (Oord et al., 2018).
Revised: C with |C| = 2048 using the k -means discretization technique from the original paper (Oord et al., 2018).

Original: The latent vector is provided to the decoder through cross-attention.
Revised: The latent vector is provided to the decoder through cross-attention.

Original: We use a linear layer before and after vector-quantization to project between model space and latent space.
Revised: We use a linear layer before and after vector-quantization to project between model space and latent space.

Original: The VQ-VAE model has 43.7 M trainable parameters in total.
Revised: The VQ-VAE model has 43.7 M trainable parameters in total.

Original: The model is trained on bar-level reconstruction by minimizing the canonical β -VQ-VAE loss without the auxiliary codebook loss (Oord et al., 2019) given by
Revised: The model is trained on bar-level reconstruction by minimizing the canonical β -VQ-VAE loss without the auxiliary codebook loss (Oord et al., 2019) given by

Original: [Equation]
Revised: [Equation]

Original: where sg( x ) denotes the stop-gradient operator, ϕ are the model parameters and β = 0 .
Revised: where sg( x ) denotes the stop-gradient operator, ϕ are the model parameters and β = 0 .

Original: The codebook is updated using the EMA update step as proposed in the original paper.
Revised: The codebook is updated using the EMA update step as proposed in the original paper.

Original: We employ random restarts (Dhariwal et al., 2020) to ensure optimal codebook usage.
Revised: We employ random restarts (Dhariwal et al., 2020) to ensure optimal codebook usage.

Original: Bars with more tokens than the context size are truncated to fit.
Revised: Bars with more tokens than the context size are truncated to fit.

Original: A. T RAINING D ETAILS
Revised: A. T RAINING D ETAILS

Original: We train each model for 100k steps with a batch size of 512 sequences.
Revised: We train each model for 100k steps with a batch size of 512 sequences.

Original: Models are optimized using the Adam optimizer (Kingma & Ba, 2017) with β 1 = 0 .
Revised: Models are optimized using the Adam optimizer (Kingma & Ba, 2017) with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , ϵ = 10 − 6 and 0 .
Revised: 999 , ϵ = 10 − 6 and 0 .

Original: 01 weight decay.
Revised: 01 weight decay.

Original: We use the inverse-square-root learning rate schedule with initial constant warmup at 10 −given by 10 − 4 / max(1 , (cid:112) n/N ) where N = 4000 is the number of warmup steps.
Revised: We use the inverse-square-root learning rate schedule with initial constant warmup at 10 −given by 10 − 4 / max(1 , (cid:112) n/N ) where N = 4000 is the number of warmup steps.

Original: For additional details and hyperparameters, we refer to the source code.
Revised: For additional details and hyperparameters, we refer to the source code.

Original: B REMI+ I NPUT R EPRESENTATION
Revised: B REMI+ I NPUT R EPRESENTATION

Original: We extend the original REMI representation (Huang & Yang, 2020) to make it suitable for general multi-track, multi-signature symbolic music sequences.
Revised: We extend the original REMI representation (Huang & Yang, 2020) to make it suitable for general multi-track, multi-signature symbolic music sequences.

Original: We make modifications that add time signature and instrument information, we determine a unique order of events and use quantization schemes that allow for accurate representation for a diverse set of music.
Revised: We make modifications that add time signature and instrument information, we determine a unique order of events and use quantization schemes that allow for accurate representation for a diverse set of music.

Original: An example of a REMI+ sequence is given in Figure 5.
Revised: An example of a REMI+ sequence is given in Figure 5.

Original: Bar Tokens.
Revised: Bar Tokens.

Original: To provide the model with additional context, we include the index of the current bar in each bar token.
Revised: To provide the model with additional context, we include the index of the current bar in each bar token.

Original: This, along with the bar embedding, should help the model retrieve the correct information from the description and help determine the end of a piece at generation time.
Revised: This, along with the bar embedding, should help the model retrieve the correct information from the description and help determine the end of a piece at generation time.

Original: Time signature.
Revised: Time signature.

Original: We add a time signature token at the beginning of each bar, indicating the time signature of the bar which is to follow.
Revised: We add a time signature token at the beginning of each bar, indicating the time signature of the bar which is to follow.

Original: We adapt the convention that time signature changes may only happen at the beginning of a bar, which is commonly true in written music.
Revised: We adapt the convention that time signature changes may only happen at the beginning of a bar, which is commonly true in written music.

Original: Instruments.
Revised: Instruments.

Original: We add instrument information as an additional token before each note event, indicating which instrument will play the following note.
Revised: We add instrument information as an additional token before each note event, indicating which instrument will play the following note.

Original: Order of events.
Revised: Order of events.

Original: In theory, the order of notes within each bar can be arbitrary without compromising the validity of the sequence.
Revised: In theory, the order of notes within each bar can be arbitrary without compromising the validity of the sequence.

Original: But to make the modelling task easier, we define a unique and deterministic order of events.
Revised: But to make the modelling task easier, we define a unique and deterministic order of events.

Original: Specifically, we sort events by (Bar, Position, EventType, Instrument, Pitch) in ascending order (valid event types are { Chord, Tempo, Note } ).
Revised: Specifically, we sort events by (Bar, Position, EventType, Instrument, Pitch) in ascending order (valid event types are { Chord, Tempo, Note } ).

Original: This order is unique since a given instrument can only ever play a single note with a given pitch at a given time 8 .
Revised: This order is unique since a given instrument can only ever play a single note with a given pitch at a given time 8 .

Original: Quantization.
Revised: Quantization.

Original: We largely follow Huang & Yang (2020) in quantization.
Revised: We largely follow Huang & Yang (2020) in quantization.

Original: The most significant deviation is the use of 12 note onset positions per quarter note instead of 4 as proposed in the original work.
Revised: The most significant deviation is the use of 12 note onset positions per quarter note instead of 4 as proposed in the original work.

Original: For example, there will be 48 unique note onset positions for the 4 / 4 time signature andnote onset positions for the 3 / 4 time signature.
Revised: For example, there will be 48 unique note onset positions for the 4 / 4 time signature andnote onset positions for the 3 / 4 time signature.

Original: This allows both triplet and sixteenth notes to be quantized accurately, which is important when considering a diverse set of music.
Revised: This allows both triplet and sixteenth notes to be quantized accurately, which is important when considering a diverse set of music.

Original: Instruments and note pitches are not quantized as they are categorical variables with 128 possible values by the MIDI specification.
Revised: Instruments and note pitches are not quantized as they are categorical variables with 128 possible values by the MIDI specification.

Original: Note velocity is quantized to 32 intervals in [0 , 128] and note duration is quantized to position intervals defined by the following mesh: 
Revised: Note velocity is quantized to 32 intervals in [0 , 128] and note duration is quantized to position intervals defined by the following mesh: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Revised: { 48 +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: { 192 +
Revised: { 192 +

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] This ensures single position accuracy up to quarter notes, then 16 th and triplet accuracy up to half notes and 8 th note accuracy up to a full note.
Revised: This ensures single position accuracy up to quarter notes, then 16 th and triplet accuracy up to half notes and 8 th note accuracy up to a full note.

Original: To limit vocabulary size, we switch to quarter note steps up to 8 full notes and half note steps up to 16 full notes after that.
Revised: To limit vocabulary size, we switch to quarter note steps up to 8 full notes and half note steps up to 16 full notes after that.

Original: Notes longer than 16 full notes are truncated to this length.
Revised: Notes longer than 16 full notes are truncated to this length.

Original: Finally, tempo change events are discretized to 32 intervals in [0 , 240] .
Revised: Finally, tempo change events are discretized to 32 intervals in [0 , 240] .

Original: [Figure]
Revised: [Figure]

Original: C E XPERT D ESCRIPTION A LGORITHM
Revised: C E XPERT D ESCRIPTION A LGORITHM

Original: Pseudocode for generating the expert description is given in Algorithm 1.
Revised: Pseudocode for generating the expert description is given in Algorithm 1.

Original: We quantize note density, mean pitch and velocity to 32 linearly spaced intervals in [0 , 12] , [0 , 128] and [0 , 128] respectively.
Revised: We quantize note density, mean pitch and velocity to 32 linearly spaced intervals in [0 , 12] , [0 , 128] and [0 , 128] respectively.

Original: Mean duration is quantized to 32 logarithmically spaced intervals in [0 , 128] positions (12 positions per quarter note).
Revised: Mean duration is quantized to 32 logarithmically spaced intervals in [0 , 128] positions (12 positions per quarter note).

Original: Chords are extracted with an adapted version of the Viterbi algorithm also used by
Revised: Chords are extracted with an adapted version of the Viterbi algorithm also used by

Original: Huang & Yang (2020).
Revised: Huang & Yang (2020).

Original: Algorithm input musical sequence x output description d d ← () b 1 , . ..
Revised: input musical sequence x output description d d ← () b 1 , . .

Original: , b n ← P ARTITION
Revised: , b n ← P ARTITION

Original: I NTO B
Revised: I NTO B

Original: ARS ( x ) for b i ∈ ( b 1 , . . .
Revised: ARS ( x ) for b i ∈ ( b 1 , . . .

Original: , b n ) do
Revised: , b n ) do

Original: N ← { n | n is a note with onset in b i }
Revised: N ← { n | n is a note with onset in b i }

Original: I ← { inst | inst is being played during b i }
Revised: I ← { inst | inst is being played during b i }

Original: C ← { chord | chord is being played during b i } q ← duration of b i in quarter notes ts ← time signature at beginning of b i nd ← | N | q mp ← 1 | N | (cid:80) n ∈ N P ITCH ( n ) mv ← 1 | N | (cid:80) n ∈ N V ELOCITY ( n ) md ← 1 | N | (cid:80) n ∈ N D URATION ( n )
Revised: C ← { chord | chord is being played during b i } q ← duration of b i in quarter notes ts ← time signature at beginning of b i nd ← | N | q mp ← 1 | N | (cid:80) n ∈ N P ITCH ( n ) mv ← 1 | N | (cid:80) n ∈ N V ELOCITY ( n ) md ← 1 | N | (cid:80) n ∈ N D URATION ( n )

Original: Quantize nd , mp , mv and md d i ← ( i, ts , nd , mp , mv , md )
Revised: Quantize nd , mp , mv and md d i ← ( i, ts , nd , mp , mv , md )

Original: ∥ list( I )
Revised: ∥ list( I )

Original: ∥ list( C ) d ← d ∥ d i end for return d
Revised: ∥ list( C ) d ← d ∥ d i end for return d

Original: D E VALUATION M ETRICS D. M ACRO O VERLAPPING A REA
Revised: D E VALUATION M ETRICS D. M ACRO O VERLAPPING A REA

Original: As used by previous work, the overlapping area (OA) metric does not consider the sequential order of the investigated feature, as feature histograms are computed over the entire sequence.
Revised: As used by previous work, the overlapping area (OA) metric does not consider the sequential order of the investigated feature, as feature histograms are computed over the entire sequence.

Original: For example, the overlapping area of x and reverse( x ) would be maximal, even though the reversed sequence does not sound like the original sequence in general.
Revised: For example, the overlapping area of x and reverse( x ) would be maximal, even though the reversed sequence does not sound like the original sequence in general.

Original: To alleviate this limitation, we adapt the OA metric to also consider temporal order.
Revised: To alleviate this limitation, we adapt the OA metric to also consider temporal order.

Original: Let x
Revised: Let x

Original: and y denote two musical sequences and let b ( x ) i and b ( y )
Revised: and y denote two musical sequences and let b ( x ) i and b ( y )

Original: i denote the i -th bar of x and y respectively.
Revised: i denote the i -th bar of x and y respectively.

Original: We compute the overlap in feature distributions for each bar by fitting a Gaussian distribution to the feature under examination (e.g. note pitch) and compute the overlapping area between the two distributions.
Revised: We compute the overlap in feature distributions for each bar by fitting a Gaussian distribution to the feature under examination (e.g. note pitch) and compute the overlapping area between the two distributions.

Original: Let this overlap be given by overlap( b ( x ) i , b ( y ) i ) .
Revised: Let this overlap be given by overlap( b ( x ) i , b ( y ) i ) .

Original: Then the macro overlapping area (MOA) between x and y is given by
Revised: Then the macro overlapping area (MOA) between x and y is given by

Original: [Equation]
Revised: [Equation]

Original: D. N ORMALIZED R OOT -M EAN -S QUARE E RROR
Revised: D. N ORMALIZED R OOT -M EAN -S QUARE E RROR

Original: In order to normalize for different feature magnitudes between different samples, we compute the normalized RMSE (NRMSE) for bar-wise note density.
Revised: In order to normalize for different feature magnitudes between different samples, we compute the normalized RMSE (NRMSE) for bar-wise note density.

Original: Let x denote the ground truth, ˆ x denote the reconstruction and N denote the length of the sequences.
Revised: Let x denote the ground truth, ˆ x denote the reconstruction and N denote the length of the sequences.

Original: Then the NRMSE is given by
Revised: Then the NRMSE is given by

Original: [Equation]
Revised: [Equation]

Original: D. C OSINE S IMILARITY
Revised: D. C OSINE S IMILARITY

Original: Let v ( x ) i and v ( y ) i denote the chroma vector (Fujishima, 1999) or grooving vector (Dixon et al., 2004) for the i -th bar in x and y respectively.
Revised: Let v ( x ) i and v ( y ) i denote the chroma vector (Fujishima, 1999) or grooving vector (Dixon et al., 2004) for the i -th bar in x and y respectively.

Original: We then average the cosine similarity over the entire sequence
Revised: We then average the cosine similarity over the entire sequence

Original: [Table]
Revised: [Table]

Original: to get the chroma/grooving similarity:
Revised: to get the chroma/grooving similarity:

Original: [Table]
Revised: [Table]

Original: E B ENCHMARK M ODELS E. H UANG ET AL . (2018)
Revised: E B ENCHMARK M ODELS E. H UANG ET AL . (2018)

Original: We reimplement the model proposed by Huang et al.
Revised: We reimplement the model proposed by Huang et al.

Original: (2018) as an unconditional baseline.
Revised: (2018) as an unconditional baseline.

Original: We train the model on the REMI+ representation to allow for direct comparison to our method.
Revised: We train the model on the REMI+ representation to allow for direct comparison to our method.

Original: We use the improved relative attention from Huang et al.
Revised: We use the improved relative attention from Huang et al.

Original: (2020) to eliminate any possible advantage arising from different attention mechanisms.
Revised: (2020) to eliminate any possible advantage arising from different attention mechanisms.

Original: We largely stick to the hyperparameters from our models, using 6 decoder layers with a hidden size of 512 and a filter size of 2048.
Revised: We largely stick to the hyperparameters from our models, using 6 decoder layers with a hidden size of 512 and a filter size of 2048.

Original: Training and optimization hyperparameters are also the same as for our models.
Revised: Training and optimization hyperparameters are also the same as for our models.

Original: E.
Revised: E.

Original: [Equation]
Revised: [Equation]

Original: We reimplement the model proposed by Choi et al.
Revised: We reimplement the model proposed by Choi et al.

Original: and train it on the REMI+ representation to allow for direct comparison to our method.
Revised: and train it on the REMI+ representation to allow for direct comparison to our method.

Original: We again use the improved relative attention from
Revised: We again use the improved relative attention from

Original: Huang et al.
Revised: Huang et al.

Original: and largely stick to the hyperparameters from the original paper, using 6 encoder and 6 decoder layers.
Revised: and largely stick to the hyperparameters from the original paper, using 6 encoder and 6 decoder layers.

Original: Unlike the original work, we do not use data augmentation since the dataset is large enough and in order to allow for fair comparison between the models.
Revised: Unlike the original work, we do not use data augmentation since the dataset is large enough and in order to allow for fair comparison between the models.

Original: Due to GPU memory constraints we reduce the context size from 2048 to 1024 and use an accumulated batch size of 16, ensuring stable training.
Revised: Due to GPU memory constraints we reduce the context size from 2048 to 1024 and use an accumulated batch size of 16, ensuring stable training.

Original: Training and optimization hyperparameters are the same as for our models.
Revised: Training and optimization hyperparameters are the same as for our models.

Original: [Equation]
Revised: [Equation]

Original: We train MuseMorphose on the REMI+ representation to allow for direct comparison to our method.
Revised: We train MuseMorphose on the REMI+ representation to allow for direct comparison to our method.

Original: Adapting the released implementation for our experiments, we reduce the context size fromto 512 tokens due to GPU memory constraints but leave all other hyperparameters as they were proposed in the original paper.
Revised: Adapting the released implementation for our experiments, we reduce the context size fromto 512 tokens due to GPU memory constraints but leave all other hyperparameters as they were proposed in the original paper.

Original: The model is trained until convergence (approx.
Revised: The model is trained until convergence (approx.

Original: 125k steps).
Revised: 125k steps).

Original: We limit the training data to a subset of the entire dataset (20k samples) due to technical limitations.
Revised: We limit the training data to a subset of the entire dataset (20k samples) due to technical limitations.

Original: This is still considerably more training data than what was used in the original paper (1k samples) and should not affect performance significantly compared to using the full dataset.
Revised: This is still considerably more training data than what was used in the original paper (1k samples) and should not affect performance significantly compared to using the full dataset.

Original: F D ESCRIPTION M IXING
Revised: F D ESCRIPTION M IXING

Original: Quantitative evaluation for this task is not straight-forward as there are two different “ground truth” samples with potentially conflicting features and it is unclear, which one the model should adhere to for which features.
Revised: Quantitative evaluation for this task is not straight-forward as there are two different “ground truth” samples with potentially conflicting features and it is unclear, which one the model should adhere to for which features.

Original: It is clear though that the model follows one of the descriptions more closely for some features, which points to some degree of separation of concerns between the two descriptions.
Revised: It is clear though that the model follows one of the descriptions more closely for some features, which points to some degree of separation of concerns between the two descriptions.

Original: Exact numbers are provided in Table 4.
Revised: Exact numbers are provided in Table 4.

Original: [Table]
Revised: [Table]

Original: G S UBJECTIVE E VALUATION
Revised: G S UBJECTIVE E VALUATION

Original: The listening study includes 7569 comparisons by 691 participants, each averaging 11 answers.
Revised: The listening study includes 7569 comparisons by 691 participants, each averaging 11 answers.

Original: In each comparison, participants were presented with two different samples and were asked to indicate, which of the two they preferred.
Revised: In each comparison, participants were presented with two different samples and were asked to indicate, which of the two they preferred.

Original: For the following ranking test, samples were chosen uniformly at random from two different generative models, the pair of which again was chosen at random.
Revised: For the following ranking test, samples were chosen uniformly at random from two different generative models, the pair of which again was chosen at random.

Original: In this setup, we treat real samples as one possible model, for which we simply sample the test distribution.
Revised: In this setup, we treat real samples as one possible model, for which we simply sample the test distribution.

Original: For the other models, we generate samples as described for the style transfer experiment (Section 6.2).
Revised: For the other models, we generate samples as described for the style transfer experiment (Section 6.2).

Original: Screenshots from the study are presented in Figure 6.
Revised: Screenshots from the study are presented in Figure 6.

Original: We apply the Wilcoxon signed-rank test to the results in order to establish a ranking of the different models.
Revised: We apply the Wilcoxon signed-rank test to the results in order to establish a ranking of the different models.

Original: FIGARO beats each baseline except ground truth with a win rate of more than 60% and a p -value of less than 10 − 7 .
Revised: FIGARO beats each baseline except ground truth with a win rate of more than 60% and a p -value of less than 10 − 7 .

Original: Out of all methods, FIGARO also has the highest win rate against real samples with a win rate of 39 .
Revised: Out of all methods, FIGARO also has the highest win rate against real samples with a win rate of 39 .

Original: 3% , which is a 6% advantage over the next best method.
Revised: 3% , which is a 6% advantage over the next best method.

Original: Rankings and corresponding p -values are reported in Table 5.
Revised: Rankings and corresponding p -values are reported in Table 5.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]