Original: A UTOMATIC M USIC P RODUCTION U SING G ENERA -
Revised: A UTOMATIC M USIC P RODUCTION U SING G ENERA -

Original: TIVE A DVERSARIAL N ETWORKS
Revised: TIVE A DVERSARIAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: When talking about computer-based music generation, two are the main threads of research: the construction of autonomous music-making systems , and the design of computer-based environments to assist musicians .
Revised: When talking about computer-based music generation, two are the main threads of research: the construction of autonomous music-making systems , and the design of computer-based environments to assist musicians .

Original: Despite consistent demand from music producers and artists, however, little effort has been done in the ﬁeld of automatic music arrangement in the audio domain.
Revised: However, even though creating accompaniments for melodies is an essential part of every producer’s and songwriter’s work, little effort has been done in the ﬁeld of automatic music arrangement in the audio domain.

Original: In this work, we propose a novel framework for automatic music arrangement from raw audio in the frequency domain .
Revised: In this contribution, we propose a novel framework for automatic music accompaniment in the Mel-frequency domain .

Original: Using several songs converted into Mel-spectrograms – a two-dimensional time-frequency representation of audio signals – we were able to automatically generate original arrangements for both bass and voice lines.
Revised: Using several songs converted into Mel-spectrograms – a two-dimensional time-frequency representation of audio signals – we were able to automatically generate original arrangements for both bass and voice lines.

Original: Treating music pieces as images (Mel-spectrograms) allowed us to reformulate our problem as an unpaired image-to-image translation problem, and to tackle it with CycleGAN, a well-established framework.
Revised: Treating music pieces as images (Mel-spectrograms) allowed us to reformulate our problem as an unpaired imageto-image translation problem, and to tackle it with CycleGAN, a well-established framework.

Original: Moreover, the choice to deploy raw audio and Mel spectrograms enabled us to more effectively model long-range dependencies, to better represent how humans perceive music, and to potentially draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century.
Revised: Moreover, the choice to deploy raw audio and Mel-spectrograms enabled us to more effectively model long-range dependencies, to better represent how humans perceive music, and to potentially draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century.

Original: Our approach was tested on two different downstream tasks: given a bass line creating credible and on-time drums, and given an acapella song arranging it to a full song.
Revised: Our approach was tested on two different downstream tasks: given a bass line creating credible and on-time drums, and given an acapella song arranging it to a full song.

Original: In absence of an objective way of evaluating the output of music generative systems, we also deﬁned a possible metric for the proposed task, partially based on human (and expert) judgement.
Revised: In absence of an objective way of evaluating the output of music generative systems, we also deﬁned a possible metric for the proposed task, partially based on human (and expert) judgement.

Original: To the best of our knowledge, we are the ﬁrst to address the music arrangement task in the audio domain, to treat music pieces as images, and to propose a quantitative approach to evaluate the model results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The development of home music production has brought signiﬁcant innovations into the process of pop music composition.
Revised: The development of home music production has brought signiﬁcant innovations into the process of pop music composition.

Original: Software like Pro Tools, Cubase, and Logic – as well as MIDI-based technologies and digital instruments – allow artists and producers toeasily manipulate recordings and create high quality songs directly from home.
Revised: Software like Pro Tools, Cubase, and Logic – as well as MIDI-based technologies and digital instruments – provide a wide set of tools to manipulate recordings and simplify the composition process for artists and producers.

Original: After recording a melody, maybe with the aid of a guitar or a piano, song writers can now start building up the arrangement one piece at a time, sometimes not even needing professional musicians or proper music training.
Revised: After recording a melody, maybe with the aid of a guitar or a piano, song writers can now start building up the arrangement one piece at a time, sometimes not even needing professional musicians or proper music training.

Original: As a result, singers and song writers – as well as producers – have started asking for tools that could facilitate, or to some extent even automate, the creation of full songs around their lyrics and melodies.
Revised: As a result, singers and song writers – as well as producers – have started asking for tools that could facilitate, or to some extent even automate, the creation of full songs around their lyrics and melodies.

Original: To meet this new demand, the goal of designing computer-based environments to assist human musicians has become central in the ﬁeld of automatic music generation (Briot et al., 2020).
Revised: To meet this new demand, the goal of designing computer-based environments to assist human musicians has become central in the ﬁeld of automatic music generation (Briot et al., 2020).

Original: IRCAM OpenMusic (Assayag et al., 1999), Sony CSL-Paris FlowComposer (Papadopoulos et al., 2016), and 
Revised: IRCAM OpenMusic (Assayag et al., 1999), Sony CSL-Paris FlowComposer (Papadopoulos et al., 2016), and Logic Pro X Easy

Original: Logic Pro X Easy Drummer are just some examples.
Revised: Drummer are just some examples.

Original: In addition, more solutions based on deep learning techniques, such as RL-Duet (Jiang et al., 2020) – a deep reinforcement learning algorithm for online accompaniment generation – or PopMAG, a transformer-based architecture which relies on a Multi-track MIDI representation of music (Ren et al., 2020), continue to be studied. 
Revised: In addition, more solutions based on deep learning techniques, such as RL-Duet (Jiang et al., 2020) – a deep reinforcement learning algorithm for online accompaniment generation – or PopMAG, a transformer-based architecture which relies on a multi-track MIDI representation of music (Ren et al., 2020), continue to be studied. A comprehensive review of the most relevant deep learning techniques applied to music is provided by (Briot et al., 2020).

Original: Most of these strategies, however, suffer from the same critical issue, which makes them less appealing in view of music production for commercial purposes: they rely on a symbolic/MIDI representation of music.
Revised: Most of these strategies, however, suffer from the same critical issue, which makes them less appealing in view of music production for commercial purposes: they rely on a symbolic/MIDI representation of music.

Original: The approach proposed in this paper, instead, is a ﬁrst attempt at automatically generating an euphonic arrangement (two or more sound patterns that produce a pleasing and har- monious piece of music) in the audio domain, given a musical sample encoded in a two-dimensional time-frequency representation (known as Mel-spectrogram).
Revised: The approach proposed in this paper, instead, is a ﬁrst attempt at automatically generating an euphonic arrangement (two or more sound patterns that produce a pleasing and harmonious piece of music) in the audio domain, given a musical sample encoded in a two-dimensional time-frequency representation (in particular, we opted for the Mel-spectrogram time-frequency representation).

Original: Although arrangement generation has been studied in the context of symbolic audio, indeed, switching to Mel-spectrograms allows us to preserve the sound heritage of other musical pieces (allowing operations such as sampling) and is more suitable for real-life cases, where voice, for instance, cannot be encoded in MIDI.
Revised: Al- though arrangement generation has been studied in the context of symbolic audio, indeed, switching to Mel-spectrograms allows us to preserve the sound heritage of other musical pieces (allowing operations such as sampling) and is more suitable for real-life cases, where voice, for instance, cannot be encoded in MIDI.

Original: We focused our attention on two different tasks of increasing difﬁculty: (i) given a bass line to create credible and on-time drums, and (ii) given the voice line, to output a new and euphonic musical arrangement.
Revised: We focused our attention on two different tasks of increasing difﬁculty: (i) given a bass line to create credible and on-time drums, and (ii) given the voice line, to output a new and euphonic musical arrangement.

Original: Incidentally, we found out that – for training samples – our model was able to reconstruct the original arrangement pretty well, even though no pairing among the Mel-spectrograms of the two domains was performed.
Revised: Incidentally, we found out that – for training samples – our model was able to reconstruct the original arrangement pretty well, even though no pairing among the Mel-spectrograms of the two domains was performed.

Original: By means of the Mel-spectrogram representation of music, we can consider the problem of automatically generating an arrangement or accompaniment for a speciﬁc musical sample equivalent to an image-to-image translation task.
Revised: By means of the Mel-spectrogram representation of music, we can consider the problem of automatically generating an arrangement or accompaniment for a speciﬁc musical sample equivalent to an image-to-image translation task.

Original: For instance, if we have the Mel-spectrogram of an acapella song, we may want to produce the Mel-spectrogram of the same song including a suitable arrangement.
Revised: For instance, if we have the Mel-spectrogram of an acapella song, we may want to produce the Mel-spectrogram of the same song including a suitable arrangement.

Original: To solve this task, we tested an unpaired image-to-image translation strategy known as CycleGAN (Zhu et al., 2017), which consists of translating an image from a source domain X to a target domain Y in the absence of paired examples, by training both the mapping from X to Y and from Y to X simultaneously, with the goal of minimizing a cycle consistency loss.
Revised: To solve this task, we tested an unpaired image-to-image translation strategy known as CycleGAN (Zhu et al., 2017), which consists of translating an image from a source domain X to a target domain Y in the absence of paired examples, by training both the mapping from X to Y and from Y to X simultaneously, with the goal of minimizing a cycle consistency loss.

Original: The aforementioned system was trained on 5s pop music samples (equivalent to 256 × 256 Melspectrograms) coming both from the Free Music Archive (FMA) dataset (Defferrard et al., 2016), and from the Demucs dataset (D ´ efossez et al., 2019).
Revised: The aforementioned system was trained on 5s pop music samples (equivalent to 256 × 256 Mel-spectrograms) coming both from the Free Music Archive (FMA) dataset (Defferrard et al., 2017; 2018), and from the Demucs dataset (D ´ efossez et al., 2019).

Original: The short sample duration does not affect the proposed methodology, at least with respect to the arrangement task we focus on, and inference can be performed also on full songs.
Revised: The short sample duration does not affect the proposed methodology, at least with respect to the arrangement task we focus on, and inference can be performed also on full songs.

Original: Part of the dataset was pre-processed ﬁrst, since the FMA songs lacked source separated channels (i.e. differentiated vocals, bass, drums, etc.).
Revised: Part of the dataset was pre-processed ﬁrst, since the FMA songs lack source separated channels (i.e. differentiated vocals, bass, drums, etc.).

Original: The required channels were extracted using Demucs (D ´ efossez et al., 2019).
Revised: The required channels were extracted using Demucs (D ´ efossez et al., 2019).

Original: The main innovations presented in this contribution are as follows: • treating music pieces as images, we developed a framework to automatically generate music arrangementfrom raw audio in the frequency domain, different from any other previous approach; • our approach is able to generate arrangements with low computational resources and limited inference time, if compared to other popular solutions for automatic music generation (Dhariwal et al., 2020); • we developed a metric – partially based on or correlated to human (and expert) judgement – to automatically evaluate the obtained results and the creativity of the proposed system, given the challenges of a quantitative assessment of music.
Revised: The main innovations presented in this contribution are as follows: (i.) treating music pieces as images, we developed a framework to automatically generate music arrangement in the Mel-frequency domain, different from any other previous approach; (ii.) our approach is able to generate arrangements with low computational resources and limited inference time, if compared to other popular solutions for automatic music generation (Dhariwal et al., 2020); (iii.) we developed a metric – partially based on or correlated to human (and expert) judgement – to automatically evaluate the obtained results and the creativity of the proposed system, given the challenges of a quantitative assessment of music.

Original: To the best of our knowledge, this is the ﬁrst work to face the automatic arrangement production task in the audio domain, thus in view of potential music production for commercial purpose, by leveraging a two-dimensional time-frequency representation.
Revised: To the best of our knowledge, this is the ﬁrst work to face the automatic arrangement production task in the audio domain by leveraging a two-dimensional time-frequency representation.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Automatic music generation. The interest surrounding automatic music generation has greatly increased in the last few years, as proven by the number of deep learning approaches proposed.
Revised: The interest surrounding automatic music generation, translation and arrangement has greatly increased in the last few years, as proven by the high numbers of solutions proposed – see (Briot et al., 2020) for a comprehensive and detailed survey. Here we present a brief overview of the key contributions both in symbolic and audio domain. Music generation & arrangement in the symbolic domain.

Original: Most ofthem aim at generating new music from scratch or at genre and instrument translation, and are based on Recurrent Neural Networks (Mehri et al., 2016; Docevski et al., 2018;
Revised: There is a very large body of research that uses a symbolic representation of music to perform music generation and arrangement. The following contributions used MIDI, piano rolls, chord and note names to feed several deep learning architectures and tackle different aspects of the music generation problem. In (Yang et al., 2017), CNNs are used for generating melody as a series of MIDI notes either from scratch, by following a chord sequence, or by conditioning on the melody of previous bars.

Original: Manzelli et al.,2018; Vasquez & Lewis, 2019; Jiang et al., 2019), Dilated Convolutional Neural Networks (Oord et al., 2016), Transformers (Dhariwal et al., 2020) or Generative Adversarial Networks (Dong et al., 2017;
Revised: In (Mangal et al., 2019; Jaques et al., 2016; Mogren, 2016; Makris et al., 2017), LSTM networks are used to generate musical notes, melodies, polyphonic music pieces, and long drum sequences, under constraints imposed by metrical rhythm information and a given bass sequence.

Original: Yang et al., 2017;Kumar et al., 2019; Oza et al., 2020).
Revised: The authors of (Yamshchikov & Tikhonov, 2017;

Original: Moreover, we can observe a great variety in terms of music domains (waveforms, spectrograms/frequency) and ways of representing music – such as raw audio, MIDI (Dong et al., 2017;
Revised: Roberts et al., 2018), instead, use VAE networks to generate melodies. In (Boulanger-Lewandowski et al., 2012), symbolic sequences of polyphonic music are modeled in a completely general pianoroll representation, while the authors of (Hadjeres & Nielsen, 2017) propose a novel architecture to generate melodies satisfying positional constraints in the style of the soprano parts of the J.S. Bach chorale harmonisations encoded in MIDI.

Original: Yang et al., 2017; Zhu et al., 2018; Manzelli et al., 2018), piano rolls (Docevski et al., 2018; Jiang et al., 2019), music sheets etc. In particular, raw audio generation is common to several sub-ﬁelds, such as speech synthesis and music generation (Zhao et al., 2020); whereas several successful so- lutions have been proposed for the speech synthesis task (Mehri et al., 2016;
Revised: In (Johnson, 2017), RNNs are used for prediction and composition of polyphonic music; in (Hadjeres et al., 2017), highly convincing chorales in the style of Bach were automatically generated using note names; (Lattner et al., 2018) added higher-level structure on generated, polyphonic music, whereas (Mao et al., 2018) designed an end-to-end generative model capable of composing music conditioned on a speciﬁc mixture of composer styles. The approach described in (Hawthorne et al., 2018), instead, relies on notes as an intermediate representation to a suite of models – namely, a transcription model based on a CNN and a RNN network (Hawthorne et al., 2017), a self-attention-based music language model (Huang et al., 2018) and a

Original: Oord et al., 2016;
Revised: WaveNet model (Oord et al., 2016) – capable of transcribing, composing, and synthesizing audio waveforms. Finally, (Zhu et al., 2018) proposes an end-to-end melody and arrangement generation framework, called XiaoIce Band, which generates a melody track with several accompaniments played by several types of instruments. As this extensive literature on music generation in the symbolic domain shows, a promising approach would be to work with symbolic music and then use state-of-the-art synthesizers to produce sounds. MIDI, music sheets and piano rolls, however, are not always easy to ﬁnd or produce.

Original: Prenger et al., 2019; Wang et al., 2019; Kaneko et al., 2019), the second one is characterized by a key bottleneck: modeling raw audio directly introduces extremely long-range dependencies, making it computationally challenging to learn the high-level semantics of  music (Dhariwal et al., 2020).
Revised: Moreover, many musicians and artists can not read music and would be more comfortable to work in a less formalized setting. Finally, state-of-the-art synthesizers, although increasingly indistinguishable from live recordings, can not yet reproduce the inﬁnite nuances of real voices and instruments. Conversely, raw audio representation could be more appealing for some creators given its ﬂexibility and little music competence required. Music generation & arrangement in the audio domain.

Original: Nevertheless, only raw audio representationcan produce, at least in the long run, appealing results in view of music production for artistic and commercial purposes.
Revised: Some of the most relevant approaches proposed so far in the ﬁeld of waveform music generation deal with raw audio representation in the time domain. Many of these approaches draw methods and ideas from the extensive literature on audio and speech synthesis. For instance, in (Prenger et al., 2019) a ﬂow-based network capable of generating high quality speech from mel-spectrograms is proposed, while in (Wang et al., 2019) the authors present a neural source-ﬁlter (NSF) waveform modeling framework that is straightforward to train and fast to generate waveforms.

Original: Some of the most relevant approaches proposed so far in the ﬁeld of music generation deal with raw audio representation in the time domain  (Oord et al., 2016; Mehri et al., 2016; Dhariwal et al., 2020; Bhave et al., 2019); nevertheless, due to the computational resources required to directly model long-range dependencies in the time domain, either short samples of music can be generated or complex and large architectures and long inference time are required.
Revised: In (Zhao et al., 2020) recent neural waveform synthesizers such as WaveNet, WaveG-low, and the neural-source-ﬁlter (NSF) models are compared. (Mehri et al., 2016) tested a model for unconditional audio generation based on generating one audio sample at a time, and (Bhave et al., 2019) applied Restricted Boltzmann Machine and LSTM architectures to raw audio ﬁles in the frequency domain in order to generate music. A fully probabilistic and autoregressive model, with the predictive distribution for each audio sample conditioned on all previous ones, is used in (Oord et al., 2016) to produce novel and often highly realistic musical fragments. (Manzelli et al., 2018) combined two types of music generation models, namely symbolic and raw audio models, to train a raw audio model based on the WaveNet architecture, but that incorporates the notes of the composition as a secondary input to the network.

Revised: Finally, in (Dhariwal et al., 2020) the authors tackled the long context of raw audio using a multi-scale VQ-VAE to compress it to discrete codes, and modeled such context through Sparse Transformers, in order to generate music with singing in the raw audio domain.

Revised: Nonetheless, due to the computational resources required to directly model long-range dependencies in the time domain, either short samples of music can be generated or complex and large architectures and long inference time are required.

Original: On the other hand, in (Vasquez & Lewis, 2019) a novel approach is discussed, which proves that long-range dependencies can be more tractably modelled in two-dimensional time-frequency representations such as Mel-spectrograms.
Revised: On the other hand, in (Vasquez & Lewis, 2019) a novel approach is discussed, which proves that long-range dependencies can be more tractably modelled in two-dimensional time-frequency representations such as Mel-spectrograms.

Original: Our work is precisely founded on this novel assumption, thus taking the best from the raw audio representation, while tackling the main issues induced by musical signals long-range dependencies thanks to the waveform-to-spectrograms conversion.
Revised: More precisely, the authors of this contribution designed a highly expressive probabilistic model and a multiscale generation procedure over Mel-spectrograms capable of generating high-ﬁdelity audio samples which capture structure at timescales. It is worth recalling, as well, that treating spectrograms as images is the current standard for many Music Information Retrieval tasks, such as music transcription (Sigtia et al., 2016) and chord recognition.

Original: Generative adversarial networks for music generation.
Revised: Generative adversarial networks for music generation.

Revised: Our work is precisely founded on this novel assumption, thus taking the best from the raw audio representation, while tackling the main issues induced by musical signals long-range dependencies thanks to the waveform-to-spectrograms conversion.

Original: Such two-dimensional representation of music paves the way to the application of several image processing techniques and image-to-image translation networks to carry out style transfer and arrangement generation (Isola et al., 2017; Zhu et al., 2017).
Revised: Such two-dimensional representation of music paves the way to the application of several image processing techniques and image-to-image translation networks to carry out style transfer and arrangement generation (Isola et al., 2017; Zhu et al., 2017).

Original: It is worth recalling that the application of GANs to music generation tasks is not new: in (Brunner et al., 2018), Generative Adversarial Networks are applied on symbolic music to perform music genre transfer; however, to the best of our knowledge, GANs have never been applied to raw audio in the frequency domain for music generation purposes.
Revised: It is worth recalling that the application of GANs to music generation tasks is not new: in (Brunner et al., 2018), Generative Adversarial Networks are applied on symbolic music to perform music genre transfer; however, to the best of our knowledge, GANs have never been applied to raw audio in the Mel-frequency domain for music generation purposes.

Original: As to the arrangement generation task, also in this case the large majority of approaches proposed in literature is based on symbolic representation of music: in (Ren et al., 2020), a novel Multi-track MIDI representation (MuMIDI) is presented, which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks by means of a Transformer-based architecture; in (Jiang et al., 2020), a deep reinforcement learning algorithm for online accompaniment generation is described.
Revised: As to the arrangement generation task, also in this case the large majority of approaches proposed in literature is based on symbolic representation of music: in (Ren et al., 2020), a novel Multi-track MIDI representation (MuMIDI) is presented, which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks by means of a Transformer-based architecture; in (Jiang et al., 2020), a deep reinforcement learning algorithm for online accompaniment generation is described.

Original: Automatic music generation challenges.

Original: Coming to the most relevant issues in the development of music generation systems, both the training and evaluation of such systems haven proven challenging, mainly because of the following reasons: (i) the available data sets for music generation tasks are challenging due to their inherent high-entropy (Dieleman et al., 2018), and (ii) the deﬁnition of an objective metric and loss is a common problem to generative models such as GANs: at now, generative models in the music domain are evaluated based on the subjective response of a pool of listeners, and just for the MIDI representation a set of simple musically informed objective metrics was proposed (Yang & Lerch, 2020).
Revised: Coming to the most relevant issues in the development of music generation systems, both the training and evaluation of such systems haven proven challenging, mainly because of the following reasons: (i) the available datasets for music generation tasks are challenging due to their inherent high-entropy (Dieleman et al., 2018), and (ii) the deﬁnition of an objective metric and loss is a common problem to generative models such as GANs: at now, generative models in the music domain are evaluated based on the subjective response of a pool of listeners, and just for the MIDI representation a set of simple musically informed objective metrics was proposed (Yang & Lerch, 2020).

Original:  M ETHOD 3. S OURCE S EPARATION FOR M USIC
Revised:  M ETHOD 3. S OURCE S EPARATION FOR M USIC

Original: We present a novel framework for automatic music arrangement generation using an adversarially trained deep learning model.
Revised: We present a novel framework for automatic music arrangement generation using an adversarially trained deep learning model.

Original: A key challenge to our approach is the scarce availability of music data featuring source separated channels (i.e. differentiated vocals, bass, drums, ...).
Revised: A key challenge to our approach is the scarce availability of music data featuring source separated channels (i.e. differentiated vocals, bass, drums, ...).

Original: To this end, we leverage Demucs by D´efossez et al., a freely available tool, which separates music into its generating sources. 
Revised: To this end, we leverage Demucs by D´efossez et al., a freely available tool, which separates music into its generating sources. Demucs features a U-NET encoder-decoder architecture with a bidirectional

Original: This solves the challenge of data availability and allows us to feed our model with the appropriate signals.
Revised: LSTM as middle hidden layer. In particular we used a pre-trained model made available by the original authors, consisting of 6 convolutional encoder and decoder blocks and a middle hidden size of length 3200 .

Original: It is worth noticing that results of this procedure are time equivariant, meaning that shifting the input mixtureby X samples will shift the output Y by the exact same amount.
Revised: Demucs is time–equivariant, meaning that shifts in the input mixture will cause a congruent shifts in the output. The model does not feature this property naturally, but it is achieved through a workaround (randomized equivariant stabilization) as explained by the original authors.

Original: While showing nice properties, at times this method produces noisy separations - with watered–down harmonics and traces of other instruments in the vocal segment - effectively hindering  later part of the pipeline.
Revised: Nonetheless, at times this method produces noisy separations – with watered-down harmonics and traces of other instruments in the vocal segment – effectively hindering the ability of later part of the pipeline to properly recognise and reconstruct accompaniment, of which harmonics are a critical part.

Revised: While better source-separation methods are available [Sota SDR = 5.85, Demucs SDR = 5.67], we chose to use Demucs because it was faster and easier to embed in our pipeline.

Revised: Moreover, for bass source separation it beats the state of the art [Sota SDR = 5.28, Demucs SDR = 6.21].

Revised: Finally, we were adamant about picking a tool using deep learning because this may open the possibility to build an end-to-end trained pipeline in the future.

Revised: This at least partially solves the challenge of data availability and allows us to feed our model with the appropriate signals.

Original: 3. M USIC R EPRESENTATION – FROM R AW A UDIO TO M EL - SPECTROGRAMS
Revised: 3. M USIC R EPRESENTATION – FROM R AW A UDIO TO M EL - SPECTROGRAMS

Original: One of the main features of our method is to choose a two-dimensional time-frequency representation of the audio samples rather than a time representation.
Revised: One of the main features of our method is to choose a two-dimensional time-frequency representation of the audio samples rather than a time representation.

Original: The spectrum is a common transformed representation for audio, obtained via a  Fourier transform.
Revised: The spectrum is a common transformed representation for audio, obtained via a Short-Time Fourier transform (STFT).

Original: Figure 1 shows a Mel-spectrogram example, a visual representation of a spectrum, where the  x axis represents time, the y axis represents the 
Revised: The discrete STFT of a given signal x : [0 : L − 1] := { 0 , 1 , . . . , L − 1 } → R leads to the k th complex Fourier coefﬁcient for the m th time frame X ( m, k ) : = (cid:80) N − 1 n =0 x ( n + mH ) · w ( n ) · e − 2 πiknN , with m ∈ [0 : M ] and K ∈ [0 : K ] , and where w ( n ) is a sampled window function of length N ∈ N and H ∈ N is the hop size, which determines the step size in which the window is to be shifted across the signal (M¨uller, 2015).

Original: Mel bins offrequencies and the third gray tone axis represents the intensity of the  sound measured in decibel (Briot et al., 2020).
Revised: The spectrogram is a two-dimensional representation of the squared magnitude of the STFT, i.e. Y ( m, k ) : = |X ( m, k ) | 2 , with m ∈ [0 : M ] and K ∈ [0 : K ] . Figure 1 shows a Mel-spectrogram example (Stevens et al., 1937), which is treated as single channel image, representing the sound intensity with respect to time – x axis – and frequency – y axis (Briot et al., 2020).

Original: This decision allows to better deal with long-range dependencies typical of such kind of data and to reduce the computational resources and inference time required.
Revised: This decision allows to better deal with long-range dependencies typical of such kind of data and to reduce the computational resources and inference time required.

Original: Moreover, the Mel scale is based on a mapping between  actual frequency  and perceived pitch  as the human auditory system does not perceive pitch in a linear manner.
Revised: Moreover, the Mel-scale is based on a mapping between the actual frequency f and perceived pitch m = 2595 · log 10 (1+ f 700 ) , as the human auditory system does not perceive pitch in a linear manner.

Original: Finally, using Mel spectrograms of pre-existing songs to train our model potentially enables to draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century.
Revised: Finally, using Mel-spectrograms of pre-existing songs to train our model potentially enables to draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century.

Original: Mel-frequency cepstral coefﬁcients are the dominant features used in speech recognition, as well as in some music modeling tasks (Logan & Robinson, 2001).

Original: After the source separation task was carried out on our song dataset, each source (and the full song) waveforms were turned into corresponding Mel-spectrograms. 
Revised: After the source separation task was carried out on our song dataset, each source (and the full song) waveforms were turned into corresponding Mel-spectrograms. This has been done using

Revised: PyTorch Audio 1 , to take advantage of robust, GPU accelerated conversion.

Revised: We decided to discard the phase information in this process, to reduce the dimensionality of the representation.

Revised: To revert back to the time-domain signal, we: (i.) apply a conversion matrix (using triangular ﬁlter banks) to convert the Mel-frequency STFT to a linear scale STFT, where the matrix is calculated using a gradient-based method (Decorsi`ere et al., 2015) to minimize the euclidean norm between the original

Revised: Mel-spectrogram and the product between reconstructed spectrogram and ﬁlter banks; (ii.) use the

Original: For the waveform-spectrogram conversion, the sampling rates r was initially set to 22050 Hz, the number of sampling points to calculate the discrete Fourier transform n fft to 2048, the number of Mel frequency binsn mels to 256 and thestep or stride between windows hop length to 512.
Revised: Grifﬁn-Lim’s algorithm (Grifﬁn & Jae Lim, 1984) to reconstruct phase information. It is worth noticing that Mel-scale conversion and the removal of STFT phases respectively discard frequency and temporal information, that results in a distortion in the recovered signal. To minimize this problem, we use a high-enough resolution of the Mel-spectrograms (Vasquez & Lewis, 2019), whose size can be tweaked with number of mels and STFT hop size parameters. Thus, the optimal parameters we found were the following ones: the sampling rate was initially set to 22050 Hz, the window length N to 2048, the number of Mel-frequency bins to 256 and the hop size H to 512.

Original: Finally, we cropped out 256 ×windows from each Mel-spectrograms with an overlapping of 50 time frames, obtaining multiple samples from each song (each equivalent to 5 seconds of music).
Revised: To ﬁt our model requirements, we cropped out 256 × 256 windows from each Mel-spectrogram with an overlapping of 50 time frames, obtaining multiple samples from each song (each equivalent toseconds of music).

Original: 3. I MAGE TO I MAGE T RANSLATION - C YCLE GAN
Revised: 3. I MAGE TO I MAGE T RANSLATION - C YCLE GAN

Original: The automatic arrangement generation task was faced through an unpaired image-to-image translation framework, by adapting the CycleGAN model1 to our purpose.
Revised: The automatic arrangement generation task was faced through an unpaired image-to-image translation framework, by adapting the CycleGAN model to our purpose.

Original: CycleGAN is a framework able to translate between domains without paired input-output examples, by assuming some underlying relationship between the domains and trying to learn that relationship.
Revised: CycleGAN is a framework able to translate between domains without paired input-output examples, by assuming some underlying relationship between the domains and trying to learn that relationship.

Original: Based on a set of images in domain X and a different set in domain Y , the algorithm learns both a mapping G : X → Y and a mapping F : Y → X , such that the output ˆ y
Revised: Based on a set of images in domain X and a different set in domain Y , the algorithm learns both a mapping G : X → Y and a mapping F : Y → X , such that the output ˆ y

Original: =
Revised: =

Original: G ( x ) for every x ∈ X , is indistinguishable from images y ∈ Y and ˆ x =
Revised: G ( x ) for every x ∈ X , is indistinguishable from images y ∈ Y and ˆ x =

Original: G ( y ) for every y ∈ Y , is indistinguishable from images x ∈ X .
Revised: G ( y ) for every y ∈ Y , is indistinguishable from images x ∈ X .

Original: The other relevant assumption is that, given a mapping G : X → Y and another mapping F : Y → X , then G and F should be inverses of each other, and both mappings should be bijections.
Revised: The other relevant assumption is that, given a mapping G : X → Y and another mapping F : Y → X , then G and F should be inverses of each other, and both mappings should be bijections.

Original: This assumption is implemented by training both the mapping G and F simultaneously, and adding a cycle consistency loss that encourages F ( G ( x ))
Revised: This assumption is implemented by training both the mapping G and F simultaneously, and adding a cycle consistency loss that encourages F ( G ( x ))

Original: ≈ x and G ( F ( y ))
Revised: ≈ x and G ( F ( y ))

Original: ≈
Revised: ≈

Original: y .
Revised: y .

Original: The cycle consistency loss is then combined with the adversarial losses on domains X and Y (Zhu et al., 2017).
Revised: The cycle consistency loss is then combined with the adversarial losses on domains X and Y (Zhu et al., 2017).

Original:  A UTOMATIC M USIC P RODUCTION
Revised: 3. A UTOMATIC M USIC P RODUCTION

Original: The method we propose takes as input a set of N music songs in the waveform domain X
Revised: The method we propose takes as input a set of N music songs in the waveform domain X

Original: = { x i } N i =1 , where x i is a waveform whose number of samples depends on the sampling rate and the audio length.
Revised: = { x i } N i =1 , where x i is a waveform whose number of samples depends on the sampling rate and the audio length.

Original: Each waveform is then separated by Demucs into three different sources.
Revised: Each waveform is then separated by Demucs into three different sources.

Original: Thus, we end up having four different WAV ﬁles for each song, which means a new set of data of the kind:
Revised: Thus, we end up having four different WAV ﬁles for each song, which means a new set of data of the kind:

Original: X NEW = { x i , v i , d i , b i } Ni =1 , where v i , b i , d i represents vocal, bass, and drums respectively.
Revised: X NEW = { x i , v i , d i , b i } Ni =1 , where v i , b i , d i represents vocal, bass, and drums respectively.

Original: Each track is then converted to its Mel-spectrogram representation.
Revised: Each track is then converted to its Mel-spectrogram representation.

Original: Since the CycleGAN model takes 256 × 256 images as input, each spectrogram is chunked into smaller pieces with an overlapping window of 50 time frames; ﬁnally, in order to obtain grayscale PNG images from the original spectrograms, we perform a discretization step in the range [0 − 255] .
Revised: Since the CycleGAN model takes 256 × 256 images as input, each spectrogram is chunked into smaller pieces with an overlapping window of 50 time frames; ﬁnally, in order to obtain one channel images from the original spectro- grams, we performed a discretization step in the range [0 − 255] .

Original: In the ﬁnal stage of our pipeline, we feed the obtained dataset to the CycleGan model, that has been adapted to the structure of this data.
Revised: In the ﬁnal stage of our pipeline, we feed the obtained dataset to the CycleGan model, that has been adapted to the structure of this data.

Original: Even though the discretization step introduces some distortion – original spectrogram values are ﬂoats – the impact on the audio quality is negligible.
Revised: Even though the discretization step introduces some distortion – original spectrogram values are ﬂoats – the impact on the audio quality is negligible.

Original: At training time,  we considered two different experimental settings. On the one hand, we take the vocals and the whole song respectively - as the model takes into account two domains X, Y - with the goal of generating an arrangement euphonic to the vocal line.
Revised: At training time, as the model takes into account two domains X and Y , we considered two different experimental settings of increasing difﬁculty: (i.) we feed the model with bass and drums lines in order to create suitable drums given a bass line; (ii.) we take the vocals and the whole song respectively, with the goal of generating an arrangement euphonic to the vocal line.

Original: On the other hand, in the second experimental setting we feed the model with bass and drums lines in order to create suitable drums given a bass line. For both tasks, we trained our model on 2 NVIDIA Tesla V100 SXM2 with 32 GB of RAM forepochs (FMA dataset), and ﬁne-tuned it for 20 more epochs (musdb18 dataset).
Revised: Solving the bass2drums task effectively would represent a ﬁrst interesting intermediary goal.

Original: Each task required 6 days of training time.
Revised: Drums and bass are usually the ﬁrst instruments to be recorded when producing a song.

Original: For both settings, as a ﬁnal step, the inverse procedure is applied, to convert back the spectrograms obtained to the waveform domain and  to evaluate the produced music.
Revised: Ideally, though, a system the automatically generates a full song given a voice line as input would be far more ambitious and disruptive because it would allow anyone to express her/himself in music.

Original:  E XPERIMENTS 4. D ATASET
Revised:  E XPERIMENTS 4. D ATASET

Original: For the quality of the generated music samples, it is important to carefully pick the training dataset.
Revised: For the quality of the generated music samples, it is important to carefully pick the training dataset.

Original: To train and test our model We decided to use the Free Music Archive (FMA), and the musdb dataset (MusDB18) that were both made available quite recently.
Revised: To train and test our model we decided to use the Free Music Archive (FMA), and the musdb dataset (Raﬁi et al., 2017) that were both made available quite recently.

Original: The Free Music Archive (FMA) is the largest publicly available data set suitable for music information retrieval tasks (Defferrard et al., 2016).
Revised: The Free Music Archive (FMA) is the largest publicly available dataset suitable for music information retrieval tasks (Defferrard et al., 2017; 2018).

Original: In its full form it provides 917 GB and 343 days of Creative Commons-licensed audio from 106,574 tracks from 16,341 artists and 14,854 albums, arranged in a hierarchical taxonomy of 161 unbalanced genres.
Revised: In its full form it provides 917 GB and 343 days of Creative Commonslicensed audio from 106,574 tracks from 16,341 artists and 14,854 albums, arranged in a hierarchical taxonomy of 161 unbalanced genres.

Original: It provides full-length and high-quality audio, pre-computed features, together with track- and user-level metadata, tags, and free-form text such as biographies.
Revised: It provides full-length and high-quality audio, pre-computed features, together with track- and user-level metadata, tags, and free-form text such as biographies.

Original: Given the size of FMA, we chose to select only pop music and its sub-genres, for a total of approximately 10,000 songs.
Revised: Given the size of FMA, we chose to select only untrimmed songs tagged as either pop, soul-RnB, or indie-rock, for a total of approximately 10,000 songs ( 700 hours of audio). It is possible to read the full list of songs at FMA website , selecting the genre.

Revised: We discarded all songs that were recorded live by ﬁltering out all albums that contained the word ”live” in the title.

Original: Finally, in order to better validate and ﬁne–tune our model we decided to also use the musdb dataset.
Revised: Finally, in order to better validate and ﬁne-tune our model we decided to also use the full musdb18 dataset.

Revised: This rather small dataset is made up of 100 tracks taken from the DSD100 dataset, 46 tracks from the MedleyDB,tracks kindly provided by Native Instruments, and 2 tracks from the Canadian rock band The Easton

Original: This rather small dataset is a unique and precious source of songs delivered in multi-track fashion.
Revised: Ellises. It represents a unique and precious source of songs delivered in multi-track fashion.

Original: Each song comes as 5 audio ﬁles – vocals, bass, drums, others, full song – perfectly separated at the master level.
Revised: Each song comes as 5 audio ﬁles – vocals, bass, drums, others, full song – perfectly separated at the master level.

Original: In the training set each song  is represented by up to 50 samples, depending on its length.
Revised: We used the 100 tracks taken from the DSD100 dataset to ﬁne-tune the model ( 6. hours), and the remaining 50 songs to test it ( 3.5 hours). It is worth noting that DEMUCS is not a perfect method for source separation: it introduces artefacts and noise of the original song in the separated sources output, making the task easier and inducing the model to learn to amplify it. For this reason our training strategy is to pre-train with the artiﬁcial FMA dataset then ﬁne-tune with musdb18. Intuitively, the former, which is much larger, helps the model to create a good representation of musical signal; the latter, which is of higher quality, contributes to reducing bias induced by the noise and to further specializing to generate a base relying only on the (clean) input given.

Revised: [Table]

Revised: 4. T RAINING OF THE C YCLE -GAN MODEL

Original: In the test set, instead, we chose only a few samples for each song due to the relative uniformity of its content: in other word, we expect our model to perform in similar ways ondifferent parts of the same song.
Revised: For both the bass2drums and voice2song tasks, we trained our model on 2 Tesla V100 SXM2 GPUs with 32 GB of RAM for 12 epochs (FMA dataset), and ﬁne-tuned it for 20 more epochs (musdb dataset). Each task required 6 days of training. For both the settings, as a ﬁnal step, the spectrograms obtained were converted to the waveform domain, to evaluate the produced music. As to the CycleGAN model used for training, we relied on the default network available at this GitHub repository .

Original: No single song is present in both datasets.
Revised: As a result, the model uses a resnet 9blocks ResNet generator and a basic 70x70 PatchGAN as a discriminator. The Adam optimizer (Kingma & Ba, 2014) was chosen both for the generators and the discriminators, with betas (0 . 5 , 0 . 999) and learning rate equal to 0 . 0002 . The batch size was set to 1. The λ weights for cycle losses were both equal to 10.

Original: 4. E XPERIMENTAL SETTING
Revised: 4. E XPERIMENTAL SETTING

Original: There is an intrinsic difﬁculty in objectively evaluating artistic artifacts such as music.
Revised: There is an intrinsic difﬁculty in objectively evaluating artistic artifacts such as music.

Revised: As a human construct, there are no objective, universal criteria for appreciating music.

Original: Many generative approaches to raw audio, such as Jukebox (Dhariwal et al., 2020; Mor et al., 2018), try to overcome this obstacle by having the results manually tagged by human experts.
Revised: Nevertheless, in order to establish some form of benchmark and allow comparisons among different approaches, many generative approaches to raw audio, such as Jukebox (Dhariwal et al., 2020) or Universal Music Translation Network (Mor et al., 2018), try to overcome this obstacle by having the results manually tagged by human experts.

Original: Although this rating may be the best in terms of quality, the result is  somehow subjective, thus different people may end up giving different or biased ratings based on their personal taste.
Revised: Although this rating may be the best in terms of quality, the result is still somehow subjective, thus different people may end up giving different or biased ratings based on their personal taste.

Original: Moreover, the computational cost and time required to manually annotate the dataset could become prohibitive even for relatively few samples (over 1000).
Revised: Moreover, the computational cost and time required to manually annotate the dataset could become prohibitive even for relatively few samples (over 1000).

Original: Accordingly, as part ofour contribution and taking inspiration from the song intelligibility score (Sharma & Wang, 2019), we propose a new metric that highly correlate with human judgment.
Revised: Aware of the great limits linked to this human-based approach and unable to ﬁnd a more convincing evaluation procedure, we propose a new metric that highly correlates with human judgment.

Original: This could represent a ﬁrst benchmark for the tasks at hand.
Revised: This could represent a ﬁrst benchmark for the tasks at hand.

Revised: The results remain somehow subjective, but at least we were able to automatically replicate our evaluators’ criteria and grades, saving time and money.

Original: 4. M ETRICS
Revised: 4. M ETRICS

Original: If we consider as a general objective for a system the capacity to assist composers and musicians, rather than to autonomously generate music, we should also consider as an evaluation criteria the satisfaction of the composer (notably, if the assistance of the computer allowed him to compose and create music that he may consider not having been possible otherwise), rather than the satisfaction of the auditors (who remain too often guided by some conformance to a current musical trend)
Revised: If we consider as a general objective for a system the capacity to assist composers and musicians, rather than to autonomously generate music, we should also consider as an evaluation criteria the satisfaction of the composer (notably, if the assistance of the computer allowed him to compose and create music that he may consider not having been possible otherwise), rather than the satisfaction of the auditors (who remain too often guided by some conformance to a current musical trend)

Original: (Briot et al., 2020).
Revised: (Briot et al., 2020).

Original: However, as previously stated, an exclusive human evaluation may be unsustainable in terms of computational cost and time required.
Revised: However, as previously stated, an exclusive human evaluation may be unsustainable in terms of computational cost and time required.

Original: Thus we carried out the following quantitative assessment of our model.
Revised: Thus we carried out the following quantitative assessment of our model.

Original: We ﬁrst produced 400 test samples – from as many different songs and authors – of artiﬁcial arrangements and drum lines starting from voice and bass lines that were not part of the training set.
Revised: We ﬁrst produced 400 test samples – from as many different songs and authors – of artiﬁcial arrangements and drum lines starting from voice and bass lines that were not part of the training set.

Original: We then asked two professional musicians and two  music producers with more than 4 years of experience to manually annotate these samples, capturing the following musical aspects: quality, euphony, coherence, intelligibility.
Revised: We then asked a professional guitarist who has been playing in a pop-rock band for more than 10 years, a professional drum player from the same band, and two pop and indie-rock music producers with more than 4 years of experience to manually annotate these samples, capturing the following musical dimensions: quality, euphony, coherence, intelligibility.

Original: More precisely, for each sample, we asked them to rate from 1 to 10 the following aspects: (i) Quality : a rating from 1 to 10 of the naturalness and absence of artifacts or noise, (ii) Contamination : a rating from 1 to 10 of the contamination by other sources, (iii) Credibility : a rating from 1 to 10 of the credibility of the sample, (iv) Time : a rating from 1 to 10 of whether the produced drums and arrangements are on time thethe bass and voice lines. 
Revised: More precisely, for each sample, we asked them to rate from 1 to 10 the following aspects: (i) Quality : a rating fromto 10 of the naturalness and absence of artifacts or noise, (ii) Contamination : a rating from 1 toof the contamination by other sources, (iii) Credibility : a rating from 1 to 10 of the credibility of the sample, (iv) Time : a rating from 1 to 10 of whether the produced drums and arrangements are on time the bass and voice lines. The choice fell on these four aspects after we asked the evaluators to list and describe the most relevant dimensions in the perceived quality of a piece of pop-rock music. The correlation matrix for all 4 annotators is shown in Table 1.

Original: Ideally, we want to produce some quantitative measure whose outputs – when applied to generated samples – highly correlates (i.e. predict) expert average grades.
Revised: Ideally, we want to produce some quantitative measure whose outputs – when applied to generated samples – highly correlates (i.e. predict) expert average grades.

Original: To achieve this goal, we trained a logistic regression model with features obtained through a comparison between the original arrangement and the model output, as well as the original drums and the artiﬁcial drums.
Revised: To achieve this goal, we trained a logistic regression model with features obtained through a comparison between the original arrangement and the model output, as well as the original drums and the artiﬁcial drums.

Original: Here are the details on how with obtained suitable features:
Revised: Here are the details on how we obtained suitable features:

Original: STOI-like features.
Revised: STOI-like features.

Original: We created a procedure – inspired by the STOI (Andersen et al., 2017) – whose output vector somehow measures the Mel frequency bins correlation throughout time between the original sample (arrangement/drums) and the fake one.
Revised: We created a procedure – inspired by the STOI (Andersen et al., 2017) – whose output vector somehow measures the Mel-frequency bins correlation throughout time between the original sample (arrangement/drums) and the fake one.

Original: The obtained vector can then be used to feed a multi regression model whose independent variable is the human score attributed to that sample.
Revised: The obtained vector can then be used to feed a multi regression model whose independent variable is the human score attributed to that sample.

Original: Here is the formalisation: 
Revised: Here is the formalisation: HumanScore = (cid:80) 256 i a

Original: [Equation] 
Revised: [Equation] [Equation]

Original: To simplify, to each pair of samples (original and generated one) a 256 element long vector is associated as follows:
Revised: To simplify, to each pair of samples (original and generated one) a 256 element long vector is associated as follows:

Original: [Equation]
Revised: S ( X , Y , l ) ( i ) = (cid:80) 256 t ( x ( t ) i − ¯ x ( t ) ) ( y ( t ) i − ¯ y ( t ) ) .

Original: Where:
Revised: Where:

Original: • X and Y are, respectively, the Mel-spectrogram matrices of original and generated samples; • a
Revised: (i.) X and Y are, respectively, the Mel-spectrogram matrices of original and generated samples; (ii.)

Original:  i is the i -th coefﬁcient for the linear regression; • x ( t ) i and y ( t ) i the i -th element of the t -th column of matrices X and Y , respectively; • ¯ x ( t ) and ¯ y ( t ) are the means along the t -th column of matrices X andY , respectively.
Revised: a i is the i -th coefﬁcient for the linear regression; (iii.) x ( t ) i and y ( t ) i the i -th element of the t -th column of matrices X and Y , respectively; (iv.) ¯ x ( t ) and ¯ y ( t ) are the means along the t -th column of matrices X and

Original:  Each feature i of the regression model is a sort of Pearson correlation coefﬁcient between row i of X and row i of Y throughout time.
Revised: Y , respectively. Each feature i of the regression model is a sort of Pearson correlation coefﬁcient between row i of X and row i of Y throughout time.

Original: FID-based features.
Revised: FID-based features.

Original: In the context of GANs result evaluation, the Fr´echet Inception distance (FID) is supposed to improve on the Inception Score by actually comparing the statistics of generated samples to real samples (Salimans et al., 2016; Heusel et al., 2017).
Revised: In the context of GANs result evaluation, the Fr ´ echet Inception distance (FID) is supposed to improve on the Inception Score by actually comparing the statistics of generated samples to real samples (Salimans et al., 2016; Heusel et al., 2017).

Original: In other words, FID measures the probabilistic distance between two multivariate Gaussians, where X r = N ( µ r , Σ r ) and
Revised: In other words, FID measures the probabilistic distance between two multivariate Gaussians, where X r = N ( µ r , Σ r ) and

Original: X g = N ( µ g , Σ g ) are the 2048-dimensional activations of the Inception-v3 pool3 layer – for real and generated samples respectively – modeled as normal distributions.
Revised: X g = N ( µ g , Σ g ) are the 2048-dimensional activations of the Inception-v3 pool3 layer – for real and generated samples respectively – modeled as normal distributions.

Original: The similarity between the two distributions is measured as follow: [Equation]
Revised: The similarity between the two distributions is measured as follow: FID = (cid:107) µ r − µ g (cid:107) 2 + Tr (Σ r + Σ g − 2(Σ r Σ g ) 1 / 2 ) .

Original: Nevertheless, since we want to assign a score to each sample, we just estimated the X r = N ( µ r , Σ r ) parameters – using different activation layers of the Inception pre-trained network – and then we calculated the probability density associated to each fake sample.
Revised: Nevertheless, since we want to assign a score to each sample, we just estimated the X r = N ( µ r , Σ r ) parameters – using different activation layers of the Inception pre-trained network – and then we calculated the probability density associated to each fake sample.

Original: Finally, we added these scores to the regression model predictors.
Revised: Finally, we added these scores to the regression model predictors.

Original: 4. E XPERIMENTAL R ESULTS
Revised: 4. E XPERIMENTAL R ESULTS

Original: For the bass2drums task, ﬁgure 3 shows the distribution of grades for the 400 test samples - averaged among all four independent evaluators and  rounded  to the closest integer.
Revised: For the bass2drums task, Figure 3 shows the distribution of grades for the 400 test samples – averaged among all four independent evaluators and over all the four dimensions. We rounded the results to the closest integer to make the plot more readable.

Original: The higher the grade, the better the sample will sound.
Revised: The higher the grade, the better the sample will sound.

Original: More precisely, after discussing the model results with the evaluators, we noticed that samples  with grade 1-3 are generally silent or very noisy.
Revised: Additionally, to fully understand what to expect from samples graded similarly, we discussed the model results with the evaluators. We collectively listened to a random set of samples and it turned out that all four raters followed similar principles in assigning the grades. Samples with grade 1-3 are generally silent or very noisy.

Original: In samples graded 4-5 few sounds start to emerge, but they are usually not very pleasant to listen to, nor coherent.
Revised: In samples graded 4-5 few sounds start to emerge, but they are usually not very pleasant to listen to, nor coherent.

Original: Grades 6identify drums that sound good, that are coherent, but that are not continuous: they tend to follow the bass line too closely.
Revised: Grades 6-7 identify drums that sound good, that are coherent, but that are not continuous: they tend to follow the bass line too closely.

Original: Finally, samples graded 8 and 9 are almost indistinguishable from real drums, both in terms of sound and timing.
Revised: Finally, samples graded 8 and 9 are almost indistinguishable from real drums, both in terms of sound and timing.

Original: In the labeling of non graded samples phase, we therefore assigned a 0 to those samples whose average grade was between 1 and 5, and 1 to those between 6 and 10.
Revised: In the labeling of non graded samples phase, we therefore assigned a 0 to those samples whose average grade was between 1 and 5, and 1 to those between 6 and 10.

Original: Finally, we trained a multilogistic regression model with both the STOI-like and the FID-based features.
Revised: Finally, we trained a multi-logistic regression model with both the STOI-like and the FID-based features.

Original: The model accuracy on test set was 87%.
Revised: The model accuracy on test set was 87%.

Original: Given this pretty good result, we could then used this trained logistic model to label 14000 different 5s fake drums clips, produced from as many real bass lines.
Revised: Given this pretty good result, we could then used this trained logistic model to label 14000 different 5s fake drums clips, produced from as many real bass lines.

Original: Two third of these were labeled as good sounding and on time.
Revised: Two third of these were labeled as good sounding and on time.

Original: Here is a private Sound Cloud playlist where you can listen to some of the most interesting results: https://soundcloud.com/user-639025674/ sets/bass2drums/s-jjccrgdXXOi .
Revised: Here is a private Sound Cloud playlist where you can listen to some of the most interesting results.

Original: Regarding instead the voice2song task, results werefar less encouraging.
Revised: Regarding instead the voice2song task, results were less encouraging.

Original: Even though some nice arrangements were produced, the model failed to properly and euphonically arrange the input voice lines.
Revised: Even though some nice arrangements were produced, the model failed to properly and euphonically arrange the input voice lines.

Original: For this reason, here we limit to report some of the best produced samples, in the hope to improve the model greatly in the following months: https://soundcloud.com/ user-639025674/sets/voice2song/s-pCPKlQfTbn8 .
Revised: For this reason, here we limit to report some of the best produced samples, in the hope to improve the model greatly in the following months.

Original: As for baselines, initially we thought about comparing our results to three particularly notable works (Dhariwal et al.,  2020; Vasquez & Lewis, 2019;
Revised: As for baselines, initially we thought about comparing our results to three particularly notable works (Dhariwal et al., [Figure] R e l a t i v e F r e q . 2020; Vasquez & Lewis, 2019;

Original: Mor et al., 2018), but after running some experiments we eventually realized that they could not be properly used for arrangement purposes.All
Revised: Mor et al., 2018), but after running some experiments we eventually realized that they could not be properly used for arrangement purposes.

Original: [Figure] i [Figure] three model produce very nice music samples, but none of them can take as input vocals or bass lines and produce a complementary arrangement.
Revised: All three model produce very nice music samples, but none of them can take as input vocals or bass lines and produce a complementary arrangement.

Original: It is possible though that these models could be ﬁne tuned to solve this new task.
Revised: It is possible though that these models could be ﬁne tuned to solve this new task.

Revised: In addition, we replicated exactly the same experiments using Pix2Pix by Isola et al., a well known paired image-to-image architecture.

Original:  Finally, with respect to the computational resources and time required to generate new arrangements, our approach shows several advantages, compared to auto-regressive models (Dhariwal et al., 2020), by leveraging the two-dimensional time-frequency representation: since the output prediction can be  parallelised, the inference time amounts to few seconds, whereas the Mel-spectrogram-waveform conversion duration depends on the input length, but it never exceeds few minutes.
Revised: Despite long training, results were very poor and quite unpleasant to listen to. Due to space concerns we do not report more details about this set of experiments. Finally, with respect to the computational resources and time required to generate new arrangements, our approach shows several advantages, compared to auto-regressive models (Dhariwal et al., 2020). Since the output prediction can be fully parallelised, the inference time amounts to a forward pass and a Mel-spectrogram-waveform inverse conversion, whose duration depends on the input length, but it never exceeds few minutes.

Original: Indeed, it is worth noting that, at inference time, arbitrary long inputs can be processed and arranged.
Revised: Indeed, it is worth noting that, at inference time, arbitrary long inputs can be processed and arranged.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this work, we presented a novel approach to automatically produce euphonic music arrangements starting from a voice line or a bass line.
Revised: In this work, we presented a novel approach to automatically produce euphonic music arrangements starting from a voice line or a bass line.

Original: We applied Generative Adversarial Networks to real music pieces, treated as grayscale images (Mel spectrograms).
Revised: We applied Generative Adversarial Networks to real music pieces, treated as grayscale images (Mel-spectrograms).

Original: Given the novelty of the problem, we proposed a reasonable procedure to properly evaluate our model outputs.
Revised: Given the novelty of the problem, we proposed a reasonable procedure to properly evaluate our model outputs.

Original: Notwithstanding the promising results, some critical issues need to be addressed before a more compelling architecture can be developed.
Revised: Notwithstanding the promising results, some critical issues need to be addressed before a more compelling architecture can be developed.

Original: First and foremost, a larger and cleaner dataset of source separated songs should be created.
Revised: First and foremost, a larger and cleaner dataset of source separated songs should be created.

Original: In fact, manually separated track always contain a big deal of noise.
Revised: In fact, manually separated track always contain a big deal of noise.

Original: Moreover, the model architecture should be further improve to focus on longer dependencies and to take into account the actual degradation of high frequencies.
Revised: Moreover, the model architecture should be further improved to focus on longer dependencies and to take into account the actual degradation of high frequencies.

Original: Finally, a certain degree of interaction and randomness should be inserted to make the model less deterministic and to give creators some control over the sample generation.
Revised: Finally, a certain degree of interaction and randomness should be inserted to make the model less deterministic and to give creators some control over the sample generation.

Original: Our contribution is nonetheless a ﬁrst step toward more realistic and useful automatic music arrangement systems and we believe that further signiﬁcant steps could be made to reach the ﬁnal goal of human-level automatic music arrangement production.
Revised: Our contribution is nonetheless a ﬁrst step toward more realistic and useful automatic music arrangement systems and we believe that further signiﬁcant steps could be made to reach the ﬁnal goal of human-level automatic music arrangement production.

Original: Already now software like Melodyne (Neub¨acker, 2011; Senior, 2009) delivers producers a powerful user interface to directly intervene on a spectrogram-based representation of audio signals to correct, perfect, reshape and restructure vocals, samples and recordings of all kinds.
Revised: Already now software like Melodyne (Neub¨acker, 2011; Senior, 2009) delivers producers a powerful user interface to directly intervene on a spectrogram-based representation of audio signals to correct, perfect, reshape and restructure vocals, samples and recordings of all kinds.

Original: In is not unlikely that in the future artists and composers will start creating their music almost like they were drawing.
Revised: It is not unlikely that in the future artists and composers will start creating their music almost like they were drawing.