Paper:
This paper introduces a new large-scale music dataset, MusicNet, to serve as a source of supervision and evaluation of machine learning methods for music research. MusicNet consists of hundreds of freely-licensed classical music recordings by 10 composers, written for 11 instruments, together with instrument/note annotations resulting in over 1 million temporal labels on 34 hours of chamber music performances under various studio and microphone conditions. The paper defines a multi-label classification task to predict notes in musical recordings, along with an evaluation protocol, and benchmarks several machine learning architectures for this task: i) learning from spectrogram features; ii) endto-end learning with a neural net; iii) end-to-end learning with a convolutional neural net. These experiments show that end-to-end models trained for note prediction learn frequency selective filters as a low-level representation of audio.
1 INTRODUCTION
Music research has benefited recently from the effectiveness of machine learning methods on a wide range of problems from music recommendation (van den Oord et al., 2013; McFee & Lanckriet, 2011) to music generation (Hadjeres & Pachet, 2016); see also the recent demos of the Google Magenta project1. As of today, there is no large publicly available labeled dataset for the simple yet challenging task of note prediction for classical music. The MIREX MultiF0 Development Set (Benetos & Dixon, 2011) and the Bach10 dataset (Duan et al., 2011) together contain less than 7 minutes of labeled music. These datasets were designed for method evaluation, not for training supervised learning methods.
This situation stands in contrast to other application domains of machine learning. For instance, in computer vision large labeled datasets such as ImageNet (Russakovsky et al., 2015) are fruitfully used to train end-to-end learning architectures. Learned feature representations have outperformed traditional hand-crafted low-level visual features and lead to tremendous progress for image classification. In (Humphrey et al., 2012), Humphrey, Bello, and LeCun issued a call to action: “Deep architectures often require a large amount of labeled data for supervised training, a luxury music informatics has never really enjoyed. Given the proven success of supervised methods, MIR would likely benefit a good deal from a concentrated effort in the curation of sharable data in a sustainable manner.”
This paper introduces a new large labeled dataset, MusicNet, which is publicly available2 as a resource for learning feature representations of music. MusicNet is a corpus of aligned labels on freely-licensed classical music recordings, made possible by licensing initiatives of the European Archive, the Isabella Stewart Gardner Museum, Musopen, and various individual artists. The dataset consists of 34 hours of human-verified aligned recordings, containing a total of 1, 299, 329 individual labels on segments of these recordings. Table 1 summarizes statistics of MusicNet.
The focus of this paper’s experiments is to learn low-level features of music from raw audio data. In Sect. 4, we will construct a multi-label classification task to predict notes in musical recordings,
1https://magenta.tensorflow.org/ 2http://homes.cs.washington.edu/˜thickstn/musicnet.html.
MusicNet
along with an evaluation protocol. We will consider a variety of machine learning architectures for this task: i) learning from spectrogram features; ii) end-to-end learning with a neural net; iii) endto-end learning with a convolutional neural net. Each of the proposed end-to-end models learns a set of frequency selective filters as low-level features of musical audio, which are similar in spirit to a spectrogram. The learned low-level features are visualized in Figure 1. The learned features modestly outperform spectrogram features; we will explore possible reasons for this in Sect. 5.
2 MUSICNET
Related Works. The experiments in this paper suggest that large amounts of data are necessary to recovering useful features from music; see Sect. 4.5 for details. The Lakh dataset, released this summer based on the work of Raffel & Ellis (2015), offers note-level annotations for many 30- second clips of pop music in the Million Song Dataset (McFee et al., 2012). The syncRWC dataset is a subset of the RWC dataset (Goto et al., 2003) consisting of 61 recordings aligned to scores using the protocol described in Ewert et al. (2009). The MAPS dataset (Emiya et al., 2010) is a mixture of acoustic and synthesized data, which expressive models could overfit. The Mazurka project3 consists of commercial music. Access to the RWC and Mazurka datasets comes at both a cost and inconvenience. Both the MAPS and Mazurka datasets are comprised entirely of piano music.
The MusicNet Dataset. MusicNet is a public collection of labels (exemplified in Table 2) for 330 freely-licensed classical music recordings of a variety of instruments arranged in small chamber ensembles under various studio and microphone conditions. The recordings average 6 minutes in length. The shortest recording in the dataset is 55 seconds and the longest is almost 18 minutes. Table 1 summarizes the statistics of MusicNet with breakdowns into various types of labels. Table 2 demonstrates examples of labels from the MusicNet dataset.
MusicNet labels come from 513 label classes using the most naive definition of a class: distinct instrument/note combinations. The breakdowns reported in Table 1 indicate the number of distinct notes that appear for each instrument in our dataset. For example, while a piano has 88 keys only 83 of them are performed in MusicNet. For many tasks a note’s value will be a part of its label, in which case the number of classes will expand by approximately an order of magnitude after taking the cartesian product of the set of classes with the set of values: quarter-note, eighth-note, triplet, etc. Labels regularly overlap in the time series, creating polyphonic multi-labels.
MusicNet is skewed towards Beethoven, thanks to the composer’s popularity among performing ensembles. The dataset is also skewed towards Solo Piano due to an abundance of digital scores available for piano works. For training purposes, researchers may want to augment this dataset to increase coverage of instruments such as Flute and Oboe that are under-represented in MusicNet. Commercial recordings could be used for this purpose and labeled using the alignment protocol described in Sect. 3.
3 DATASET CONSTRUCTION
MusicNet recordings are freely-licensed classical music collected from the European Archive, the Isabella Stewart Gardner Museum, Musopen, and various artists’ collections. The MusicNet labels are retrieved from digital MIDI scores, collected from various archives including the Classical Archives (classicalarchives.com) Suzuchan’s Classic MIDI (suzumidi.com) and HarfeSoft (harfesoft.de). The methods in this section produce an alignment between a digital score and a corresponding freely-licensed recording. A recording is labeled with events in the score, associated to times in the performance via the alignment. Scores containing 6, 550, 760 additional labels are available on request to researchers who wish to augment MusicNet with commercial recordings.
Music-to-score alignment is a long-standing problem in the music research and signal processing communities (Raphael, 1999). Dynamic time warping (DTW) is a classical approach to this problem. An early use of DTW for music alignment is Orio & Schwarz (2001) where a recording is
3http://www.mazurka.org.uk/
aligned to a crude synthesis of its score, designed to capture some of the structure of an overtone series. The method described in this paper aligns recordings to synthesized performances of scores, using side information from a commercial synthesizer. To the best of our knowledge, commercial synthesis was first used for the purpose of alignment in Turetsky & Ellis (2003).
The majority of previous work on alignment focuses on pop music. This is more challenging than aligning classical music because commercial synthesizers do a poor job reproducing the wide variety of vocal and instrumental timbers that appear in modern pop. Furthermore, pop features inharmonic instruments such as drums for which natural metrics on frequency representations–including `2–are not meaningful. For classical music to score alignment, a variant of the techniques described in Turetsky & Ellis (2003) works robustly. This method is described below; we discuss the evaluation of this procedure and its error rate on MusicNet in the appendix.
In order to align the performance with a score, we need to define a metric that compares short segments of the score with segments of a performance. Musical scores can be expressed as binary vectors in E × K where E = {1, . . . , n} and K is a dictionary of notes. Performances reside in RT×p, where T ∈ {1, . . . ,m} is a sequence of time steps and p is the dimensionality of the spectrogram at time T . Given some local cost function C : (Rp,K)→ R, a score Y ∈ E ×K, and a performance X ∈ RT×p, the alignment problem is to
minimize t∈Zn n∑ i=1 C(Xti ,Yi) subject to t0 = 0, tn = m, ti ≤ tj if i < j.
(1)
Dynamic time warping gives an exact solution to the problem in O(mn) time and space. The success of dynamic time warping depends on the metric used to compare the score and the performance. Previous works can be broadly categorized into three groups that define an alignment cost C between segments of music x and score y by injecting them into a common normed space via maps Ψ and Φ:
C(x, y) = ‖Ψ(x)− Φ(y)‖ (2)
The most popular approach–and the one adopted by this paper–maps the score into the space of the performance (Orio & Schwarz, 2001; Turetsky & Ellis, 2003; Soulez et al., 2003). An alternative approach maps both the score and performance into some third space, commonly a chromogram space (Hu et al., 2003; Izmirli & Dannenberg, 2010; Joder et al., 2013). Finally, some recent methods consider alignment in score space, taking Φ = Id and learning Ψ (Garreau et al., 2014; Lajugie et al., 2016).
With reference to the general cost (2), we must specify the maps Ψ,Φ, and the norm ‖ · ‖. We compute the cost in the performance feature space Rp, hence we take Ψ = Id. For the features, we use the log-spectrogram with a window size of 2048 samples. We use a stride of 512 samples between features. Hence adjacent feature frames are computed with 75% overlap. For audio sampled at 44.1kHz, this results in a feature representation with 44, 100/512 ≈ 86 frames per second. A discussion of these parameter choices can be found in the appendix. The map Φ is computed by a synthetizer: we used Plogue’s Sforzando sampler together with Garritan’s Personal Orchestra 4 sample library.
For a (pseudo)-metric on Rp, we take the `2 norm ‖ · ‖2 on the low 50 dimensions of Rp. Recall that Rp represents Fourier components, so we can roughly interpret the k’th coordinate of Rp as the energy associated with the frequency k × (22, 050/1024) ≈ k × 22.5Hz, where 22, 050Hz is the Nyquist frequency of a signal sampled at 44.1kHz. The 50 dimension cutoff is chosen empirically: we observe that the resulting alignments are more accurate using a small number of low-frequency bins rather than the full space Rp. Synthesizers do not accurately reproduce the high-frequency features of a musical instrument; by ignoring the high frequencies, we align on a part of the spectrum where the synthesis is most accurate. The proposed choice of cutoff is aggressive compared to usual settings; for instance, Turetsky & Ellis (2003) propose cutoffs in the 2.5kHz range. The fundamental frequencies of many notes in MusicNet are higher than the 50 × 22.5Hz ≈ 1kHz cutoff. Nevertheless, we find that all notes align well using only the low-frequency information.
4 METHODS
We consider identification of notes in a segment of audio x ∈ X as a multi-label classification problem, modeled as follows. Assign each audio segment a binary label vector y ∈ {0, 1}128. The 128 dimensions correspond to frequency codes for notes, and yn = 1 if note n is present at the midpoint of x. Let f : X → H indicate a feature map. We train a multivariate linear regression to predict ŷ given f(x), which we optimize for square loss. The vector ŷ can be interpreted as a multi-label estimate of notes in x by choosing a threshold c and predicting label n iff ŷn > c. We search for the value c that maximizes F1-score on a sampled subset of MusicNet.
4.1 RELATED WORK
Learning on raw audio is studied in both the music and speech communities. Supervised learning on music has been driven by access to labeled datasets. Pop music labeled with chords (Harte, 2010) has lead to a long line of work on chord recognition, most recently Korzeniowsk & Widmer (2016). Genre labels and other metadata has also attracted work on representation learning, for example Dieleman & Schrauwen (2014). There is also substantial work modeling raw audio representations of speech; a current example is Tokuda & Zen (2016). Recent work from Google DeepMind explores generative models of raw audio, applied to both speech and music (van den Oord et al., 2016).
The music community has worked extensively on a closely related problem to note prediction: fundamental frequency estimation. This is the analysis of fundamental (in contrast to overtone) frequencies in short audio segments; these frequencies are typically considered as proxies for notes. Because access to large labeled datasets was historically limited, most of these works are unsupervised. A good overview of this literature can be found in Benetos et al. (2013). Variants of non-negative matrix factorization are popular for this task; a recent example is Khlif & Sethu (2015). A different line of work models audio probabilistically, for example Berg-Kirkpatrick et al. (2014). Recent work by Kelz et al. (2016) explores supervised models, trained using the MAPS piano dataset.
4.2 MULTI-LAYER PERCEPTRONS
We build a two-layer network with features fi(x) = log ( 1 + max(0,wTi x) ) . We find that compression introduced by a logarithm improves performance versus a standard ReLU network (see Table 3). Figure 1 illustrates a selection of weights wi learned by the bottom layer of this network. The weights learned by the network are modulated sinusoids. This explains the effectiveness of spectrograms as a low-level representation of musical audio. The weights decay at the boundaries, analogous to Gabor filters in vision. This behavior is explained by the labeling methodology: the audio segments used here are approximately 1/3 of a second long, and a segment is given a note
label if that note is on in the center of the segment. Therefore information at the boundaries of the segment is less useful for prediction than information nearer to the center.
4.3 (LOG-)SPECTROGRAMS
Spectrograms are an engineered feature representation for musical audio signals, available in popular software packages such as librosa (McFee et al., 2015). Spectrograms (resp. log-spectrograms) are closely related to a two-layer ReLU network (resp. the log-ReLU network described above). If x = (x1, . . . , xt) denotes a segment of an audio signal of length t then we can define
Speck(x) ≡ ∣∣∣∣∣ t−1∑ s=0 e−2πiks/txs ∣∣∣∣∣ 2 = ( t−1∑ s=0 cos(2πks/t)xs )2 + ( t−1∑ s=0 sin(2πks/t)xs )2 .
These features are not precisely learnable by a two-layer ReLU network. But recall that |x| = max(0, x) + max(0,−x) and if we take weight vectors u,v ∈ RT with us = cos(2πks/t) and vs = sin(2πks/t) then the ReLU network can learn
fk,cos(x) + fk,sin(x) ≡ |uTx|+ |vTx| = ∣∣∣∣∣ t−1∑ s=0 cos(2πks/t)xs ∣∣∣∣∣+ ∣∣∣∣∣ t−1∑ s=0 sin(2πks/t)xs ∣∣∣∣∣ . We call this family of features a ReLUgram and observe that it has a similar form to the spectrogram; we merely replace the x 7→ x2 non-linearity of the spectrogram with x 7→ |x|. These features achieve similar performance to spectrograms on the classification task (see Table 3).
4.4 WINDOW SIZE
When we parameterize a network, we must choose the width of the set of weights in the bottom layer. This width is called the receptive field in the vision community; in the music community it is called the window size. Traditional frequency analyses, including spectrograms, are highly sensitive to the window size. Windows must be long enough to capture relevant information, but not so long that they lose temporal resolution; this is the classical time-frequency tradeoff. Furthermore, windowed frequency analysis is subject to boundary effects, known as spectral leakage. Classical signal processing attempts to dampen these effects with predefined window functions, which apply a mask that attenuates the signal at the boundaries (Rabiner & Schafer, 2007).
The proposed end-to-end models learn window functions. If we parameterize these models with a large window size then the model will learn that distant information is irrelevant to local prediction, so the magnitude of the learned weights will attenuate at the boundaries. We therefore focus on two window sizes: 2048 samples, which captures the local content of the signal, and 16,384 samples, which is sufficient to capture almost all relevant context (again see Figure 1).
4.5 REGULARIZATION
The size of MusicNet is essential to achieving the results in Figure 1. In Figure 3 (Left) we optimize a two-layer ReLU network on a small subset of MusicNet consisting of 65, 000 monophonic data points. While these features do exhibit dominant frequencies, the signal is quite noisy. Comparable noisy frequency selective features were recovered by Dieleman & Schrauwen (2014); see their Figure 3. We can recover clean features on a small dataset using heavy regularization, but this destroys classification performance; regularizing with dropout poses a similar tradeoff. By contrast, Figure 3 (Right) shows weights learned by an unregularized two-layer network trained on the full MusicNet dataset. The models described in this paper do not overfit to MusicNet and optimal performance (reported in Table 3) is achieved without regularization.
4.6 CONVOLUTIONAL NETWORKS
Previously, we estimated ŷ by regressing against f(x). We now consider a convolutional model that regresses against features of a collection of shifted segments x` near to the original segment x. The learned features of this network are visually comparable to those learned by the fully connected network (Figure 1). The parameters of this network are the receptive field, stride, and pooling regions.
The results reported in Table 3 are achieved with 500 hidden units using a receptive field of 2, 048 samples with an 8-sample stride across a window of 16, 384 samples. These features are grouped into average pools of width 16, with a stride of 8 features between pools. A max-pooling operation yields similar results. The learned features are consistent across different parameterizations. In all cases the learned features are comparable to those of a fully connected network.
5 RESULTS
We hold out a test set of 3 recordings for all the results reported in this section:
• Bach’s Prelude in D major for Solo Piano. WTK Book 1, No 5. Performed by Kimiko Ishizaka. MusicNet recording id 2303. • Mozart’s Serenade in E-flat major. K375, Movement 4 - Menuetto. Performed by the Soni Ventorum Wind Quintet. MusicNet recording id 1819. • Beethoven’s String Quartet No. 13 in B-flat major. Opus 130, Movement 2 - Presto. Released by the European Archive. MusicNet recording id 2382.
The test set is a representative sampling of MusicNet: it covers most of the instruments in the dataset in small, medium, and large ensembles. The test data points are evenly spaced segments separated by 512 samples, between the 1st and 91st seconds of each recording. For the wider features, there is substantial overlap between adjacent segments. Each segment is labeled with the notes that are on in the middle of the segment.
We evaluate our models on three scores: precision, recall, and average precision. The precision score is the count of correct predictions by the model (across all data points) divided by the total number
of predictions by the model. The recall score is the count of correct predictions by the model divided by the total number of (ground truth) labels in the test set. Precision and recall are parameterized by the note prediction threshold c (see Sect. 4). By varying c, we construct precision-recall curves (see Figure 4). The average precision score is the area under the precision-recall curve.
A spectrogram of length n is computed from 2n samples, so the linear 1024-point spectrogram model is directly comparable to the MLP runs with 2048 raw samples. Learned features4 modestly outperform spectrograms for comparable window sizes. The discussion of windowing in Sect. 4.4 partially explains this. Figure 5 suggests a second reason. Recall (Sect. 4.3) that the spectrogram features can be interpreted as the magnitude of the signal’s inner product with sine waves of linearly spaced frequencies. In contrast, the proposed networks learn weights with frequencies distributed similarly to the distribution of notes in MusicNet (Figure 5). This gives the network higher resolution in the most critical frequency regions.
ACKNOWLEDGMENTS
We thank Bob L. Sturm for his detailed feedback on an earlier version of the paper. We also thank Brian McFee and Colin Raffel for fruitful discussions. Sham Kakade acknowledges funding from the Washington Research Foundation for innovation in Data-intensive Discovery. Zaid Harchaoui acknowledges funding from the program ”Learning in Machines and Brains” of CIFAR.
4A demonstration using learned MLP features to synthesize a musical performance is available on the dataset webpage: http://homes.cs.washington.edu/˜thickstn/demos.html
B ALIGNMENT PARAMETER ROBUSTNESS
The definitions of audio featurization and the alignment cost function were contingent on several parameter choices. These choices were optimized by systematic exploration of the parameter space. We investigated what happens as we vary each parameter and made the choices that gave the best results in our listening tests. Fine-tuning of the parameters yields marginal gains.
The quality of alignments improves uniformly with the quality of synthesis. The time-resolution of labels improves uniformly as the stride parameter decreases; minimization of stride is limited by system memory constraints. We find that the precise phase-invariant feature specification has little effect on alignment quality. We experimented with spectrograms and log-spectrograms using windowed and un-windowed signals. Alignment quality seemed to be largely unaffected.
The other parameters are governed by a tradeoff curve; the optimal choice is determined by balancing desirable outcomes. The Fourier window size is a classic tradeoff between time and frequency resolution. The `2 norm can be understood as a tradeoff between the extremes of `1 and `∞. The `1 norm is too egalitarian: the preponderance of errors due to synthesis quality add up and overwhelm the signal. On the other hand, the `∞ norm ignores too much of the signal in the spectrogram. The spectrogram cutoff, discussed in Sec. 3, is also a tradeoff between synthesis quality and maximal use of information
C ADDITIONAL ERROR ANALYSIS
For each model, using the test set described in Sect. 5, we report accuracy and error scores used by the MIR community to evaluate the Multi-F0 systems. Definitions and a discussion of these metrics are presented in Poliner & Ellis (2007).
D PRECISION & RECALL CURVES
0.0 0.2 0.4 0.6 0.8 1.0 recall
Figure 7: The 500 node, 2048 raw sample MLP.
0.0 0.2 0.4 0.6 0.8 1.0 recall
Figure 9: The average pooling model.
0.0 0.2 0.4 0.6 0.8 1.0 recall
Figure 11: The convolutional model.
E ADDITIONAL RESULTS
We report additional results on splits of the test set described in Sect. 5.

Review:
The paper introduces a new dataset called MusicNet (presumably analogous to ImageNet), featuring dense ground truth labels for 30+ hours of classical music, which is provided as raw audio. Such a dataset is extremely valuable for music information retrieval (MIR) research and a dataset of this size has never before been publicly available. It has the potential to dramatically increase the impact of modern machine learning techniques (e.g. deep learning) in this field, whose adoption has previously been hampered by a lack of available datasets that are large enough. The paper is clear and well-written.
The paper also features some "example" experiments using the dataset, which I am somewhat less excited about. The authors decided to focus on one single task that is not particularly challenging: identifying pitches in isolated segments of audio. Pitch information is a fairly low-level characteristic of music. Considering that isolated fragments are used as input, this is a relatively simple problem that probably doesn't even require machine learning to solve adequately, e.g. peak picking on a spectral representation could already get you pretty far. It's not clear what value the machine learning component in the proposed approach actually adds, if any. I could be wrong about this as I haven't done the comparison myself, but I think the burden is on the authors to demonstrate that using ML here is actually useful.
I would argue that one of the strenghts of the dataset is the variety of label information it provides, so a much more convincing setup would have been to demonstrate many different prediction tasks for both low-level (e.g. pitch, onsets) and high-level (e.g. composer) characteristics, perhaps with fewer and simpler models -- maybe even sticking to spectrogram input and forgoing raw audio input for the time being, as this comparison seems orthogonal to the introduction of the dataset itself. As it stands, I feel that the fact that the experiments are relatively uninteresting detracts from the main point of the paper, which is to introduce a new public dataset that is truly unique in terms of its scale and scope.
That said, the experiments seem to have been conducted in a rigorous fashion and the evaluation and analysis of the resulting models is properly executed.
Re: Section 4.5, it is rather unsurprising to me that a pitch detector would learn filters that resemble pitches (i.e. sinusoids), although the observation that this requires a relatively large amount of data is interesting. However, it would be more interesting to demonstrate that this is also the case for higher-level tasks. The authors favourably compare the features learnt by their model with prior work on end-to-end learning from raw audio, but neglect that the tasks considered in this work were much more high-level.
Some might also question whether ICLR is the appropriate venue to introduce a new dataset, but personally I think it's a great idea to submit it here, seeing as it will reach the right people. I suppose this is up to the organisers and the program committee, but I thought it important to mention this, because I don't think this paper merits acceptance based on its experimental results alone.