C OUPLED
R ECURRENT
M ODELS
FOR
P OLYPHONIC
M USIC C OMPOSITION
Anonymous authors
Paper under double-blind review
A BSTRACT
This work describes a novel recurrent model for music composition, which accounts for the rich statistical structure of polyphonic music.
There are many ways to factor the probability distribution over musical scores; we consider the merits of various approaches and propose a new factorization that decomposes a score into a collection of concurrent, coupled time series: “parts.”
The model we propose borrows ideas from both convolutional neural models and recurrent neural models; we argue that these ideas are natural for capturing music’s pitch invariances, temporal structure, and polyphony.
We train generative models for homophonic and polyphonic composition on the
KernScores dataset (Sapp, 2005), a collection of 2,300 musical scores comprised of around 2.8 million notes spanning time from the Renaissance to the early 20th century.
While evaluation of generative models is known to be hard (Theis et al., 2016), we present careful quantitative results using a unit-adjusted cross entropy metric that is independent of how we factor the distribution over scores.
We also present qualitative results using a blind discrimination test.
I NTRODUCTION
The composition of music using statistical models has been strongly inﬂuenced by developments in deep learning; see Briot et al.
(2017) for a recent survey of this ﬁeld.
Previous work in this ﬁeld mostly focuses on either monophonic scores (Sturm et al., 2016;
Jaques et al., 2017; Roberts et al., 2018), rhythmically simple polyphonic scores (Liang et al., 2017; Hadjeres et al., 2017;
Huang et al., 2017), or rhythmically simpliﬁed encodings of more complex scores (BoulangerLewandowski et al., 2012).
In this paper we consider rhythmically complex, polyphonic scores.
To this end, we propose a new generative modeling task on the KernScores dataset, a diverse collection small-ensemble western music.
We seek to understand how well we can model the local
distribution of rhythmically complex, polyphonic music
Many notable successes in deep learning are achieved in domains where natural weight-sharing schemes allow models to borrow strength from similar patterns in different locations: convolutions in vision, for example, or autoregressive models in language.
Polyphonic music has rich spatial and temporal structure that is potentially amenable to such weight-sharing schemes.
In this paper, we will consider how we should factor the probability distribution over scores to allow our models to take advantage of shared patterns, and also how to construct models that explicitly leverage these patterns with shared weights .
In Section 2 of this paper, we will discuss previous approaches to monophonic and polyphonic music composition.
In Section 3 we introduce two new generative modeling tasks on the KernScores dataset: a single-part, homophonic prediction task and a multi-part, polyphonic prediction test.
We additionally discuss quantitative evaluation and introduce an adjusted cross-entropy rate that is invariant to the approach we use to factor the score.
In Section 4 we discuss several approaches to factoring the distribution over scores, and propose a new approach that exploits the structure of music to allow us to share weights in our models.
Section 5 proposes models for this factored distribution and identiﬁes opportunities for weight-sharing.
We present quantitative and qualitative evaluations of these models in section 6.
R ELATED W ORKS
Early efforts to build statistical models of music focus on single part, monophonic sequences (melodies).
Possibly the ﬁrst statistical model for music generation was proposed by Pinkerton (1956).
This work was followed concretely by Brooks et al.
(1956), who built a Markov transition model estimated on small music corpora.
A proliferation of work on computer-generated music and data-driven musicology followed these pioneering works in the 1960’s and 1970’s; see Roads (1980) for a survey.
An important development during this era was the application of Chomsky-inspired grammatical analysis to music, exempliﬁed by Kohonen (1989); this latter work contemplates the generation of two concurrent musical parts, one of the earliest examples of polyphonic generation.
The ﬁrst application of neural networks to algorithmic melody composition was proposed by Todd (1989).
This work prompted followup by Mozer (1994), who altered the representation of the input to Todd’s model using pitch geometry ideas inspired by Shepard (1982); the relative pitch and note-embedding schemes considered in the present paper can be seen as a data-driven approach to capturing some of these geometric concepts.
Neural melody generation was revisited by Eck &
Schmidhuber (2002), using long short-term memory models.
More recent work on melodic composition experiments with techniques to capturing longer-term structure than classic recurrent models provide.
Jaques et al.
(2017) explore reinforcement learning as a tool for eliciting long-term structure, expanding on ideas ﬁrst considered by Franklin (2001).
Roberts et al.
(2018) also attempt to capture long-term structure, proposing a variational auto-encoder for this purpose.
The work on polyphonic music is considerably younger.
The aforementioned work of Kohonen (1989) considers two-part composition.
Another early precursor to polyphonic models was introduced by Ebcio˘glu (1988), who proposed an expert system to harmonize 4-part Bach chorales.
The harmonization task became popular, along with the Bach chorales dataset.
See Allan & Williams (2006) for a classic discussion of this problem.
Lavrenko & Pickens (2003) directly address multipart polyphony, albeit using a simpliﬁed preprocessed encoding of scores that throws away duration information.
Maybe the ﬁrst work with a fair claim to consider polyphonic music in full generality is BoulangerLewandowski et al.
This paper proposed a coarse discrete temporal factorization of musical scores (for a discussion of this and other factorizations, see Section 4) and explored a variety of neural architectures several music datasets (including the Bach chorales).
Many works on polyphonic models since 2012 have focused on the dataset and encoding introduced in Boulanger-Lewandowski et al.
(2012), notably Vohra et al.
and Johnson (2017).
These works typically focus on quantitative log-likelihood improvements, and the degree to which these quantitative improvements correlate with quality is less clear.
For the Bach dataset, qualitative success is more deﬁnitive.
Recently, concurrent work by Liang et al.
(2017) and Hadjeres et al.
(2017) proposed models of the Bach chorales with large-sample discrimination tests demonstrating the convincing quality of their results.
In this line of work, quantitative results are lacking; Liang et al.
(2017) learn a generative model and could in principle report cross entropies, although their work focuses on the qualitative study.
The system proposed in Hadjeres et al.
(2017) optimizes a pseudo-likelihood, so its losses cannot be easily compared to generative models.
Quantitative metrics are reintroduced for the chorales in Huang et al.
Both the latter papers propose non-sequential Gibbs-sampling schemes for generation, in contrast to the ancestral sampler used by Liang et al.
Huang et al.
(2017) make the case that a non-sequential sampling scheme is important for generating plausible compositions.
D ATASET AND E VALUATION
The models presented in this paper are trained on data from the KernScores library (Sapp, 2005), a collection of early modern, classical, and romantic era digital scores assembled by musicologists and researchers associated with Stanford’s CCARH.
This dataset consists of over 2,300 scores containing approximately 2.8 million note labels.
Tables 1 and 2 give a sense of the contents of the dataset.
[Table]
Early
[Table]
Haydn
[Table]
We contrast this dataset and its Humdrum encoding with the MIDI datasets used by most of the works we have discussed in this paper (a notable exception is Lavrenko & Pickens (2003), who used data derived from the same KernScores collection consided here).
MIDI is an operational format in the sense that it consists of a stream of instructions that describe a musical performance.
Indeed it was designed as a protocol for communicating digital performances, rather than digital scores.
For example, it cannot explicitly represent concepts such as quarter-notes or eighth-notes or rests, only notes of a certain duration or the absense of notes.
Heuristics are necessary to display a MIDI ﬁle as a visual score and, if the MIDI wasn’t prepared speciﬁcally for this purpose, these heuristics are liable to fail badly.
For example, a MIDI ﬁle might encode “stacatto” articulations by shortening the length of certain notes; the heuristics to determine the value of a note (quarter-note, eighth-note, etc.) based on its length become exceedingly complicated in such cases.
[Table]
It is not impossible to construct a high-quality dataset of MIDI scores; but the Humdrum format, designed consciously by musicologists to encode scores, helps to ensure the quality of the data by enforcing constraints that are absent from MIDI.
Polyphonic music needs a new benchmark dataset.
As Huang et al.
(2017) point out, the dataset introduced by Boulanger-Lewandowski et al.
(2012) is too coarsely preprocessed to continue to serve this purpose.
And the Bach chorales dataset is too small to sustain much further research.
The KernScores collection considered here is readily available, reasonably large, and is structurally guaranteed have high quality.
3. E VALUATION
Let p denote the unknown distribution over musical scores S , and let q be our model of p .
We want to measure the quality of q by its cross-entropy to p .
Because the entropy of a score grows with its length T , we will consider a cross-entropy rate.
2 By convention, we measure time in units of beats, so our cross-entropy rate will have units of bits per beat.
Deﬁning cross-entropy for a continuous process generally requires some care.
But in the case of music, we observe that notation occurs at rational points in time, and for rational durations.
We can therefore quantize time using the ﬁnest denominator ∆ of times that appear in the dataset and deﬁne
[Equation]
This quantity is invariant to further reﬁnement of the discretization.
Suppose we quantize S at a rate ∆ / 2 ; then
[Equation]
The odd terms vanish under our assumption that ∆ was the ﬁnest denominator of notation in the dataset.
We can think of ∆ as the resolution of the score process.
Observe that Deﬁnition 1 is independent of any choice about how we factor p : it is a cross entropy measure of the joint distribution.
As we will discuss in Section 4, there are many ways to construct a generative model of scores.
These choices lend themselves to different natural cross-entropies with their own units, depending on how we factor.
By converting to units of bits per beat given by Deﬁnition 1, we can compare results under different factorizations.
F ACTORING THE D ISTRIBUTION OVER S CORES
Polyphonic scores consist of features (notes and other notation) of variable length that overlap each other in quasi-continuous time.
To factor the probability distribution over scores, we must somehow impose a sequential order upon the data.
There is a loose partial order on scores implied by time but, in contrast to language, this order is not total.
This slack admits many reasonable ways to factor the distribution over scores.
Most previous work factors a score by discretizing time.
As we discussed in the previous section, there is a time resolution to all musical processes and we can discretize at this resolution without losing information.
From this perspective, music looks like a large binary matrix of notes crossed with time; entry ( t, n ) in this matrix indicates whether note n is on at time t .
3 We can then generate music one slice of time at a time, generating a slice of time with a vector-valued prediction as in
Boulanger-Lewandowski et al.
or imposing an order (e.g. low to high) on notes and further factoring into binary predictions as in Liang et al.
The above factorization, while popular, is computationally difﬁcult for rhythmically complex music.
The process resolution ∆ required to discretize scores without loss of information is the common denominator of all rhythmic events in the corpus.
A corpus that contains both triplets and thirtysecond notes, for example, would require a discretization of 48 positions per a beat.
The datasets considered in Boulanger-Lewandowski et al.
(2012) are discretized at either 1 or 2 positions per beat; as discussed by Huang et al.
(2017), this downsampling is quite destructive to the structure of music, more analogous to dropping words from a sentence than downsampling an image.
Hardware capabilities may eventually overcome the computational obstruction to discretized score modeling, but we take a different approach in this paper that scales better with rhythmic complexity.
One alternative to discretizing time and predicting notes at each time step is instead to operationalize scores.
From this perspective, a score becomes a long sequence of instructions: start playing C, start playing E, advance to the next time step, stop playing C, etc.
We can think of this approach as a run-length encoding of the discrete factorization.
It has been considered recently for the related task of modeling expressive performances by Oore et al.
A similar factorization was proposed by Walder (2016), although that work does not implement a complete model.
Operationalized runlength encodings greatly reduce the computational costs of learning and generation, at the expense of segmenting a score into highly non-linear segments.
The number of items in the sequence between the beginning and the end of a note depends on how many other notes begin or end in the interim.
Contrast this with the discrete factorization, for which every quarter-note (for example) lasts for exactly 1 / ∆ time slices.
In this paper, we adopt a factorization inspired by the operationalized perspectives.
First, we decompose a polyphonic score into a collection of parts.
We can loosely think of a part as the set of notes in a score assigned to a particular voice or instrument.
Each part is homophonic and therefore run-lengths in a part correspond to the duration of notes (in contrast to operationalized full scores, where run-lengths have no musical interpretation).
We learn to predict the next homophony (note, chord, or rest) in a part, making a prediction in the part that has advanced the least far in time, and breaking ties between parts by an arbitrary (ﬁxed) order that we impose on the parts in each score.
To predict the homophony, we impose an order pitches from low to high frequency, and make a sequence of binary predictions of whether each pitch is conditioned on lower-frequency pitches.
M ODELS AND W EIGHT -S HARING
Homophonic models Factoring the score as discussed in Section 4 allows us to think of the polyphonic composition problem as a collection of coupled homophonic composition problems. We therefore consider a homophonic composition task on the KernScores dataset’s parts: this is a singlepart prediction task that generalizes classic monophonic prediction tasks to allow for chords.
We explore a variety of fully connected, convolutional, and recurrent models for this task and ﬁnd that the recurrent architecture works quite well.
Table 5 summarizes these experiments; the third block of Experiments (15-21) compare the most interesting architectures over a reasonable amount of history.
The remaining question is: how do we encode the history of a polyphonic score, and how do we model correlations between parts in this history?
We encode the history of a score as an orderbinary tensor x ∈ { 0 , 1 } T × P × ( N + D ) , indexed by a time axis of length T , a part axis of length P , and an ( N + D ) -dimensional feature axis consisting of N -dimensional multi-hot vector of notes, and a D -dimensional one-hot vector of durations.
The note bits x t,p,n for n ∈ { 0 , . .
. , N − 1 } indicate whether note n is on at time t in part p .
The duration bits x t,p,d for d ∈ { N, N + D − 1 } indicate the duration of a homophonic event in part p beginning at time t or, if the previous state of part p continues at time t , a special continuation duration ‘ ∗ ’ is indicated.
In this way, we interlace the events in each part between each other: see Figure 2 for a visual example of this encoding.
Our hope is that a recurrent network designed for the single-part task would be relatively unhampered when retrained on part data interspersed with these continuations; Experiment 22 in Tablesuggests that this is the case; performance degrades slightly in comparison to Experiment 21, but this is to be expected because a length-10 history interspersed with continuations is effectively a slightly shorter history.
This encourages us to consider the coupled recurrent models described below.
Polyphonic models We now consider the full polyphonic task; results for this task are summarized in Table 3. We must now model correlations between parts in the history tensor, which we achieve coupling the representations of the individual parts.
One natural extension of a recurrent neural network part model to multiple concurrent parts is a hierarchical architecture:
[Equation]
Equation 2 is a recurrent network that builds a state estimate h p,t of an individual part x p at time t based on transition weights W p , emission weights W x , and non-linear activation a .
Equation 3 is a global network that integrates the states of the part (weights W hp ) with the previous global state (weights W h ) to build a coupled global state h t at time t .
Because the order of the parts is arbitrary, we sum over their states before feeding them into the global network.
At each time step, we can use the learned state of each part together with the global state to predict what follows (see Figure 1).
Another natural extension of a recurrent part model is to directly integrate the state of the other parts’ states into each individual part’s state, resulting in a distributed state architecture:
[Equation]
We ﬁnd that the distributed architecture underperforms the hierarchical architecture (see Table 3;
Experiments 2 and 3) although this comparison is not conclusive: for example, the hierarchical model’s implementation has more parameters than the distributed model.
For the hierarchical model, we can also consider whether the global state representation is as sensitive to history-length as the parts.
Could we make successful predictions using only the ﬁnal state of each part, rather than coupling the states at each step?
Experiments (4,5,6) in Table 3 suggest that this is not the case.
In the remainder of this section, we explore a variety of weight-sharing ideas that are somewhat orthogonal to our methods for factoring and modeling scores.
These ideas may be of general interest for both monophonic and polyphonic composition, beyond the speciﬁc models under consideration.
5. AUTOREGRESSIVE MODELING
To build a generative model over sequential data x = ( x 1 , . .
, x t ) , rather than directly model the distribution p ( x ) , it often makes sense to factor the joint distribution into conditionals p ( x t | x 1: t ) and make an autoregressive assumption p ( x t | x 1: t )
=
p ( x s | x 1: s ) for all s, t ∈ N .
We can then learn a single conditional distribution p ( x t | x 1: t ) and share model parameters across all time translations.
If the data is conditionally stationary (or nearly so) this approach is extremely effective (analogous to convolution for vision problems).
Scores are not quite conditionally stationary; their distribution varies substantially depending on the position within the beat.
For example, the distribution has quite a lot of variance on the beat and new notes are frequently introduced.
In contrast, notes almost never begin an ε -fraction of time after the beat and the distribution is quite peaked on the notes initiated in the previous time-step.
To address this non-stationarity, we follow the lead of Johnson (2017) and Hadjeres et al.
and augment our history tensor with a one-hot location feature vector ( (cid:96) in Table 5) that indicates the subdivision of the beat for which we are presently making predictions.
5 Compare the loss of duration models (Loss t ) with and without these features in Experiment pairs (3,4), (6,7), (10,11), (12,13), and (15,16)
5. PART DECOMPOSITION
We have previously discussed decomposing a score into multiple parts.
This presents us with an opportunity to share weights between part models by imposing the assumption p ( x t,p | x t, 1: p , x 1: t ) =
p ( x t,q | x t, 1: q , x 1: t ) for all parts p, q .
This corresponds to learning a single set of weights W p in equations (2) and (4), rather than learning unique part-indexed weights W p i for each part p i . Indeed, because the index of a part is arbitrary, the weights W p i should converge to the same values for all i ; sharing a single set of weights W p accelerates learning by enforcing this property.
5. RELATIVE PITCH
With a little effort, we can perform a similar weight-sharing scheme over the notes as we did over time and parts.
This idea was ﬁrst proposed in Johnson (2017).
Recall that we factor the vector of pitches in part p at time t into a sequence of binary predictions, from lowest to highest pitch.
Instead of building an individual predictor for each pitch conditioned on the notes in the history tensor, we can build a single predictor that conditions on a shifted version of the history tensor centered around the note we want to predict.
By convolving this predictor over the pitch axis of the history tensor, we can make a prediction at each note location based on a relativized view of the history: see Figure 3 for a visualization of this transformation.
Like with the time axis, we observe that the distribution over notes is not quite conditionally stationary along the note-class axis.
For example, a truly relative predictor would generate notes uniformly across the note-class axis, whereas the actual distribution of notes concentrates around middle C.
Therefore we augment our history tensor with a one-hot pitch-class feature vector 1 n that indicates the note n for which we are presently making a prediction.
This allows us to take full advantage of all available information when making a prediction, while borrowing strength from shared harmonic patterns in different keys or octaves.
We compare absolute pitch-indexed classiﬁers ( lin n ) to a single, relative pitch classiﬁer ( lin ) in Table 5: compare the loss of pitch models (Loss n ) in
Experiments (2,3,4), (5,6,7), (8,9,10), (11,12,13), and (15,16).
5. PITCH EMBEDDINGS
Borrowing the concept of a word embedding from natural language processing, we consider learned embeddings of pitch vectors, denoted by c in Table 5.
For recurrent models, we do not observe substantial performance beneﬁts to learning these embeddings: compare Experiments (20,21) in
Table 5.
However, we do ﬁnd that we can learn quite compact embeddings (16 dimensions for the experiments presented in this paper) without sacriﬁcing performance, and working on these compact embeddings speeds up processing time for learning and generation.
We also ﬁnd that a simpledimensional ﬁxed embedding of pitches f , in which we quotient each pitch class by octave, reduces overﬁtting for the rhythmic model while preserving performance.
E XPERIMENTAL R ESULTS
Quantitative Results.
The single-part (homophonic) and multi-part (polyphonic) prediction tasks are presented in tables 5 and 3 respectively.
We advise caution in thinking about these results.
Small differences in the log-loss can have large effects on the quality of output, especially if the differences are attributable to missing information.
For example, failing to include a pitch-class feature vector in the relative models (described in Section 5.3) has a catastrophic impact on generated sequences, even though the log loss gap between is not always large.
Detailed discussions of the individual results in these tables are presented in context in Section 5.
[Table]
Qualitative results We asked twenty study participants to listen to a variety of audio clips, synthesized from either a real composition or from the output of one of our models: Experiment 4 in
Table 3.
For each clip, participants were asked to rate whether they thought the clip was written by a computer or by a human composer.
Participants were presented with clips of varying length, from 10 frames of data (2-3 seconds; the length of the model’s Markov window) to 50 frames of data ( or more seconds).
We expected that participant success would improve with the length of the clips, but we did not ﬁnd this to be the case; indeed, even among the longest clips (around 20 seconds) participants occasionally identiﬁed an artiﬁcial clip as a human composition.
Results are presented in Table 4.
[Table]
As with the quantitative results, we again urge caution in interpreting these qualitative results.
Our study results superﬁcially suggest that we have done well in modeling the short-term structure of our dataset (we make no claims to have captured long-term structure; indeed, the Markov windows of our models preclude this).
But it is not clear that humans are good (or should be good) at the task of identifying plausible or implausible local structure in music.
It is also unclear how to use such studies to compare between models, where differences would be less pronounced.
Indeed, it is not even clear how to prompt a user to discriminate in such a setting.
C ONCLUSION
Given the difﬁculties evaluation generative models, it may be important to pay further attention to downstream tasks.
One important downstream task is music transcription, which is considered together with polyphonic composition in Boulanger-Lewandowski et al.
(2012) and more recently in
Sigtia et al.
Both these systems operate on performance-aligned label sequences: a warping of a score to an expressive performance.
More work would be necessary to generate an actual score that correctly identiﬁes the value of notes (e.g. quarter note, or half-note) and not just the durations of notes in the audio.
The authors would also like to point out the oddity of training generative models on such a diverse set of music: from Josquin to Joplin.
Music is inherently a low-resource learning problem; for comparison, modern language models are regularly trained on datasets larger than the entire classical music canon (Chelba et al., 2014).
Fortunately, music has a much lower entropy rate than language.
But we may need new tools to learn properly to compose in the style of Mozart. [Table] = [Table]
=
[Table]
ˆ y t =
lin ( x 1 ) , ˆ y n = lin ( x 1 , y t , y 1: n )
[Table]
y
[Table]
=
lin ( x 1 , (cid:96) ) , ˆ y n = lin ( x [Table] y t [Table]
y
[Table]
n
[Table]
[Table]
[Table]
= lin
[Table]
= lin
[Table]
( x
[Table]
y
[Table]
y
[Table]
[Table]
=
[Table]
[Table] 372k ˆ y t = lin ◦ fc ( f ( x 10 ) , (cid:96) ) ˆ y n = lin ◦ fc ( c ( x 10 ) , y t , y 1: n , 1 n ) 4.
250k ˆ y t =
lin ◦ conv 5 ( f ( x 10 ) , (cid:96) )
ˆ y n = lin ◦ conv 5 ( c ( x 10 ) , y t , y, 1 n ) 4.
769k ˆ y t =
lin ◦ conv 3 ◦ conv 5 ( f ( x 10 ) , (cid:96) )
ˆ y n = lin ◦ conv 3 ◦ conv 5 ( c ( x 10 ) , y t , y 1: n , 1 n ) 4.
342k ˆ y t =
lin ◦ rnn ( x 10 , (cid:96) )
ˆ y n = lin ◦ rnn ( x 10 , y t , y 1: n , 1 n )) 4.
283k ˆ y t =
lin ◦ rnn ( f ( x 10 ) , (cid:96) )
ˆ y n = lin ◦ rnn ( c ( x 10 ) , y t , y 1: n , 1 n )) 4.
301k ˆ y t =
lin ◦ rnn ( f (˜ x 10 ) , (cid:96) )
ˆ y n = lin ◦ rnn ( c (˜ x 10 ) , y t , y 1: n , 1 n )) 4.
A P IANO M USIC
For piano music, we need to draw a distinction between an instrument and a part.
Consider the piano score given in Figure 4.
This single piano part is more comparable to a complete score than the individual parts of, for example, a string quartet (compare the piano score in Figure 4 to the quartet score in Figure 2).
Indeed, an educated musician would read this score in four distinct parts: a high sequence of quarter and eighth notes, two middle sequences of sixteenth notes, and a low sequence of quarter notes.
In measure 12, the lowest two parts combine into a single bass line of sixteenth notes.
These part divisions are indicated in score through a combination of beams, slurs, and other visual queues.
We do not model these visual indicators; instead we rely on part annotations provided by the
KernScores dataset.
The provision of these annotations is a strong point in favor of the KernScores dataset’s Humdrum format; although in principle formats like MIDI can encode this information, in practice they typically collect all notes for a single instrument into a single track, or possibly two tracks (for the treble and bass staves, as seen in the ﬁgure) in the case of piano music.
In extremely rare cases, this distinction between instrument and part must also be made for stringed instruments; a notable example is Beethoven’s string quartet number 14, in the fourth movement in measures 165 and 173, where the four instruments each separate into two distinct parts creating brief moments of 8-part harmony.
The physical constraints of stringed instruments discourage more widespread use of these polyphonies.
For vocal music, of course, physical constraints prevent intrainstrument polyphony entirely.
As Figure 4 illustrates, these more abstract parts can weave in and out of existence.
Two parts can merge with each other; a single part can split in two; new parts can emerge spontaneously.
The
KernScores data provides annotations that describe this behavior.
We can represent these dynamics of parts as a P × P ﬂow matrix at each time step (where P is an upper bound on the number of parts; in our corpus P = 6 ) that describes where each part moves in the next step.
At most time steps, this ﬂow matrix is the identity matrix.
The state-based models discussed in this paper can easily be adjusted to accommodate these ﬂows.
If two parts merge, sum their states; if a part splits in two, duplicate its state.
These operations amount to hitting the vector of state estimates for the parts with the ﬂow matrix at each time step.
However, we do not currently model the ﬂow matrix.
Because the ﬂow matrix for piano music contains some (small) amount of entropy, we therefore exclude piano music from the results reported in Table 3. We do however include the piano music in training.
B S EVERAL WAYS TO FACTOR THE DISTRIBUTION OVER SCORES
As discussed in section 4, we want to order a factor a score over notes.
A score consists of notes, each of which has a start time, part assignment, duration, and pitch.
Formally, we can think of a score as a list of notes S , where each note is a tuple ( s, p, n, d ) where s ∈ R + is the start time of the note (units of beats), p ∈ { 1 , . . . , P } is the part assignment of the note, n ∈ { 1 , . . . , N } is the pitch of the note (e.g. MIDI code), and d ∈ R + is the duration of the note (units of beats).
We would like to factor the distribution over scores, to take advantage of its autoregressive nature and other structure.
We are immediately faced with the question of how to handle continuous time.
B. D ISCRETE “R ASTER ” F ACTORIZATION
One approach to making predictions in a continuous domain is to discretize the domain.
As discussed in Section 5, we can choose a discrete time step ∆ > 0 such that s i
= k i ∆ and d i = j i ∆ with k ∈ N for all times s i and durations
d i in all scores.
In other words, we can discretize a score without losing information: we just need to specify its contents at each point on a discrete grid of resolution ∆ .
To this end we deﬁne a discretized score x ∈ { 0 , 1 } T × P × 2 N where, at each time step t ∈ { 1 , . . .
, T } , for each part p ∈ { 1 , . . .
, P } and each note n ∈ { 1 , . . .
, N } , we set x t,p,n =iff note n is on at time t in part p, x t,p, 2 n =iff note n begins at time t in part
p.
Observe that both a “note” bit and an “onset” bit are required to faithfully represent the score S ; with only a single bit to indicate the occurrence of a note, both the scores in Figure 7 would be confused with the score in Figure 5.
We can then factor the distribution over scores into a binary-valued conditional distributions: p ( x
[Equation]
=
[Equation]
(cid:89)
[Equation]
n
[Equation]
x
[Equation]
x
[Equation]
p
[Equation]
1: n ) .
Discretization is the approach taken by Boulanger-Lewandowski et al.
(2012), with some caveats.
First, their work doesn’t model parts (this can be recovered in our exposition by setting P = 1 ).
Second, their work doesn’t take the ﬁnal step of factoring p ( x t,p | x 1: t , x t, 1: p ) into binary-valued distributions; they choose rather to directly target the vector x t with NADE.
Third, their work using a single bit to indicate notes, thus losing expressivity in the manner discussed in Figure 7.
Fourth, rather than work with the process resolution ∆ , their work discretizes at a coarse resolution of eighth notes (quarter notes in the case of the JSB chorales segment of their corpus) thus losing substantial information.
Figure 6 illustrates the effect of discretizing the score in Figure 5 at eighth-note resolution.
B. R UN -L ENGTH F ACTORIZATION
Training and sampling from a model over a discrete factorization of scores at the process resolution ∆ can be expensive, prompting some earlier works to discretize at a coarser resolution (as discussed in the previous section).
One approach to preserve ﬁne rhythmic structure (e.g. triplets and thirtysecond notes) without committing to a ﬁne discretization is to factor a score into run-lengths.
To this end, we deﬁne a run-length encoded score x ∈ (cid:0) N ⊕ { 0 , 1 } P × 2 N (cid:1) T where, at each time index t ∈ { 1 , . . .
{+, T } , we set x t, 
+}
= 1 d t , where d t is the duration of the event at time index t, x t, 1 ,p,n =iff note n is on at time t in part p, x t, 1 ,p, 2 n =iff note n begins at time t in part
p.
Crucially, the sequence x t is non-linear in the index t : entry x t +1 occurs d t beats after entry x t , in contrast to the raster where x t +1 always occurs a constant interval ∆ after x t .
We can then factor the distribution over scores into conditional distributions over binary note values and natural-number duration values:
[Equation]
p
[Equation]
n
[Equation]
[Equation]
[Equation]
[Equation]
[Equation]
x
[Equation]
1 , 1: p , x t, 1 ,p, 1: n ) .
Because music typically doesn’t evolve at the ﬁnest possible resolution ∆ , we save a substantial amount of computation by predicting run-lengths x t, 0 ∈ N rather than re-iterating the predictions x t,p,n at successive time steps.
One criticism of the run-length factorization is that, when notes of different durations overlap in a score, the longer notes are chopped up along the boundaries of the short notes as illustrated in
Figure 8.
Instead of predicting musically meaningful quantities like note values (quarter, eighth, dotted-eighth, etc.)
instead we predict run-length chunks.
B. O PERATIONALIZED F ACTORIZATION
Both the raster and run-length factorizations discussed above don’t directly recognize the concept of note-values: quarter-notes, half-notes, eighth-notes, etc.. One approach to factorization that explicitly models note-values is to describe a score as a sequence of operations that produce the score, and then factor this sequence.
Formally, we can describe an operationalized score as a sequence x ∈ ( N ⊕ ( N ⊕ { 0 , 1 } P × N ))
L of length L , where each index
x i is three-hot with the following semantics:
[Equation]
[Equation]
x i is a time-shift operation: time advances by d steps , x i, 1 ,v = 1 and x i, 2 ,n =x i is a note operation: note n with note-value v begins playing .
This approach explicitly learns to predict note values v .
We remark that there are other operational factorizations that capture note values implicitly, like the raster or run-length encodings.
Another factorization, similar to Oore et al.
and inspired by the popular MIDI format, operationalizes a score as a sequence x ∈ ( N ⊕{ 0 , 1 } P × N ⊕{ 0 , 1 } P × N ) L of length L , where each index
x i is one-hot with the following semantics:
[Equation]
i
[Equation]
[Equation]
is a time-shift operation: time advances by d steps , x
[Equation]
[Equation]
p
[Equation]
n =x i is a note-on operation: note n begins playing in part p, x i, 2 ,p,n =x i is a note-off operation: note n stops playing in part
p.
In either case, we can then factor the distribution over these sequences:
[Equation]
While the ﬁrst operational approach describes above explicitly predicts note values, both operational factorizations require us to predict time-shifts, which are analogous to the durations predicting in the run-length factorization described in Appendix B.2.
We will now turn our attention to a single-part setting where run-lengths are equivalent to note values, thus eliminating these strange intermediate quantities.
B. R UN -L ENGTH F ACTORIZATION FOR P ARTS
Observe in Figure 8 that if we were to run-length encode or operationalize the top and bottom staff individually, the run-lengths or time-shifts respectively would correspond to note-values.
Because these parts are individually homophonic–notes within a part move in rhythmic lock-step–we do not observe the phenomenon where short notes overlap long notes in time, resulting in chopped up run-lengths.
Applying the run-length factorization described in Appendix B.2 we see that, in the context of homophonic parts, run-lengths are equivalent to note-values and therefore the run-length factorization explicitly models these note values.
The KernScores dataset takes this homophony property as the deﬁnition of a part (they call parts “voices”) and therefore encode the score in Figure 5 as two parts; in cases of polyphony as in Figure 4, the KernScores dataset decomposes the score into additional parts to maintain the homophony property.
Therefore, we can construct a run-length factorization of the KernScores parts that explicitly models note-values.
We describe a part by x ∈ ( N ⊕ { 0 , 1 } N ) T where, at each time index t ∈ { 1 , . .
{+, T } , we set x t, 
+}
= 1 d t , where d t is the duration of the note (or chord) at time index t, x t, 1 ,n =iff note n is on at time t.
In contrast to the run-length encoding described in Appendix B.2, we no longer need a second bit to distinguish onsets from continuations; because notes don’t get chopped up, we will never need a bit to indicating that we are re-iterating a note that was initiated in a previous run.
We can then proceed to factor the part as:
[Equation]
Learning the conditional distributions of pitches p ( x t,n | x 1: t , x t, 0 , x t, 1 , 1: n ) and note-values p ( x t, 0 | x 1: t ) is the homophonic modeling task considered in Section 5.
B. I NTERLACING P ARTS TO F ACTOR A S CORE
Let S be a score consisting of P parts.
The run-lengths that comprise each part can be a totally ordered by the timestamp each run-length begins.
We extend this to a total order across parts using these same timestamps, breaking ties (where run-lengths begin at the same time in multiple parts) using the arbitrary given by the indices of the parts.
For an index i in this total order, let P i be the part associated with this index and T
i be the time (by deﬁnition, T i is a non-decreasing sequence).
Finally, let L p be the number of run-lengths in part
p and deﬁne L ≡ (cid:80)
Pp =1 L p .
For example, consider the score in Figure 5.
In this case L 0 = 6 (the treble part) and L 1 = 8 (the bass part).
The time T and part P index sequences are given by
T = [0, 0, .5, 1, 1, 1.5, 1.75, 2, 2, 2.5, 3, 3, 3.5, 3.75],
P = [0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0].
{+We describe a score by x ∈ ( N ⊕ { 0 , 1 } N ) L where, at each index x i , we set x i, 
+}
= 1 d i , where d i is the duration of the note (or chord) at time T i in part P
i , x i, 1 ,n =iff note n is on at time T i in part P i .
We then proceed to factor this distribution in the same manner as the homophonic parts model:
[Equation]
This factorization models the next run in part P
( i ) as we did in the homophonic model, but additionally condition on runs in other parts that were initiated prior to the run we are currently predicting (and also on runs initiated at the same time in parts with indices less than p ).
Note that we do not need to predict T and P ; they are completely determined by the sequential ordering given on the x i ’s.
This is the polyphonic modeling task considered in Section 5.