Original: M USIC T RANSFORMER : G ENERATING MUSIC WITH LONG - TERM STRUCTURE
Revised: M USIC T RANSFORMER : G ENERATING MUSIC WITH LONG - TERM STRUCTURE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Music relies heavily on repetition to build structure and meaning.
Revised: Music relies heavily on repetition to build structure and meaning.

Original: Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure.
Revised: Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure.

Original: The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence.
Revised: The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence.

Original: This suggests that self-attention might also be well-suited to modeling music.
Revised: This suggests that self-attention might also be well-suited to modeling music.

Original: In musical composition and performance, however, relative timing is critically important.
Revised: In musical composition and performance, however, relative timing is critically important.

Original: Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018).
Revised: Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018).

Original: This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length.
Revised: This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length.

Original: We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length.
Revised: We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length.

Original: This enables us to demonstrate that
Revised: This enables us to demonstrate that

Original: a
Revised: a

Original: Transformer with our modiﬁed relative attention mechanism can generate minutelong (thousands of step, four times the length in Oore et al.
Revised: Transformer with our modiﬁed relative attention mechanism can generate minutelong (thousands of step, four times the length in Oore et al.

Original: compositions with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies 1 .
Revised: compositions with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies 1 .

Original: We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.
Revised: We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A musical piece often consists of recurring elements at various levels, from motifs to phrases to sections such as verse-chorus.
Revised: A musical piece often consists of recurring elements at various levels, from motifs to phrases to sections such as verse-chorus.

Original: To generate a coherent piece, a model needs to reference elements that came before, sometimes in the distant past, repeating, varying, and further developing them to create contrast and surprise.
Revised: To generate a coherent piece, a model needs to reference elements that came before, sometimes in the distant past, repeating, varying, and further developing them to create contrast and surprise.

Original: Intuitively, self-attention (Parikh et al., 2016) appears to be a good match for this task.
Revised: Intuitively, self-attention (Parikh et al., 2016) appears to be a good match for this task.

Original: Self-attention over its own previous outputs allows an autoregressive model to access any part of the previously generated output at every step of generation.
Revised: Self-attention over its own previous outputs allows an autoregressive model to access any part of the previously generated output at every step of generation.

Original: By contrast, recurrent neural networks have to learn to proactively store elements to be referenced in a ﬁxed size state or memory, potentially making training much more difﬁcult.
Revised: By contrast, recurrent neural networks have to learn to proactively store elements to be referenced in a ﬁxed size state or memory, potentially making training much more difﬁcult.

Original: We believe that repeating self-attention in multiple, successive layers of a Transformer decoder (Vaswani et al., 2017) helps capture the multiple levels at which self-referential phenomena exist in music.
Revised: We believe that repeating self-attention in multiple, successive layers of a Transformer decoder (Vaswani et al., 2017) helps capture the multiple levels at which self-referential phenomena exist in music.

Original: In its original formulation, the Transformer relies on absolute position representations, using either positional sinusoids or learned position embeddings that are added to the per-position input representations.
Revised: In its original formulation, the Transformer relies on absolute position representations, using either positional sinusoids or learned position embeddings that are added to the per-position input representations.

Original: Recurrent and convolutional neural networks instead model position in relative terms:
Revised: Recurrent and convolutional neural networks instead model position in relative terms:

Original: RNNs through their recurrence over the positions in their input, and CNNs by applying kernels that effectively choose which parameters to apply based on the relative position of the covered input representations.
Revised: RNNs through their recurrence over the positions in their input, and CNNs by applying kernels that effectively choose which parameters to apply based on the relative position of the covered input representations.

Original: Music has multiple dimensions along which relative differences arguably matter more than their absolute values; the two most prominent are timing and pitch.
Revised: Music has multiple dimensions along which relative differences arguably matter more than their absolute values; the two most prominent are timing and pitch.

Original: To capture such pairwise relations between representations, Shaw et al.
Revised: To capture such pairwise relations between representations, Shaw et al.

Original: (2018) introduce a relation-aware version of self-attention which they use successfully to modulate self-attention by the distance between two positions.
Revised: (2018) introduce a relation-aware version of self-attention which they use successfully to modulate self-attention by the distance between two positions.

Original: We extend this approach to capture relative timing and optionally also pitch, which yields improvement in both sample quality and perplexity for JSB Chorales.
Revised: We extend this approach to capture relative timing and optionally also pitch, which yields improvement in both sample quality and perplexity for JSB Chorales.

Original: As opposed to the original Transformer, samples from a Transformer with our relative attention mechanism maintain the regular timing grid present in this dataset.
Revised: As opposed to the original Transformer, samples from a Transformer with our relative attention mechanism maintain the regular timing grid present in this dataset.

Original: The model furthermore captures global timing, giving rise to regular phrases.
Revised: The model furthermore captures global timing, giving rise to regular phrases.

Original: The original formulation of relative attention (Shaw et al., 2018) requires O
Revised: The original formulation of relative attention (Shaw et al., 2018) requires O

Original: ( L 2 D ) memory where L is the sequence length and D is the dimension of the model’s hidden state.
Revised: ( L 2 D ) memory where L is the sequence length and D is the dimension of the model’s hidden state.

Original: This is prohibitive for long sequences such as those found in the Piano-e-Competition dataset of human-performed virtuosic, classical piano music.
Revised: This is prohibitive for long sequences such as those found in the Piano-e-Competition dataset of human-performed virtuosic, classical piano music.

Original: In Section 3.4, we show how to reduce the memory requirements to O ( LD ) , making it practical to apply relative attention to long sequences.
Revised: In Section 3.4, we show how to reduce the memory requirements to O ( LD ) , making it practical to apply relative attention to long sequences.

Original: The Piano-e-Competition dataset consists of MIDI recorded from performances of competition participants, bearing expressive dynamics and timing on the granularity of < 10 miliseconds.
Revised: The Piano-e-Competition dataset consists of MIDI recorded from performances of competition participants, bearing expressive dynamics and timing on the granularity of < 10 miliseconds.

Original: Discretizing time on a ﬁxed grid that would yield unnecessarily long sequences as not all events change on the same timescale.
Revised: Discretizing time on a ﬁxed grid that would yield unnecessarily long sequences as not all events change on the same timescale.

Original: We hence adopt a sparse, MIDI-like, event-based representation from (Oore et al., 2018), allowing a minute of music with 10 milisecond resolution to be represented at lengths around 2K, as opposed to 6K to 18K on a ﬁxed-grid representation with multiple performance attributes.
Revised: We hence adopt a sparse, MIDI-like, event-based representation from (Oore et al., 2018), allowing a minute of music with 10 milisecond resolution to be represented at lengths around 2K, as opposed to 6K to 18K on a ﬁxed-grid representation with multiple performance attributes.

Original: As position in sequence no longer corresponds to time, a priori it is not obvious that relative attention should work as well with such a representation.
Revised: As position in sequence no longer corresponds to time, a priori it is not obvious that relative attention should work as well with such a representation.

Original: However, we will show in Section 4.2 that it does improve perplexity and sample quality over strong baselines.
Revised: However, we will show in Section 4.2 that it does improve perplexity and sample quality over strong baselines.

Original: We speculate that idiomatic piano gestures such as scales, arpeggios and other motifs all exhibit a certain grammar and recur periodically, hence knowing their relative positional distances makes it easier to model this regularity.
Revised: We speculate that idiomatic piano gestures such as scales, arpeggios and other motifs all exhibit a certain grammar and recur periodically, hence knowing their relative positional distances makes it easier to model this regularity.

Original: This inductive bias towards learning relational information, as opposed to patterns based on absolute position, suggests that the Transformers with relative attention could generalize beyond the lengths it was trained on, which our experiments in Section 4.2.1 conﬁrm.
Revised: This inductive bias towards learning relational information, as opposed to patterns based on absolute position, suggests that the Transformers with relative attention could generalize beyond the lengths it was trained on, which our experiments in Section 4.2.1 conﬁrm.

Original: 1. C ONTRIBUTIONS Domain contributions
Revised: 1. C ONTRIBUTIONS Domain contributions

Original: We show the ﬁrst successful use of Transformers in generating music that exhibits long-term structure.
Revised: We show the ﬁrst successful use of Transformers in generating music that exhibits long-term structure.

Original: Before our work, LSTMs were used at timescales of 15s (~500 tokens) on the Piano-e-Competition dataset (Oore et al., 2018).
Revised: Before our work, LSTMs were used at timescales of 15s (~500 tokens) on the Piano-e-Competition dataset (Oore et al., 2018).

Original: Our work show that Transformers not only achieve state-of-the-art perplexity on modeling these complex expressive piano performances, but can also generate them at the scale of 60s (~2000 tokens) with remarkable internal consistency.
Revised: Our work show that Transformers not only achieve state-of-the-art perplexity on modeling these complex expressive piano performances, but can also generate them at the scale of 60s (~2000 tokens) with remarkable internal consistency.

Original: Our relative attention mechanism is essential to the model’s quality.
Revised: Our relative attention mechanism is essential to the model’s quality.

Original: In listening tests (see Section 4.2.3), samples from models with relative self-attention were perceived as more coherent than the baseline
Revised: In listening tests (see Section 4.2.3), samples from models with relative self-attention were perceived as more coherent than the baseline

Original: Transformer model from Vaswani et al.
Revised: Transformer model from Vaswani et al.

Original: Relative attention not only enables Transformers to generate continuations that elaborate on a given motif, but also to generalize and generate in consistent fashion beyond the length it was trained on (see Section 4.2.1).
Revised: Relative attention not only enables Transformers to generate continuations that elaborate on a given motif, but also to generalize and generate in consistent fashion beyond the length it was trained on (see Section 4.2.1).

Original: In a seq2seq setup, Transformers can generate accompaniments conditioned on melodies, enabling users to interact with the model.
Revised: In a seq2seq setup, Transformers can generate accompaniments conditioned on melodies, enabling users to interact with the model.

Original: Algorithmic contributions
Revised: Algorithmic contributions

Original: The space complexity of the relative self attention mechanism in its original formulation (Shaw et al., 2018) made it infeasible to train on sequences of sufﬁcient length to capture long-range structure in longer musical compositions.
Revised: The space complexity of the relative self attention mechanism in its original formulation (Shaw et al., 2018) made it infeasible to train on sequences of sufﬁcient length to capture long-range structure in longer musical compositions.

Original: Addressing this we present a crucial algorithmic improvement to the relative self attention mechanism, dramatically reducing its memory requirements from O ( L 2 D ) to O ( LD ) .
Revised: Addressing this we present a crucial algorithmic improvement to the relative self attention mechanism, dramatically reducing its memory requirements from O ( L 2 D ) to O ( LD ) .

Original: For example, we reduce the memory consumption from 1 .
Revised: For example, we reduce the memory consumption from 1 .

Original: GB to 0 .
Revised: GB to 0 .

Original: 52 MB per layer for a sequence of length L = 2048 and hidden-state size D = 

Revised: 52 MB per layer for a sequence of length L = 2048 and hidden-state size D =

Original: (per head D h = DH = 64 , where typically H = 8 for the number of heads) (see Table 1), allowing us to use GPUs to train the relative self-attention Transformer on long sequences.
Revised: (per head D h = DH = 64 , where typically H = 8 for the number of heads) (see Table 1), allowing us to use GPUs to train the relative self-attention Transformer on long sequences.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Sequence models have been the canonical choice for modeling music, from Hidden Markov Models to RNNs and Long Short Term Memory networks (e.g., Eck & Schmidhuber, 2002; Liang, 2016;
Revised: Sequence models have been the canonical choice for modeling music, from Hidden Markov Models to RNNs and Long Short Term Memory networks (e.g., Eck & Schmidhuber, 2002; Liang, 2016;

Original: Oore et al., 2018), to bidirectional LSTMs (e.g., Hadjeres et al., 2017).
Revised: Oore et al., 2018), to bidirectional LSTMs (e.g., Hadjeres et al., 2017).

Original: Successful application of sequential models to polyphonic music often requires serializing the musical score or performance into a single sequence, for example by interleaving different instruments or voices.
Revised: Successful application of sequential models to polyphonic music often requires serializing the musical score or performance into a single sequence, for example by interleaving different instruments or voices.

Original: Alternatively, a 2D pianoroll-like representation (see A.1 for more details) can be decomposed into a sequence of multi-hot pitch vectors, and their joint probability distributions can be captured using Restricted
Revised: Alternatively, a 2D pianoroll-like representation (see A.1 for more details) can be decomposed into a sequence of multi-hot pitch vectors, and their joint probability distributions can be captured using Restricted

Original: Boltzmann Machines (Smolensky, 1986; Hinton et al., 2006) or Neural Autoregressive Distribution
Revised: Boltzmann Machines (Smolensky, 1986; Hinton et al., 2006) or Neural Autoregressive Distribution

Original: Estimators (NADE; Larochelle & Murray, 2011).
Revised: Estimators (NADE; Larochelle & Murray, 2011).

Original: Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks (e.g., Dong et al., 2018) or as orderless
Revised: Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks (e.g., Dong et al., 2018) or as orderless

Original: NADEs (Uria et al., 2014; 2016)
Revised: NADEs (Uria et al., 2014; 2016)

Original: (e.g., Huang et al., 2017).
Revised: (e.g., Huang et al., 2017).

Original: Lattner et al.
Revised: Lattner et al.

Original: (2018) use self-similarity in style-transfer fashion, where the self-similarity structure of a piece serves as a template objective for gradient descent to impose similar repetition structure on an input score.
Revised: (2018) use self-similarity in style-transfer fashion, where the self-similarity structure of a piece serves as a template objective for gradient descent to impose similar repetition structure on an input score.

Original: Self-attention can be seen as a generalization of self-similarity; the former maps the input through different projections to queries and keys, and the latter uses the same projection for both.
Revised: Self-attention can be seen as a generalization of self-similarity; the former maps the input through different projections to queries and keys, and the latter uses the same projection for both.

Original: Dot-product self-attention is the mechanism at the core of the Transformer, and several recent works have focused on applying and improving it for image generation, speech, and summarization (Parmar et al., 2018; Povey et al., 2018; Liu et al., 2018).
Revised: Dot-product self-attention is the mechanism at the core of the Transformer, and several recent works have focused on applying and improving it for image generation, speech, and summarization (Parmar et al., 2018; Povey et al., 2018; Liu et al., 2018).

Original: A key challenge encountered by each of these efforts is scaling attention computationally to long sequences.
Revised: A key challenge encountered by each of these efforts is scaling attention computationally to long sequences.

Original: This is because the time and space complexity of self-attention grows quadratically in the sequence length.
Revised: This is because the time and space complexity of self-attention grows quadratically in the sequence length.

Original: For relative self-attention (Shaw et al., 2018) this is particularly problematic as the space complexity also grows linearly in the dimension, or depth, of the per-position representations.
Revised: For relative self-attention (Shaw et al., 2018) this is particularly problematic as the space complexity also grows linearly in the dimension, or depth, of the per-position representations.

Original:  M ODEL 3. D ATA REPRESENTATION
Revised:  M ODEL 3. D ATA REPRESENTATION

Original: We take a language-modeling approach to training generative models for symbolic music.
Revised: We take a language-modeling approach to training generative models for symbolic music.

Original: Hence we represent music as a sequence of discrete tokens, with the vocabulary determined by the dataset.
Revised: Hence we represent music as a sequence of discrete tokens, with the vocabulary determined by the dataset.

Original: Datasets in different genres call for different ways of serializing polyphonic music into a single stream and also discretizing time.
Revised: Datasets in different genres call for different ways of serializing polyphonic music into a single stream and also discretizing time.

Original: The JSB Chorale dataset consists of four-part scored choral music, which can be represented as a matrix where rows correspond to voices and columns to time discretized to sixteenth notes.
Revised: The JSB Chorale dataset consists of four-part scored choral music, which can be represented as a matrix where rows correspond to voices and columns to time discretized to sixteenth notes.

Original: The matrix’s entries are integers that denote which pitch is being played.
Revised: The matrix’s entries are integers that denote which pitch is being played.

Original: This matrix can than be serialized in raster-scan fashion by ﬁrst going down the rows and then moving right through the columns (see A.1 for more details).
Revised: This matrix can than be serialized in raster-scan fashion by ﬁrst going down the rows and then moving right through the columns (see A.1 for more details).

Original: Compared to JSB Chorale, the piano performance data in the
Revised: Compared to JSB Chorale, the piano performance data in the

Original: Piano-e-Competition dataset includes expressive timing information at much ﬁner granularity and more voices.
Revised: Piano-e-Competition dataset includes expressive timing information at much ﬁner granularity and more voices.

Original: For the Piano-e-Competition we therefore use the performance encoding proposed by Oore et al.
Revised: For the Piano-e-Competition we therefore use the performance encoding proposed by Oore et al.

Original: (2018) which consists of a vocabulary of 128 NOTE_ON events, 128 NOTE_OFF s, 100 TIME_SHIFT s allowing for expressive timing at 10ms and 32 VELOCITY bins for expressive dynamics (see A.2 for more details).
Revised: (2018) which consists of a vocabulary of 128 NOTE_ON events, 128 NOTE_OFF s, 100 TIME_SHIFT s allowing for expressive timing at 10ms and 32 VELOCITY bins for expressive dynamics (see A.2 for more details).

Original: 3. B ACKGROUND : S ELF - ATTENTION IN T RANSFORMER
Revised: 3. B ACKGROUND : S ELF - ATTENTION IN T RANSFORMER

Original: The Transformer decoder is a autoregressive generative model that uses primarily self-attention mechanisms, and learned or sinusoidal position information.
Revised: The Transformer decoder is a autoregressive generative model that uses primarily self-attention mechanisms, and learned or sinusoidal position information.

Original: Each layer consists of a self-attention sub-layer followed by a feedforward sub-layer.
Revised: Each layer consists of a self-attention sub-layer followed by a feedforward sub-layer.

Original: The attention layer ﬁrst transforms a sequence of L D -dimensional vectors X = ( x 1 , x 2 , . . .
Revised: The attention layer ﬁrst transforms a sequence of L D -dimensional vectors X = ( x 1 , x 2 , . . .

Original: , x L ) into queries Q = XW Q , keys K = XW K , and values V = XW V , where W Q , W K , and W V are each D
Revised: , x L ) into queries Q = XW Q , keys K = XW K , and values V = XW V , where W Q , W K , and W V are each D

Original: × D square matrices.
Revised: × D square matrices.

Original: Each L × D query, key, and value matrix is then split into
Revised: Each L × D query, key, and value matrix is then split into

Original: H L
Revised: H L

Original: × D h parts or attention heads, indexed by h , and with dimension D h = DH , which allow the model to focus on different parts of the history.
Revised: × D h parts or attention heads, indexed by h , and with dimension D h = DH , which allow the model to focus on different parts of the history.

Original: The scaled dot-product attention computes a sequence of vector outputs for each head as
Revised: The scaled dot-product attention computes a sequence of vector outputs for each head as

Original: [Equation]
Revised: [Equation]

Original: The attention outputs for each head are concatenated and linearly transformed to get Z , a L by D dimensional matrix.
Revised: The attention outputs for each head are concatenated and linearly transformed to get Z , a L by D dimensional matrix.

Original: A upper triangular mask ensures that queries cannot attend to keys later in the sequence.
Revised: A upper triangular mask ensures that queries cannot attend to keys later in the sequence.

Original: For other details of the Transfomer model, such as residual connections and learning rates, the reader can refer Vaswani et al.
Revised: For other details of the Transfomer model, such as residual connections and learning rates, the reader can refer Vaswani et al.

Original: The feedforward (FF) sub-layer then takes the output Z from the previous attention sub-layer, and performs two layers of point-wise dense layers on the depth
Revised: The feedforward (FF) sub-layer then takes the output Z from the previous attention sub-layer, and performs two layers of point-wise dense layers on the depth

Original: D dimension, as shown in Equation 2.
Revised: D dimension, as shown in Equation 2.

Original: W 1 , W 2 , b 1 , b 2 are weights and biases of those two layers.
Revised: W 1 , W 2 , b 1 , b 2 are weights and biases of those two layers.

Original: [Equation]
Revised: [Equation]

Original: 3. R ELATIVE POSITIONAL SELF - ATTENTION
Revised: 3. R ELATIVE POSITIONAL SELF - ATTENTION

Original: As the Transformer model relies solely on positional sinusoids to represent timing information, Shaw et al.
Revised: As the Transformer model relies solely on positional sinusoids to represent timing information, Shaw et al.

Original: (2018) introduced relative position representations to allow attention to be informed by how far two positions are apart in a sequence.
Revised: (2018) introduced relative position representations to allow attention to be informed by how far two positions are apart in a sequence.

Original: This involves learning a separate relative position embedding
Revised: This involves learning a separate relative position embedding

Original: E r of shape ( H, L, D h ) , which has an embedding for each possible pairwise distance r =
Revised: E r of shape ( H, L, D h ) , which has an embedding for each possible pairwise distance r =

Original: j k − i q between a query and key in position i q and j k respectively.
Revised: j k − i q between a query and key in position i q and j k respectively.

Original: The embeddings are ordered from distance − L + 1 to 0 , and are learned separately for each head.
Revised: The embeddings are ordered from distance − L + 1 to 0 , and are learned separately for each head.

Original: In Shaw et al.
Revised: In Shaw et al.

Original: (2018), the relative embeddings interact with queries and give rise to a S rel , an L × L dimensional logits matrix which modulates the attention probabilities for each head as:
Revised: (2018), the relative embeddings interact with queries and give rise to a S rel , an L × L dimensional logits matrix which modulates the attention probabilities for each head as:

Original: [Equation]
Revised: [Equation]

Original: We dropped head indices for clarity.
Revised: We dropped head indices for clarity.

Original: Our work uses the same approach to infuse relative distance information in the attention computation, while signiﬁcantly improving upon the memory footprint for computing S rel .
Revised: Our work uses the same approach to infuse relative distance information in the attention computation, while signiﬁcantly improving upon the memory footprint for computing S rel .

Original: For each head, Shaw et al.
Revised: For each head, Shaw et al.

Original: (2018) instantiate an intermediate tensor R of shape ( L, L, D h ) , containing the embeddings that correspond to the relative distances between all keys and queries.
Revised: (2018) instantiate an intermediate tensor R of shape ( L, L, D h ) , containing the embeddings that correspond to the relative distances between all keys and queries.

Original: Q is then reshaped to an ( L, 1 , D h ) tensor, and S rel =
Revised: Q is then reshaped to an ( L, 1 , D h ) tensor, and S rel =

Original: QR (cid:62) .
Revised: QR (cid:62) .

Original: This incurs a total space complexity of O
Revised: This incurs a total space complexity of O

Original: ( L 2 D ) , restricting its application to long sequences.
Revised: ( L 2 D ) , restricting its application to long sequences.

Original: 3. M EMORY EFFICIENT IMPLEMENTATION OF RELATIVE POSITION - BASED ATTENTION
Revised: 3. M EMORY EFFICIENT IMPLEMENTATION OF RELATIVE POSITION - BASED ATTENTION

Original: We improve the implementation of relative attention by reducing its intermediate memory requirement from O ( L 2 D ) to O ( LD ) , with example lengths shown in Table 1.
Revised: We improve the implementation of relative attention by reducing its intermediate memory requirement from O ( L 2 D ) to O ( LD ) , with example lengths shown in Table 1.

Original: We observe that all of the terms we need from QR (cid:62) are already available if we directly multiply Q with E r , the relative position embedding.
Revised: We observe that all of the terms we need from QR (cid:62) are already available if we directly multiply Q with E r , the relative position embedding.

Original: After we compute QE r (cid:62) , its ( i q , r ) entry contains the dot product of the query in position i q with the embedding of relative distance r .
Revised: After we compute QE r (cid:62) , its ( i q , r ) entry contains the dot product of the query in position i q with the embedding of relative distance r .

Original: However, each relative logit ( i q , j k ) in the matrix S rel from Equation 3 should be the dot product of the query in position i q and the embedding of the relative distance j k − i q , to match up with the indexing in QK (cid:62) .
Revised: However, each relative logit ( i q , j k ) in the matrix S rel from Equation 3 should be the dot product of the query in position i q and the embedding of the relative distance j k − i q , to match up with the indexing in QK (cid:62) .

Original: We therefore need to “skew”
Revised: We therefore need to “skew”

Original: QE r (cid:62) so as to move the relative logits to their correct positions, as illustrated in Figure 1 and detailed in the next section.
Revised: QE r (cid:62) so as to move the relative logits to their correct positions, as illustrated in Figure 1 and detailed in the next section.

Original: The time complexity for both methods are O
Revised: The time complexity for both methods are O

Original: ( L 2 D ) , while in practice our method is 6x faster at length 650.
Revised: ( L 2 D ) , while in practice our method is 6x faster at length 650.

Original: Implementation
Revised: Implementation

Original: Relative memory
Revised: Relative memory

Original: Maximal L
Revised: Maximal L

Original: L = L =
Revised: L = L =

Original: L =
Revised: L =

Original: Shaw et al.
Revised: Shaw et al.

Original: O ( L 2 D + L 2 ) 108 + 1.
Revised: O ( L 2 D + L 2 ) 108 + 1.

Original: 1100 +3100 +
Revised: 1100 +3100 +

Original: Ours
Revised: Ours

Original: O ( LD + L 2 ) 0.17 +
Revised: O ( LD + L 2 ) 0.17 +

Original: [Table]
Revised: [Table]

Original: 0.52 +0.

Revised: 0.52 +0.

Original: +
Revised: +

Original: 3.4. T HE “ SKEWING ” PROCEDURE
Revised: 3.4. T HE “ SKEWING ” PROCEDURE

Original: Hence, we propose a “skewing” procedure to transform an absolute-by-relative ( i q , r ) indexed matrix into an absolute-by-absolute ( i q , j k ) indexed matrix.
Revised: Hence, we propose a “skewing” procedure to transform an absolute-by-relative ( i q , r ) indexed matrix into an absolute-by-absolute ( i q , j k ) indexed matrix.

Original: The row indices i q stay the same while the columns indices are shifted according to the following equation: j k = r − ( L − 1)
Revised: The row indices i q stay the same while the columns indices are shifted according to the following equation: j k = r − ( L − 1)

Original: + i q .
Revised: + i q .

Original: For example in Figure 1 the upper right green dot in position (0 , 2) of QE r (cid:62) after skewing has a column index of 2 − (3 − 1) + 0 = 0 , resulting in a position of (0 , 0) in S rel .
Revised: For example in Figure 1 the upper right green dot in position (0 , 2) of QE r (cid:62) after skewing has a column index of 2 − (3 − 1) + 0 = 0 , resulting in a position of (0 , 0) in S rel .

Original: We outline the steps illustrated in
Revised: We outline the steps illustrated in

Original: Figure 1 below.
Revised: Figure 1 below.

Original: Pad a dummy column vector of length L before
Revised: Pad a dummy column vector of length L before

Original: the leftmost
Revised: the leftmost

Original: column.
Revised: column.

Original: Reshape the matrix to have shape ( L +1 , L ) .
Revised: Reshape the matrix to have shape ( L +1 , L ) .

Original: (This step assumes NumPy-style row-major ordering.)
Revised: (This step assumes NumPy-style row-major ordering.)

Original: Slice that matrix to retain only the last l rows and all the columns, resulting in a ( L, L ) matrix again, but now absolute-by-absolute indexed, which is the S rel that we need.
Revised: Slice that matrix to retain only the last l rows and all the columns, resulting in a ( L, L ) matrix again, but now absolute-by-absolute indexed, which is the S rel that we need.

Original: 3. R ELATIVE LOCAL ATTENTION
Revised: 3. R ELATIVE LOCAL ATTENTION

Original: For very long sequences, the quadratic memory requirement of even baseline Transformer is impractical.
Revised: For very long sequences, the quadratic memory requirement of even baseline Transformer is impractical.

Original: Local attention has been used for example in Wikipedia and image generation (Liu et al., 2018;
Revised: Local attention has been used for example in Wikipedia and image generation (Liu et al., 2018;

Original: Parmar et al., 2018) by chunking the input sequence into non-overlapping blocks.
Revised: Parmar et al., 2018) by chunking the input sequence into non-overlapping blocks.

Original: Each block then attends to itself and the one before, as shown by the smaller thumbnail on the top right corner of
Revised: Each block then attends to itself and the one before, as shown by the smaller thumbnail on the top right corner of

Original: Figure 2.
Revised: Figure 2.

Original: To extend relative attention to the local case, we ﬁrst note that the right block has the same conﬁguration as in the global case (see Figure 1) but much smaller: ( LM ) 2 (where M is the number of blocks, and N be the resulting block length) as opposed to L 2 .
Revised: To extend relative attention to the local case, we ﬁrst note that the right block has the same conﬁguration as in the global case (see Figure 1) but much smaller: ( LM ) 2 (where M is the number of blocks, and N be the resulting block length) as opposed to L 2 .

Original: The left block is unmasked with relative indices running from -1 (top right) to - 2 N + 1 (bottom left).
Revised: The left block is unmasked with relative indices running from -1 (top right) to - 2 N + 1 (bottom left).

Original: Hence, the learned E r for the local case has shape (2 N − 1 , N ) .
Revised: Hence, the learned E r for the local case has shape (2 N − 1 , N ) .

Original: Similar to the global case, we ﬁrst compute QE r (cid:62) and then use the following procedure to skew it to have the same indexing as QK (cid:62) , as illustrated in Figure 2.
Revised: Similar to the global case, we ﬁrst compute QE r (cid:62) and then use the following procedure to skew it to have the same indexing as QK (cid:62) , as illustrated in Figure 2.

Original: Pad a dummy column vector of length N after the rightmost column.
Revised: Pad a dummy column vector of length N after the rightmost column.

Original: Flatten the matrix and then pad with a dummy row of length N − 1 . 3.
Revised: Flatten the matrix and then pad with a dummy row of length N − 1 . 3.

Original: Reshape the matrix to have shape ( N + 1 , 2 N − 1) .
Revised: Reshape the matrix to have shape ( N + 1 , 2 N − 1) .

Original: Slice that matrix to retain only the ﬁrst N rows and last N columns, resulting in a ( N, N ) matrix.
Revised: Slice that matrix to retain only the ﬁrst N rows and last N columns, resulting in a ( N, N ) matrix.

Original:  E XPERIMENTS 4. J.S. B ACH C HORALES
Revised:  E XPERIMENTS 4. J.S. B ACH C HORALES

Original: J.S. Bach chorales is a canonical dataset used for evaluating generative models for music 3 (e.g.,
Revised: J.S. Bach chorales is a canonical dataset used for evaluating generative models for music 3 (e.g.,

Original: Allan & Williams, 2005; Boulanger-Lewandowski et al., 2012; Liang, 2016; Hadjeres et al., 2016;
Revised: Allan & Williams, 2005; Boulanger-Lewandowski et al., 2012; Liang, 2016; Hadjeres et al., 2016;

Original: Huang et al., 2017).
Revised: Huang et al., 2017).

Original: It consists of score-based four-part chorales.
Revised: It consists of score-based four-part chorales.

Original: We ﬁrst discretize the scores onto a 16th-note grid, and then serialize it by iterating through all the voices within a time step and then advancing time (see A.1 for more details).
Revised: We ﬁrst discretize the scores onto a 16th-note grid, and then serialize it by iterating through all the voices within a time step and then advancing time (see A.1 for more details).

Original: As there is a direct correspondence between position in sequence and position on the timing/instrument grid in a piece, adding relative position representations could make it easier to learn this grammar.
Revised: As there is a direct correspondence between position in sequence and position on the timing/instrument grid in a piece, adding relative position representations could make it easier to learn this grammar.

Original: We indeed see relative attention drastically improve negative log-likelihood (NLL) over baseline Transformer (Table 2).
Revised: We indeed see relative attention drastically improve negative log-likelihood (NLL) over baseline Transformer (Table 2).

Original: This improvement is also reﬂected in sample quality.
Revised: This improvement is also reﬂected in sample quality.

Original: The samples now maintain the necessary timing/instrument grid, always advancing four steps before advancing in time.
Revised: The samples now maintain the necessary timing/instrument grid, always advancing four steps before advancing in time.

Original: As local timing is maintained, the model is able to capture timing on a more global level, giving rise to regular phrasing, as shown in Figure 3.
Revised: As local timing is maintained, the model is able to capture timing on a more global level, giving rise to regular phrasing, as shown in Figure 3.

Original: In addition to relative attention, we also explored enhancing absolute timing through concatenating instead of adding the sinusoids to the input embeddings.
Revised: In addition to relative attention, we also explored enhancing absolute timing through concatenating instead of adding the sinusoids to the input embeddings.

Original: This allows the model to more directly learn its absolute positional mapping.
Revised: This allows the model to more directly learn its absolute positional mapping.

Original: This further improves performance for both the baseline and relative transformer (Table 2).
Revised: This further improves performance for both the baseline and relative transformer (Table 2).

Original: We compare against C OCONET as it is one of the best-performing models that has also been evaluated on the 16-note grid using the canonical dataset split.
Revised: We compare against C OCONET as it is one of the best-performing models that has also been evaluated on the 16-note grid using the canonical dataset split.

Original: To directly compare, we re-evaluated C OCONET to obtain note-wise losses on the validation set 4 .
Revised: To directly compare, we re-evaluated C OCONET to obtain note-wise losses on the validation set 4 .

Original: For the Transformer models (abbreviated as TF), we implemented our attention mechanisms in the
Revised: For the Transformer models (abbreviated as TF), we implemented our attention mechanisms in the

Original: Tensor2Tensor framework (Vaswani et al., 2018).
Revised: Tensor2Tensor framework (Vaswani et al., 2018).

Original: We use 8 heads, and keep the query, key (att) and value hidden size (hs) ﬁxed within a conﬁg.
Revised: We use 8 heads, and keep the query, key (att) and value hidden size (hs) ﬁxed within a conﬁg.

Original: We tuned number of layers (L in {4,5,6}), attention hidden size (att in {256, 512}) and pointwise feedforward hidden size (ff in {512, 1024}).
Revised: We tuned number of layers (L in {4,5,6}), attention hidden size (att in {256, 512}) and pointwise feedforward hidden size (ff in {512, 1024}).

Original: 4.1. G ENERALIZING RELATIVE ATTENTION TO CAPTURE RELATIONAL INFORMATION
Revised: 4.1. G ENERALIZING RELATIVE ATTENTION TO CAPTURE RELATIONAL INFORMATION

Original: A musical event bears multiple attributes, such as timing, pitch, instrument etc.
Revised: A musical event bears multiple attributes, such as timing, pitch, instrument etc.

Original: To capture more relational information, we extend relative attention to capture pairwise distances on additional attributes.
Revised: To capture more relational information, we extend relative attention to capture pairwise distances on additional attributes.

Original: We learn separate relative embeddings for timing E t and also pitch E p .
Revised: We learn separate relative embeddings for timing E t and also pitch E p .

Original: E t has entries corresponding to how many sixteenth notes apart are two positions, while E p embeds the pairwise pitch interval.
Revised: E t has entries corresponding to how many sixteenth notes apart are two positions, while E p embeds the pairwise pitch interval.

Original: However this approach is not directly scalable beyond J.S. Bach Chorales because it involves explicitly gathering relative embeddings for R t and R p , resulting in a memory complexity of O
Revised: However this approach is not directly scalable beyond J.S. Bach Chorales because it involves explicitly gathering relative embeddings for R t and R p , resulting in a memory complexity of O

Original: ( L 2 D ) as in Shaw et al.
Revised: ( L 2 D ) as in Shaw et al.

Original: This is due to relative information being computed based on content as opposed to content-invariant information such as position in sequence.
Revised: This is due to relative information being computed based on content as opposed to content-invariant information such as position in sequence.

Original: It was sufﬁcient to add the extra timing signals to the ﬁrst layer, perhaps because it is closest to the raw input content.
Revised: It was sufﬁcient to add the extra timing signals to the ﬁrst layer, perhaps because it is closest to the raw input content.

Original: Here, the relative logits are computed from three terms, S rel =
Revised: Here, the relative logits are computed from three terms, S rel =

Original: Skew ( QE r )
Revised: Skew ( QE r )

Original: + Q ( R t + R p ) in contrast with other layers that only have one term, Skew ( QE r ) .
Revised: + Q ( R t + R p ) in contrast with other layers that only have one term, Skew ( QE r ) .

Original: 4. P IANO - E -C OMPETITION
Revised: 4. P IANO - E -C OMPETITION

Original: We use the ﬁrst 6 years of of Piano-e-Competition because these years have corresponding MIDI data released 5 , resulting in about 1100 pieces, split 80/10/10.
Revised: We use the ﬁrst 6 years of of Piano-e-Competition because these years have corresponding MIDI data released 5 , resulting in about 1100 pieces, split 80/10/10.

Original: Each piece is MIDI data capturing a classical piano performance with expressive dynamics and timing, encoded with the MIDI-like representation described in Section A.2.
Revised: Each piece is MIDI data capturing a classical piano performance with expressive dynamics and timing, encoded with the MIDI-like representation described in Section A.2.

Original: We trained on random crops of 2000-token sequences and employed two kinds of data augmentation: pitch transpositions uniformly sampled from {− 3 , − 2 , . . .
Revised: We trained on random crops of 2000-token sequences and employed two kinds of data augmentation: pitch transpositions uniformly sampled from {− 3 , − 2 , . . .

Original: , 2 , 3 } half-steps, and time stretches uniformly sampled from the set { 0 . 95 , 0 . 975 , 1 . 0 , 1 . 025 , 1 . 05 } .
Revised: , 2 , 3 } half-steps, and time stretches uniformly sampled from the set { 0 . 95 , 0 . 975 , 1 . 0 , 1 . 025 , 1 . 05 } .

Original: We compare to Magenta’s PerformanceRNN (LSTM, which ﬁrst used this dataset) (Oore et al., 2018) and LookBack RNN (LSTM with attention) (Waite, 2016).
Revised: We compare to Magenta’s PerformanceRNN (LSTM, which ﬁrst used this dataset) (Oore et al., 2018) and LookBack RNN (LSTM with attention) (Waite, 2016).

Original: LookBack RNN uses an input representation that requires monophonic music with barlines which is information that is not present in performed polyphonic music data, hence we simply adopt their architecture.
Revised: LookBack RNN uses an input representation that requires monophonic music with barlines which is information that is not present in performed polyphonic music data, hence we simply adopt their architecture.

Original: Table 3 shows that
Revised: Table 3 shows that

Original: Transformer-based architectures ﬁts this dataset better than LSTM-based models.
Revised: Transformer-based architectures ﬁts this dataset better than LSTM-based models.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We implemented our attention mechanisms in the Tensor2Tensor framework (Vaswani et al., 2018), and used the default hyperparameters for training, with 0.1 learning rate, 0.1 dropout, and early stopping.
Revised: We implemented our attention mechanisms in the Tensor2Tensor framework (Vaswani et al., 2018), and used the default hyperparameters for training, with 0.1 learning rate, 0.1 dropout, and early stopping.

Original: We compare four architectures, varying on two axes: global versus local, and regular versus relative attention.
Revised: We compare four architectures, varying on two axes: global versus local, and regular versus relative attention.

Original: We found that reducing the query and key hidden size (att) to half the hidden size (hs) works well and use this relationship for all of the models, while tuning on number of layers (L) and ﬁlter size (fs).
Revised: We found that reducing the query and key hidden size (att) to half the hidden size (hs) works well and use this relationship for all of the models, while tuning on number of layers (L) and ﬁlter size (fs).

Original: We use block size (bs) 512 for local attention.
Revised: We use block size (bs) 512 for local attention.

Original: We set the maximum relative distance to consider to half the training sequence length for relative global attention, and to the full memory length (which is two blocks) for relative local attention.
Revised: We set the maximum relative distance to consider to half the training sequence length for relative global attention, and to the full memory length (which is two blocks) for relative local attention.

Original: Table 3 show that relative attention (global or local) outperforms regular self-attention (global or local).
Revised: Table 3 show that relative attention (global or local) outperforms regular self-attention (global or local).

Original: All else being equal, local and global attention perform similarly.
Revised: All else being equal, local and global attention perform similarly.

Original: Each though local attention does not see all the history at once, it can build up a larger receptive ﬁeld across layers.
Revised: Each though local attention does not see all the history at once, it can build up a larger receptive ﬁeld across layers.

Original: This can be an advantage in the future for training on much longer sequences, as local attention requires much less memory.
Revised: This can be an advantage in the future for training on much longer sequences, as local attention requires much less memory.

Original: 4.2. Q UALITATIVE PRIMING EXPERIMENTS
Revised: 4.2. Q UALITATIVE PRIMING EXPERIMENTS

Original: When primed with an initial motif (Chopin’s Étude Op.
Revised: When primed with an initial motif (Chopin’s Étude Op.

Original: 10, No. 5) shown in the top left corner of
Revised: 10, No. 5) shown in the top left corner of

Original: Figure 4, we see the models perform qualitatively differently.
Revised: Figure 4, we see the models perform qualitatively differently.

Original: Transformer with relative attention elaborates the motif and creates phrases with clear contour which are repeated and varied.
Revised: Transformer with relative attention elaborates the motif and creates phrases with clear contour which are repeated and varied.

Original: Baseline
Revised: Baseline

Original: Transformer uses the motif in a more uniform fashion, while LSTM uses the motif initially but soon drifts off to other material.
Revised: Transformer uses the motif in a more uniform fashion, while LSTM uses the motif initially but soon drifts off to other material.

Original: Note that the generated samples are twice as long as the training sequences.
Revised: Note that the generated samples are twice as long as the training sequences.

Original: Relative attention was able to generalize to lengths longer than trained but baseline Transformer deteriorates beyond its training length.
Revised: Relative attention was able to generalize to lengths longer than trained but baseline Transformer deteriorates beyond its training length.

Original: See Appendix C for visualizations of how the our Relative
Revised: See Appendix C for visualizations of how the our Relative

Original: Transformer attends to past motifs.
Revised: Transformer attends to past motifs.

Original: 4.2. H ARMONIZATION : C ONDITIONING ON MELODY
Revised: 4.2. H ARMONIZATION : C ONDITIONING ON MELODY

Original: To explore the sequence-to-sequence setup of Transformers, we experimented with a conditioned generation task where the encoder takes in a given melody and the decoder has to realize the entire performance, i.e. melody plus accompaniment.
Revised: To explore the sequence-to-sequence setup of Transformers, we experimented with a conditioned generation task where the encoder takes in a given melody and the decoder has to realize the entire performance, i.e. melody plus accompaniment.

Original: The melody is encoded as a sequence of tokens as in
Revised: The melody is encoded as a sequence of tokens as in

Original: Waite (2016), quantized to a 100ms grid, while the decoder uses the performance encoding described in Section 3.1 (and further illustrated in A.2).
Revised: Waite (2016), quantized to a 100ms grid, while the decoder uses the performance encoding described in Section 3.1 (and further illustrated in A.2).

Original: We use relative attention on the decoder side and show in Table 4 it also improves performance.
Revised: We use relative attention on the decoder side and show in Table 4 it also improves performance.

Original: [Table]
Revised: [Table]

Original: 4.2. H UMAN EVALUATIONS
Revised: 4.2. H UMAN EVALUATIONS

Original: To compare the perceived sample quality of models trained on the Piano-e-Competition dataset, and their ability to generate a continuation for a priming sequence, we carried out a listening test study comparing the baseline Transformer, our Transformer with relative-attention, PerformanceRNN (LSTM), and the validation set.
Revised: To compare the perceived sample quality of models trained on the Piano-e-Competition dataset, and their ability to generate a continuation for a priming sequence, we carried out a listening test study comparing the baseline Transformer, our Transformer with relative-attention, PerformanceRNN (LSTM), and the validation set.

Original: Participants were presented with two musical excerpts (from two different models that were given the same priming sequence) and asked to rate which one is more musical on a Likert scale.
Revised: Participants were presented with two musical excerpts (from two different models that were given the same priming sequence) and asked to rate which one is more musical on a Likert scale.

Original: For each model, we generated 10 samples each with a different prime, and compared them to three other models, resulting in 60 pairwise comparisons.
Revised: For each model, we generated 10 samples each with a different prime, and compared them to three other models, resulting in 60 pairwise comparisons.

Original: Each pair was rated bydifferent participants, yielding a total of 180 comparisons.
Revised: Each pair was rated bydifferent participants, yielding a total of 180 comparisons.

Original: Figure 5 shows the number of comparisons in which an excerpt from each model was selected as more musical.
Revised: Figure 5 shows the number of comparisons in which an excerpt from each model was selected as more musical.

Original: The improvement in sample quality from using relative attention over the baseline Transformer model was statistically signiﬁcant (See Appendix B for the analysis).
Revised: The improvement in sample quality from using relative attention over the baseline Transformer model was statistically signiﬁcant (see Appendix B for the analysis), both in aggregate and between the pair.

Original:  LSTMs performed better in the study than the Transformer, despite having higher perplexity. This may be due to samples being short (10s to 15s), weakening the comparison on long-term structure.
Revised: Even though in aggregate LSTMs performed better in the study than the Transformer, despite having higher perplexity, but when compared against each other head to head, the results were not statistically signiﬁcant (see Table 5 in Appendix B).

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we demonstrated that the Transformer equipped with relative attention is very wellsuited for generative modeling of symbolic music.
Revised: In this work we demonstrated that the Transformer equipped with relative attention is very wellsuited for generative modeling of symbolic music.

Original: The quality of samples from our model leaves us enthusiastic about this direction of research.
Revised: The quality of samples from our model leaves us enthusiastic about this direction of research.

Original: Moreover, the ability to expand upon a primer, in particular, suggests potential applications as creative tool.
Revised: Moreover, the ability to expand upon a primer, in particular, suggests potential applications as creative tool.

Original: The signiﬁcant improvement from relative attention highlights a shortcoming of the original Transformer that might also limit its performance in other domains.
Revised: The signiﬁcant improvement from relative attention highlights a shortcoming of the original Transformer that might also limit its performance in other domains.

Original: Improving the Transformer’s ability to capture periodicity at various time scales, for instance, or relations between scalar features akin to pitch could improve time-series models.
Revised: Improving the Transformer’s ability to capture periodicity at various time scales, for instance, or relations between scalar features akin to pitch could improve time-series models.

Original: Our memory-efﬁcient implementation enables the application of relative attention to much longer sequences such as long texts or even audio waveforms, which signiﬁcantly broadens the range of problems to which it could be applied.
Revised: Our memory-efﬁcient implementation enables the application of relative attention to much longer sequences such as long texts or even audio waveforms, which signiﬁcantly broadens the range of problems to which it could be applied.

Original: A D OMAIN - SPECIFIC REPRESENTATIONS
Revised: A D OMAIN - SPECIFIC REPRESENTATIONS

Original: Adapting sequence models for music requires making decisions on how to serialize a polyphonic texture.
Revised: Adapting sequence models for music requires making decisions on how to serialize a polyphonic texture.

Original: The data type, whether score or performance, makes certain representations more natural for encoding all the information needed while still resulting in reasonable sequence lengths.
Revised: The data type, whether score or performance, makes certain representations more natural for encoding all the information needed while still resulting in reasonable sequence lengths.

Original: A. S ERIALIZED INSTRUMENT / TIME GRID (J.S.B ACH C HORALES )
Revised: A. S ERIALIZED INSTRUMENT / TIME GRID (J.S.B ACH C HORALES )

Original: The ﬁrst dataset, J.S. Bach Chorales, consists of four-part score-based choral music.
Revised: The ﬁrst dataset, J.S. Bach Chorales, consists of four-part score-based choral music.

Original: The time resolution is sixteenth notes, making it possible to use a serialized grid-like representation.
Revised: The time resolution is sixteenth notes, making it possible to use a serialized grid-like representation.

Original: Figureshows how a pianoroll (left) can be represented as a grid (right), following (Huang et al., 2017).
Revised: Figureshows how a pianoroll (left) can be represented as a grid (right), following (Huang et al., 2017).

Original: The rows show the MIDI pitch number of each of the four voices, from top to bottom being soprano ( S ), alto ( A ), tenor ( T ) and bass ( B ), while the columns is discretized time, advancing in sixteenth notes.
Revised: The rows show the MIDI pitch number of each of the four voices, from top to bottom being soprano ( S ), alto ( A ), tenor ( T ) and bass ( B ), while the columns is discretized time, advancing in sixteenth notes.

Original: Here longer notes such as quarter notes are broken down into multiple repetitions.
Revised: Here longer notes such as quarter notes are broken down into multiple repetitions.

Original: To serialize the grid into a sequence, we interleave the parts by ﬁrst iterating through all the voices at time step 1, and then move to the next column, and then iterate again from top to bottom, and so on.
Revised: To serialize the grid into a sequence, we interleave the parts by ﬁrst iterating through all the voices at time step 1, and then move to the next column, and then iterate again from top to bottom, and so on.

Original: The resulting sequence is S 

Revised: The resulting sequence is S

Original: A 1 T 1 B 1 S 2 A 2 T 2 B 2 ... , where the subscript gives the time step.
Revised: A 1 T 1 B 1 S 2 A 2 T 2 B 2 ... , where the subscript gives the time step.

Original: After serialization, the most common sequence length is 1024.
Revised: After serialization, the most common sequence length is 1024.

Original: Each token is represented as onehot in pitch.
Revised: Each token is represented as onehot in pitch.

Original: [Figure]
Revised: [Figure]

Original: A. MIDI- LIKE EVENT - BASED (P IANO - E -C OMPETITION )
Revised: A. MIDI- LIKE EVENT - BASED (P IANO - E -C OMPETITION )

Original: The second dataset, Piano-e-Competition, consists of polyphonic piano performances with expressive timing and dynamics.
Revised: The second dataset, Piano-e-Competition, consists of polyphonic piano performances with expressive timing and dynamics.

Original: The time resolution here is on the millisecond level, so a grid representation would result in sequences that are too long.
Revised: The time resolution here is on the millisecond level, so a grid representation would result in sequences that are too long.

Original: Instead, the polyphonic performance is serialized into a sequence of one hot encoded events as proposed in (Oore et al., 2018).
Revised: Instead, the polyphonic performance is serialized into a sequence of one hot encoded events as proposed in (Oore et al., 2018).

Original: First, the input MIDI ﬁles are preprocessed to extend note durations based on sustain pedal control events.
Revised: First, the input MIDI ﬁles are preprocessed to extend note durations based on sustain pedal control events.

Original: The sustain pedal is considered to be down whenever a sustain control change is encountered with a value > = 64 ; the sustain pedal is then considered up after a control change with a value < 64 .
Revised: The sustain pedal is considered to be down whenever a sustain control change is encountered with a value > = 64 ; the sustain pedal is then considered up after a control change with a value < 64 .

Original: Within a period where the sustain pedal is down, the duration of each note is extended to either the beginning of the next note of the same pitch or the end of the sustain period, whichever happens ﬁrst.
Revised: Within a period where the sustain pedal is down, the duration of each note is extended to either the beginning of the next note of the same pitch or the end of the sustain period, whichever happens ﬁrst.

Original: If the original duration extends beyond the time when the sustain pedal is down, that original duration is used.
Revised: If the original duration extends beyond the time when the sustain pedal is down, that original duration is used.

Original: Next, the MIDI note events are converted into a sequence from the following set of vocabulary:
Revised: Next, the MIDI note events are converted into a sequence from the following set of vocabulary:

Original: NOTE_ON events for starting a note of with one of the 128 MIDI pitches, 128 NOTE_OFF events for ending a note with one of the 128 MIDI pitches, 100 TIME_SHIFT events representing forward time shifts in 10ms increments from 10ms to 1s, and 32 SET_VELOCITY events representing the velocity for future NOTE_ON events in the form of the 128 possible MIDI velocities quantized into 32 bins.
Revised: NOTE_ON events for starting a note of with one of the 128 MIDI pitches, 128 NOTE_OFF events for ending a note with one of the 128 MIDI pitches, 100 TIME_SHIFT events representing forward time shifts in 10ms increments from 10ms to 1s, and 32 SET_VELOCITY events representing the velocity for future NOTE_ON events in the form of the 128 possible MIDI velocities quantized into 32 bins.

Original: An example performance encoding is illustrated in Figure 7.
Revised: An example performance encoding is illustrated in Figure 7.

Original: [Figure]
Revised: [Figure]

Original: B S UPPLEMENT OF LISTENING TEST B. S TUDY PROCEDURE
Revised: B S UPPLEMENT OF LISTENING TEST B. S TUDY PROCEDURE

Original: Participants were presented with two musical excerpts that shared a common priming sequence.
Revised: Participants were presented with two musical excerpts that shared a common priming sequence.

Original: For each excerpt, the priming sequence was played, followed by 2.5 seconds of silence, followed by the priming sequence again and a continuation of that sequence.
Revised: For each excerpt, the priming sequence was played, followed by 2.5 seconds of silence, followed by the priming sequence again and a continuation of that sequence.

Original: The continuations were either sampled from one of the models or extracted from our validation set.
Revised: The continuations were either sampled from one of the models or extracted from our validation set.

Original: We evaluated all possible pairs in the space of data and model samples, except from the same model.
Revised: We evaluated all possible pairs in the space of data and model samples, except from the same model.

Original: Each continuation had a length ofevents using the encoding described in Section A.2.
Revised: Each continuation had a length ofevents using the encoding described in Section A.2.

Original: This corresponds to the length the models were trained on to remove the deteriorating effect that happens with baseline Transformer when asked to generate beyond the length it was trained on.
Revised: This corresponds to the length the models were trained on to remove the deteriorating effect that happens with baseline Transformer when asked to generate beyond the length it was trained on.

Original: Participants were asked which excerpt they thought was more musical on a Likert scale.
Revised: Participants were asked which excerpt they thought was more musical on a Likert scale of 1 to 5.

Revised: The pair is laid out left versus right, with 1 indicating the left is much more musical, 2 the left is slightly more musical, 3 being a tie, 4 being the right is slightly more musical, and 5 the right is much more musical.

Original: 180 such ratings were collected, with each source involved inpair-wise comparisons and each comparison completed by 3 different participants.
Revised: For each model, we generated 10 samples each with a different prime, and compared them to three other models, resulting in 60 pairwise comparisons. Each pair was rated by 3 different participants, yielding a total of 180 comparisons.

Original: B. A NALYSIS
Revised: B. A NALYSIS

Original: A Kruskal-Wallis H test of the ratings showed that there was a statistically signiﬁcant difference between the models: χ 2 (2) = 63 .
Revised: A Kruskal-Wallis H test of the ratings showed that there was a statistically signiﬁcant difference between the models: χ 2 (2) = 63 .

Original: 84 , p = 8 . 86 e-14 < 0 . 01 .
Revised: 84 , p = 8 . 86 e-14 < 0 . 01 .

Original: Table 5 show a post-hoc analysis on the comparisons within each pair, using the Wilcoxon signed-rank test for matched samples.
Revised: Table 5 show a post-hoc analysis on the comparisons within each pair, using the Wilcoxon signed-rank test for matched samples.

Original: Tableshows a post-hoc analysis of how well each model performed when compared to all pairs, and compares each model’s aggregate against each other, using the Mann–Whitney U test for independent samples.
Revised: Tableshows a post-hoc analysis of how well each model performed when compared to all pairs, and compares each model’s aggregate against each other, using the Mann–Whitney U test for independent samples.

Original: We use a Bonferroni correction on both to correct for multiple comparisons.
Revised: We use a Bonferroni correction on both to correct for multiple comparisons.

Revised: The win and loss counts bucket scores 4, 5 and scores 1, 2 respectively, while the tieing score is 3.

Original: Both within the pair and in aggregates, participants rated samples from our relative Transformer as more musical thansamples from the baseline Transformer with p < 0 .
Revised: Both within pairs and between aggregates, participants rated samples from our relative Transformer as more musical than the baseline Transformer with p < 0 .

Original: We did not observe a consistent statistically signiﬁcant difference between the other model pairs, baseline transformer versus LSTM and LSTM versus relative Transformer.
Revised: For within pairs, we did not observe a consistent statistically signiﬁcant difference between the other model pairs, baseline transformer versus LSTM and LSTM versus relative Transformer.

Original: When looking at the aggregates, relative
Revised: When comparing between aggregates, LSTM was overall perceived as more musical than baseline Transformer.

Original:  Transformer came  close to outperforming LSTM with p = 0 .
Revised: Relative Transformer came a bit close to outperforming LSTM with p = 0 .

Original: When we listen to the samples  the two are very different, with relative Transformer bearing much more long-term structure, but the effects were probably less pronounced in the listening test because we used samples around 10s to 15s, which is half the length of those shown in Figure 4 to prevent the baseline Transformer from deteriorating.
Revised: When we listen to the samples from the two, they do sound qualitatively different. Relative Transformer often exhibits much more structure (as shown in Figure 4), but the effects were probably less pronounced in the listening test because we used samples around 10s to 15s, which is half the length of those shown in Figure 4 to prevent the baseline Transformer from deteriorating.

Original: This weakens the comparison on long-term structure.
Revised: This weakens the comparison on long-term structure.

Original: When compared to real music from the validation test, we see that in aggregates, real music was better than LSTM and baseline Transformer.
Revised: When compared to real music from the validation set, we see that in aggregates, real music was better than LSTM and baseline Transformer.

Original: There was no statistical signiﬁcant difference between real music and relative Transformer.
Revised: There was no statistical signiﬁcant difference between real music and relative Transformer.

Original: This is probably again due to the samples being too short as real music is deﬁnitely still better.
Revised: This is probably again due to the samples being too short as real music is deﬁnitely still better.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C V ISUALIZING SOFTMAX ATTENTION
Revised: C V ISUALIZING SOFTMAX ATTENTION

Original: One advantage of attention-based models is that we can visualize its attention distribution 3.
Revised: One advantage of attention-based models is that we can visualize its attention distribution 3.

Original: This gives us a glimpse of how the model might be building up recurring structures and how far it is attending back.
Revised: This gives us a glimpse of how the model might be building up recurring structures and how far it is attending back.

Original: The pianorolls in the visualizations below is a sample generated from Transformer with relative attention.
Revised: The pianorolls in the visualizations below is a sample generated from Transformer with relative attention.

Original: Each ﬁgure shows a query (the source of all the attention lines) and previous memories being attended to (the notes that are receiving more softmax probabiliy is highlighted in).
Revised: Each ﬁgure shows a query (the source of all the attention lines) and previous memories being attended to (the notes that are receiving more softmax probabiliy is highlighted in).

Original: The coloring of the attention lines correspond to different heads and the width to the weight of the softmax probability.
Revised: The coloring of the attention lines correspond to different heads and the width to the weight of the softmax probability.

Original: D P REVIOUS FIGURES FOR THE “ SKEWING ” PROCEDURE
Revised: D P REVIOUS FIGURES FOR THE “ SKEWING ” PROCEDURE

Original: [Figure] [Figure]
Revised: [Figure] [Figure]