Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation
Abstract
Music sequences are typically very long, e.g., with over 10 , 000 tokens, makingthem hard to model by Transformer due to its quadratic complexity of self-attention.
Although there are many Transformer variants aiming at modeling long sequencesin natural language processing, directly using them in music generation is suboptimal, because music has both short-term structures and long-term structures, whichis very different from text.
In this paper, we propose Museformer, a Transformerwith a novel fine- and coarse-grained attention for symbolic music generation.
Specifically, with fine-grained attention, a token of a specific bar directly attends toall the tokens of the bars that are most relevant in regard to music structures (e.g.,the previous 1st, 2nd, 4th and 8th bars, selected based on musical knowledge); withcoarse-grained attention, a token only attends to the summarization of the otherbars rather than each token of them so as to reduce computational cost.
The advantages of our model are two-fold.
First, it can capture both music structure-relatedcorrelations via the fine-grained attention, and other contextual information viathe coarse-grained attention.
Second, it is efficient and can well model over 4 ×longer music sequences compared to its full-attention counterpart.
Both objectiveand subjective experimental results demonstrate its ability to generate long musicsequences with high quality and better structures.
Our generated music samplescan be found at https://museformer.github.io/ .
Introduction
Symbolic music generation aims at generating music scores automatically and has drawn more andmore attention in recent years [1–3].
Since music can be represented in organized sequences ofdiscrete tokens just like text, Transformer-based models, which have demonstrated to work wellon text generation, are increasingly applied in music generation [3–9].
While the self-attentionmechanism empowers Transformer to directly capture the correlations over the whole input sequence,the quadratic complexity limits its scalability to music sequences especially for the multi-instrumentpolyphonic music, as they are typically very long, e.g., over 10 , 000 tokens.
Although many Transformer variants in natural language processing (NLP) have been proposed tohandle long sequences [10–12], directly applying them in music generation is suboptimal, becausein addition to being very long, music has its unique bar-aware structures that are very different fromtext.
Simply summarized, the structures include both the short-term structures and the long-termstructures [13].
The former one means that adjacent bars usually follow the same rules, such as thesame rhythmic patterns or chord progression; the latter one is mainly embodied in the long-distancerepetitions, where a bar is a repetition of a previous bar at a long distance.
Figure 1 shows an exampleof the song Twinkle, Twinkle, Little Star .
As can be seen, every two consecutive bars repeat the samerhythmic pattern, which is a short-term structure, and at the 9th bar, the song starts to repeat themelodic pattern at the beginning, which is a long-term structure.
In this paper, we call these bars that
[Figure]
Pno.
Piano
Synth. Synth.
[Equation]
[Figure]
  are the most relevant to the current bar in structure as the structure-related bars .
Obviously, directlyreferencing these bars is essential for music generation, only by doing which the model can possiblygenerate structured music pieces.
Note that the structure-related bars can be either close or far fromeach other, which obviously violates the common assumption in NLP that the most recent content isthe most relevant.
Thus, Transformers in NLP may not perform well on modeling music sequences,especially for capturing the long-term structures.
To better model the long music sequence in regard to the music structures, we propose Museformer, along-sequence Transformer with a novel fine- and coarse-grained attention (FC-Attention in short)for symbolic music generation.
It respects the fact that bar is an important unit for music, and all theattention operations are conducted with respect to the bar ranges.
For a specific bar, we select out theprevious structure-related bars , and leverage two different attention schemes namely fine-grainedattention and coarse-grained attention on the structure-related bars and the other bars, respectively.
Specifically, with fine-grained attention, a token of a specific bar directly attends to all the tokens ofthe structure-related bars, for better learning the structure-related correlations.
With coarse-grainedattention, the token attends to only the summarization of the other bars rather than each token of them,to decrease the computation and space complexity while also remain the necessary and concentratedinformation of these bars.
Note that in Museformer, the structure-related bars mentioned above are not necessarily the mostrecent adjacent bars but can also include some distant ones, to cater to the long-term structuresof music.
These bars are selected based on musical knowledge, and include both the short-termstructure-related bars (e.g., the previous 1st and 2nd bars) and the long-term structure-related bars(basically they are 4 bars or a multiple of 4 bars away from the current, e.g., the previous 4th, 8thbars).
Experimental results show that the proposed selection of the structure-related bars can helpdecrease the perplexity and generate music that exhibits better structures.
The main contributions of this work are summarized as follows:
• We propose Museformer, a Transformer model with a fine- and coarse-grained attentionfor music generation.
It captures the correlations of structure-related bars via fine-grained
attention for better learning the music structures, as well as the necessary and concentratedinformation of the other bars through their summarization via coarse-grained attention.
• We carefully select the structure-related bars based on musical knowledge, which cover boththe short-term structures and the long-term structures.
• The computation complexity and space complexity are reduced from quadratic to linear
with respect to the sequence length.
This enables Museformer to scale up to long musicsequences.•
Experimental results show that Museformer can generate full-song level music with highquality and better structures.
The generated music samples can be found at https://museformer.github.io/ .
Source code is currently uploaded in the supplement and willbe released on GitHub.
Related Work2.1 Symbolic Music Generation
Symbolic music generation uses machines to compose music scores automatically.
This fascinatingidea has been attracting more and more people to work on it, and the solutions continuously evolvefrom rules [1] or probability models [14, 15, 2] to deep learning methods [4–9].
In recent years, the Transformer-based models have achieved great success in many text generationtasks, thus have also been increasingly applied in the similar music generation tasks.
Huang et al.[3] apply Transformer in generating music for the first time, and show that it can achieve the stateof-the-art performance.
While Transformer shows promising results in music generation as it doesin other sequential data processing tasks, it suffers from the quadratic complexity, which limits theapplications on the typically long music sequences.
To resolve this problem, works on symbolic musicgeneration come up with different solutions.
One popular solution is to modify the representationmethod that transfers the symbolic music file (e.g., the MIDI format) to the sequential tokens, such as
REMI in [16] and compound word in [17].
While the new representation methods indeed decreasethe sequence length of music to certain extent, Transformer still cannot handle the overly long musicsequences.
Another solution is to use a long-sequence Transformer variant as the backbone.
Forexample, Huang and Yang [16] leverage Transformer-XL [10], and [17] leverages Linear Transformerwith a kernel-based linearized attention.
We will review more long-sequence Transformer in §2.2.
Although these models can process long sequences, they are proposed in regard to the NLP problems.
In view of the fact that music has its unique structures different from text, directly applying them inmusic generation is suboptimal, as we have mentioned in §1 and will give a more extensive discussionin the following section.
The existing Transformer-based models for music generation directly borrowthe long-sequence Transformers from NLP to generate long music sequence, without consideringmusic’s characteristics.
To the best of our knowledge, we are the first to propose a Transformervariant that are targeted at symbolic music generation.
2.2 Long-Sequence Transformers
[Figure]
Various kinds of Transformer variants in NLP have been proposed to tackle the document-level tasks.
One famous kind is the recurrent Transformers, which chunk the sequence into blocks, and encodeeach block with the gradient-stopped memory of several previous blocks [10, 18, 19].
Another onecalled sparse attention is to sparsify the attention matrix with predefined patterns such as slidingwindows, or with techniques such as hashing functions [11, 20, 21].
Some other models use linearizedattention, which approximates the pairwise softmax-based attention with kernel-based computationin linear time complexity [12, 22].
The existing works on music generation directly adopt some of those long-sequence Transformers toprocess long music sequences, but we argue that it is optimal due to the different structures of musiccompared to text.
A common and reasonable assumption in NLP is that the nearest content is the mostrelevant, but it is obviously not true for the music structures.
As we have shown in §1 that besides theshort-term structures, there are many long-term structures in music.
Models only focusing on the mostrecent content, such as the recurrent or the sliding window-based Transformers, can be able to capturethe short-term structures but may not well capture the long-term structures.
As for the linearizedattention whose attention ranges can cover the whole input, it only approximates the attention weightson the whole sequence and does not model the structure-related bars in a fine-grained and targetedway, which harms the performance, as we will show in the experiments.
Museformer
In order to handle the typically long music sequences, as well as capture both the short- and long-termmusic structures, we propose Museformer for symbolic music generation.
It basically follows theoriginal Transformer structure [23].
The core component of Museformer is the fine- and coarsegrained attention (FC-Attention), which replaces the original self-attention module to tackle thechallenges in long music sequence modeling.
Museformer takes the music tokens as input.
For each bar in the input, we insert a summary tokenafter it to aggregate its local information.
The embedding layer embeds the overall token sequence barbarbarx 1, x 1, x 1, x 2, x 2, x 3, x 3, x 1, x 1, x 1, x 2, x 2, x 3, x 3, sssss 3 ssss 3 …… …… ……
barbarbar 3 x 1,1 x 1,2 x
1,3 s 1 x 2,1 x 2,2 s 2 x 3,1 x 3,2 sb a r 1 x 1 , 1 x 1 ,x 1 ,b a r 2 x 2 , 1 x 2 ,b a r 3 x 3 , 1 x 3 ,b a r 1 sb a r 2 sb a r 3 s(b)
The corresponding attention layouts.
into a vector space, and concatenates the bar embedding and the beat embedding as the positioninformation for music, followed by a linear projection.
The following Museformer layers model thecontextual representation, and the hidden state output by the last layer is fed into the output layer topredict the next token.
We first introduce the necessary preliminaries and notations in §3.1, then describe the details of
FC-Attention in §3.2, and finally discuss the merits of our model in §3.
3.1 Preliminaries and
Notations
Attention
The attention mechanism [23] is the basis of FC-Attention.
It receives two sequentialinputs, namely target X ∈ R n tgt × d and source X
′ ∈
R n src × d , where n tgt and n src are the sequencelengths of target and source respectively, d is the embedding dimension.
It computes contextualrepresentation for each item x i ∈ R 1 × d of target X as
[Equation]
Here, we formulate attention with respect to a single item x i , because in our model, we assigndifferent X ′ for different x i ∈ X .
We use the multi-head version in our implementation, but omit itsequations for simplicity.
Token Notations
Museformer receives the music token sequence as input, denoted by X =( X 1 , . . .
, X b ) , where b is the number of music bars, and each bar X i =
( x i, 1 , . . .
, x i, | X i | ) contains| X i | tokens.
We then insert a summary token after each bar to summarize its local information.
Thesummary token for the i -th bar is denoted by s i , and the summary token sequence is denoted as
S =
( s 1 , · · · , s b ) .
3.2 Fine- and Coarse-Grained Attention
In this section, we describe the proposed FC-Attention in detail.
The basic idea of FC-Attention isthat, instead of attending to all the tokens of the sequence which causes the quadratic complexity,a token of a specific bar only directly attends to the structure-related bars that are essential forgenerating structured music (fine-grained attention), and for the other bars, the token only attendsto their summary tokens to obtain concentrated information (coarse-grained attention).
To achievethis, we need to summarize the local information of each bar through a summarization step, anddecide which bars to be attended via fine-grained attention through structure-related bar selection ,and finally aggregate the fine-grained and coarse-grained information through a combination process.
Summarization
In the summarization step, we aggregate the information [Figure] qu [Figure]
n
[Figure] and that of the music tokens X i = [ x i, 1 , . . . , x i, | X i | ] ∈ R | X i |× d , the summarization of this bar is
[Equation]
[Equation]
) where Attn ( · ) is defined in Equation 1, [ · ] is the concatenation operation.
Each summary token attendsto music tokens that belong to its corresponding bar as well as the summary token itself.
Structure-Related Bar Selection
One important issue is that what are the structure-related bars.
We conduct a statistical analysis on the POP909 dataset[24], where the segments of a song areannotated, through which we can easily calculate the distribution of the repetition intervals, i.e., thedistance between two repetitive bars.
As shown in Figure 3, music tends to repeat every 4 bars or itsmultipliers in most cases, and this also complies with our musical expertise and experience.
Based onthis musical knowledge, we select 8 bars as the structured-related bars, which include the previous 1st,2nd, 4th, 8th, 16th, 24th and 32nd bars, where the 1st and the 2nd bar are for capturing the short-termstructures while the others are for capturing the long-term structures.
Combination
In the combination step, we aggregate the information of the tokens belonging tothe structure-related bars or within the current bar, as well as the summarization of the other bars, soas to update the contextual representation of music tokens.
The updated representation for x i,j is
[Equation]
[Equation] In this equation, N ( i ) is the set of the indices of the previous structure-related bars with respectto the i -th bar, and ¯ N ( i ) is the set of the indices of other previous bars.
X N ( i ) corresponds to thematrix formed by stacking { X i | i ∈ N ( i ) } , and similarly ˜ S
¯ N ( i ) corresponds to the matrix formedby stacking { ˜ s i | i ∈ ¯
N ( i ) } .
Finally, X i,k ≤ j corresponds to the matrix that stacks { x i,k
| x i,k ∈
X i , k ≤ j } .
In other words, the music token only attends to those music tokens that are in thestructure-related bars, or its previous tokens within the current bar.
For other bars, it only attends totheir summary tokens.
3.3 Merits of Museformer
We explain how the designs of FC-Attention make Museformer more suitable for the music generationtasks compared to the existing Transformers.
First, the structure-related bar selection ensures that FC-Attention can capture both the short-term andthe long-term structures of music.
Compared to most long-sequence Transformers that only focus onthe most recent content (e.g., recurrent Transformers and sliding window attention), the selected barsinclude both the neighboring ones to capture the short-term structures and the distant ones to capturethe long-term structures.
Secondly, the combination of the fine-grained attention and the coarse-grained attention makes itcapable of concentrating on the structure-related bars, and also remaining the necessary informationof
the other bars through the summarization.
In contrast, sparse attention models leverage stronginductive biases to make each specific token only attend to a subset of tokens and simply drop theothers, which may loses necessary information of the whole sequence.
Thirdly, Museformer is efficient for handling long music sequences.
A music token of a specific baronly directly attends to those music tokens in the structure-related bars, and indirectly attends to theother bars via the summary tokens.
Compared to the number of all the music tokens, the numberof summary tokens is much smaller, which drastically decreases the time complexity and memoryconsumption.
Experiments4.1 Experiment Settings
In this subsection, we give a brief introduction to the experiment settings.
Please refer to Appendix Afor more details.
Dataset and Music Representation
We select the widely-used Lakh MIDI (LMD) dataset
[25],which is a multi-instrument dataset.
We preprocess it and obtain the final dataset for the experiments,which contains 29 , 940 songs ( 1 , 727 hours), each song contains 95 bars on average, and each bar isrepresented by 158 tokens on average.
We randomly split them by 8 / 1 / 1 for training/validation/test,respectively.
To represent symbolic music as token sequences, we use a REMI-like representationmethod [16] for its simplicity.
Figure 1 shows an example.
Implementation
We implement our model with PyTorch 1 and fairseq 2 .
For efficiently computing
FC-Attention, we transfer the attention layouts into the blocksparse forms and compute them with
OpenAI Triton 3 .
We also write CUDA kernels to generate the attention layouts efficiently.
Model and Training Configurations
The core parameters of our model include layer number = 4 ,hidden size = 512 , number of attention heads = 8 , and FFN hidden size = 2 , 048 .
During training,the batch size is set to 4 songs.
Following [23], we use the Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
98and ε = 10 − 9 , and the learning rate is warmed up over the first 16 , 000 steps linearly to a peak valueof 0 .
00005 , and then decreases based on the inverse square root of the steps.
The L2 weight decayis set to 0 .
During inference, we use the top- k sampling with k = 8 , and generate the sequencetokens continuously until the end of sentence token is generated
Baselines
We select four typical and widely-used baseline models, most of which have beenadopted in music generation:
• Music Transformer [3] is a vanilla Transformer with a memory-efficient “skewing” relative position embedding implementation.
• Transformer-XL [10] is a recurrent Transformer that encodes the sequence chunk by chunk,using the previous chunks as the memory to carry the past information. • Longformer [11] leverages a sliding window sparse attention to encode the local information.• Linear Transformer [12] uses a kernel-based attention to decrease the computation com- plexity and memory consumption.
All the baseline models are set with comparable hyper-parameters as Museformer.
For Music
Transformer that adopts a full attention and thus cannot process long sequences due to the memorylimit, we chunk the input into multiple samples during training, while allows it to generate longersequences during inference, to test the generalization on long music sequences.
Objective Evaluation
In the objective evaluation, we use Perplexity (PPL) to evaluate theperformance, which is a very common metric in the generation tasks to measure whether a model cancorrectly predict the tokens.
The lower the value is, the better the model can correctly predict.
In ourexperiment, perplexity is calculated on the test set and over the first 1 , 024 , 5 , 120 , or 10 , 240 tokensof each sample.
Subjective Evaluation
The most canonical way to evaluate a music generation model is thehuman listening test.
We apply each model to randomly generate 100 music pieces, and invite {+
 [Table] [Table] people,+} where 7 of them have some music-related learning experiences, to score these music pieces.
Specifically, for each participant, we randomly construct 5 groups, where each group contains 5 musicpieces that are generated by Museformer and the 4 baseline models
[Table]
respectively. Participants areasked to score these music pieces from 1 (lowest) to 10 (highest) over the following subjective metrics: 1) Musicality : whether the music is pleasant and interesting, and is real enough just like human-made music.
2) Short-term structure : whether the music shows good structure in neighboring content, such as good repetitions and reasonable development.
3) Long-term structure : whether the music shows good structure in long distance, such as song-level repetitions and long-distance connections.
Overall : an overall score.
We also calculate the preference score based on the overall score,which is defined as the ratio of the winning times.
4.2 Comparison with Previous Models
Table 1 shows the results of the objective evaluation.
As we can see: 1) Music Transformer achievescomparable performance with other models on generating short music pieces ( 1 , 024 tokens), butundergoes a severe declining when moving to longer ones, which implies that a model trained on shortmusic sequences cannot well generalize to long music sequences.
Accordingly, an appropriate longsequence Transformer model is crucial in music generation tasks.
Although Linear Transformerdrastically decreases the complexity, its performance is relatively poor compared to other longsequence Transformers on the music generation task, since it loses accurate information of both theshort- and long-term structure-related bars.
The proposed Museformer can consistently achievethe best results on various sequence lengths, especially on generating long music sequences, whichdemonstrates the effectiveness of Museformer on music generation tasks.
Furthermore, we present the results of the subjective evaluation in Table 2 and Figure 4, which showthat Museformer achieves the best performance compared to all the best models.
Specifically, 1)
Museformer gets the highest scores on all the metrics, and the improvement is statistically significant( p ≤ 0 .
05 , see Appendix C for the details).
On the structure metrics, especially the long-termstructure, Museformer exceeds other models by a large gap, indicating that the proposed FC-Attentioncan empower the model to capture the correlations in distant bars.
4.3 Ablation Study
We verify the effectiveness of the Museformer components as follows: 1) To show the effectiveness ofthe coarse-grained attention, we compare Museformer with the setting w/o coarse-grained attention ,
Museformer
Linear Transformer
Longformer
Transformer-XL
Music Transformer
Preference score
[Table]
where each music token only directly attends to the music tokens in the structure-related bars.
2) Toshow the effectiveness of the structure-related bar selection for the fine-grained attention, we compare
Museformer with the setting w/o structure-related bar selection , where we directly use the mostrecent 8 bars for the fine-grained attention instead of the selected structure-related bars.
The results are presented in Table 3.
Museformer consistently achieves better performance withcoarse-grained attention and structured-related bar selection.
The contribution of these two settingsgrows when the sequence length is getting longer, and this is reasonable because: 1)
The coarsegrained attention helps Museformer obtain summarization of more previous bars, if the sequence islong and there are more bars in the sequence.
There exist more structures in a long music sequence,where the selected bars can help capture the correlations.
4.4 Complexity Analysis
We further analyze the complexity of Museformer.
Suppose the sequence length is n , the average barnumber is b , the average bar length is m , and the number of selected structure-related bars is k .
For
FC-Attention, the time complexity of the summarization step is O ( n ) , and that of the update stepis O (( km + b ) n ) , so the overall complexity of FC-Attention is O (( km + b ) n ) .
Since k and m aresmall numbers, and b is quite small in the real world, the complexity of FC-Attention is linear to thenumber of tokens.
With the decreased complexity, Museformer can encode much longer sequence than the full attention,allowing us to generate full-song music.
We train Museformer and vanilla Transformer on thevalidation set to test the memory consumption and the running time.
With batch size = 1 , we increasethe maximum sequence length until we use up the 32 GB of memory on a V100 GPU, and record thepeak memory consumption 4 at each length and the running time for training one epoch.
Figure 5shows the results.
As we can observe, the memory consumption of Museformer increases linearlywith respect to the maximum sequence length, and can process over 4 × longer music sequences thanvanilla Transformer, making it feasible in generating full-song music.
Museformer also achievesfaster training times when the maximum sequence length is greater than around 5 , 000 , which is acommon length for music sequences.
Note that, after the maximum sequence length exceeds 32 , 768 ,the memory consumption of Museformer does not increase as fast as in the previous range, and this
[Figure]
may because there is only a few samples in the dataset that are longer than 30 , 000 , and the memoryconsumption is also related to the lengths of the bars.
4.5 Case Study
We present the piano track of a generated music piece in Figure 6, which exhibits the reasonableshort-term structure and long-term structure.
As we can see, the 21st bar repeats the 19th bar, whichis a short-term structure commonly existing in music.
Moreover, the 35th–38th bars repeat the19th–22nd bars, and the interval is large enough (16 bars) to make it a song-level long-term {+structure.
+}
[Figure]
[Figure]
     
           
[Equation]
                                                                                 
    
                                             =   
Conclusion
Music generation is an interesting and challenging problem.
In this paper, we propose Museformer totackle the long-sequence music generation task with a carefully designed fine- and coarse-grainedattention.
The fine-grained attention is applied to selected structure-related bars, which can successfully capture both the short- and long-term music structures, while the coarse-grained attentioncan get a sketch of all the other bars based on the summary tokens.
Experimental results show thatthe proposed Museformer is superior than previous music generation models and long-sequencetransformer models, especially on generating music with song-level structure.
The limitation of Museformer is that, the efficiency of the dynamic sparse attention largely dependson a very sophisticated implementation, and the running speed and the memory consumption stillhave large room for improvement compared to the theoretical complexity.
In future work, we willexplore the possibility of the further efficiency improvement, and apply the FC-Attention to the musicunderstanding tasks.
Checklist
For all authors
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and
scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We do
not see any potential negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB)