Byte Pair Encoding for Symbolic Music (2023)

Abstract: When used with deep learning, the symbolic music modality is often coupled with language model architectures. To do so, the music needs to be tokenized, i.e. converted into a sequence of discrete tokens. This can be achieved by different approaches, as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed tokenizations rely on small vocabularies of tokens describing the note attributes and time events, resulting in fairly long token sequences, and a sub-optimal use of the embedding space of language models. Recent research has put efforts on reducing the overall sequence length by merging embeddings or combining tokens. In this paper, we show that Byte Pair Encoding, a compression technique widely used for natural language, significantly decreases the sequence length while increasing the vocabulary size. By doing so, we leverage the embedding capabilities of such models with more expressive tokens, resulting in both better results and faster inference in generation and classification tasks. The [source code is shared on Github](https://github.com/Natooz/bpe-symbolic-music), along with a [companion website](https://Natooz.github.io/BPE-Symbolic-Music). Finally, BPE is directly implemented in [MidiTok](https://github.com/Natooz/MidiTok), allowing the reader to easily benefit from this method.

## introduction when used with deep learning, the symbolic music modality is mostly represented as discrete and used with language models (lm) such as transformers @xcite . these models receive sequences of tokens as input, convert them to learned embedding vectors representing their semantic features in a continuous space, and process these embeddings for the task at hand. a token is a distinct element, known within a finite vocabulary. for natural language, a token can be a word, subword or punctuation mark. for symbolic music, tokens usually represent note attributes or time events, such as pitch or duration. tokenizing music, i.e., converting raw data into tokens, can be achieved by several ways, as music can be composed of simultaneous tracks, of simultaneous notes with several attributes such as their pitch and duration. multiple approaches exist to represent these features. recently, the token representation of symbolic music has been studied @xcite , with the goal to improve 1) the results, e.g. the quality of generated music or the accuracy of classification tasks, and; 2) the efficiency of the models. the former is tackled with more expressive representations @xcite @xcite , and the latter by representations based on either token combinations @xcite , or embedding pooling @xcite @xcite , which reduce the overall sequence length. still, these tokenizations are based on tokens only representing the values of time events and note attributes. this comes with a big limitation: these tokens do not carry much information by themselves. we can assume that their embeddings does not either. by analogy to natural language, these tokens are closer to the character level than word level. yet, a powerful feature of lms is their ability to learn (embedding) representations of discrete elements such as tokens, and leverage this information for reasoning and downstream tasks. for natural language, lms are usually coupled with vocabularies containing up to 50k tokens, represented on a few hundreds dimensions (often between 512 and 2048). using a vocabulary containing fewer tokens than the number of dimensions used to represent them appears as a suboptimal usage of such models. moreover, the expressive information carried by music is deduced from the combinations of its notes and their attributes. considering the infinite possible music arrangements, we can suppose that using solely note attribute embeddings imposes to lms a heavier modeling effort than embeddings of potential whole note successions that would be more expressive and explicit. in this paper, we show that byte pair encoding (bpe, described in section 3) applied to symbolic music allows to address the two goals just mentioned, while outperforming the previous methods and making the model learn better distributed embeddings. to the best of our knowledge, bpe has yet not been studied for the symbolic music modality, although it can be applied on top of any music tokenization that does not perform embedding pooling. this work aims at closing this gap by shedding light on the results and performance gains of using bpe: ## byte pair encoding byte pair encoding (bpe) @xcite is a data compression technique. it converts the most recurrent successive bytes in a corpus into newly created ones. for instance, in the character sequence aabaabaacaa, the sub-sequence aa occurs three times and is the most recurrent one. learning and applying bpe on this sequence would replace aa with a new symbol, e.g., d, resulting in a compressed sequence dbdbdcd. the latter can be reduced again by replacing the db subsequence, giving eedcd. in practice bpe is learned on a corpus until the vocabulary reaches a target size. bpe learning is described by the pseudo-code of algorithm 1. ## bpe learning bpe allows to significantly reduce the sequence length. as shown in figure 2 , the ratio of average number tokens representing a beat can be reduced up to more than 50%. as bpe replaces recurrent pair of bytes in the data, the average number of byte combinations of the vocabulary tends to first quickly increase, then more slowly grow. the inverse tendency can be observed on the tokens per beat ratios shown in appendix e complements this analysis by shedding light on the types of the underlying tokens represented by the newly learned tokens. ## music generation music generation is a popular application of deep learning models @xcite . we ought to experiment on this task to demonstrate the benefits of bpe on music modeling. for this task, we choose to use the maestro dataset @xcite , which is made of 1k pairs of audio and midi files of classical piano performances. each midi file is made of one piano track, with dynamic melodies and complex harmonies. we generate autoregressively the next 512 tokens of input prompts from the test subsets of the gi-antmidi dataset, filtering the logits by keeping the top @xmath0, 95 probability mass (nucleus sampling @xcite ) and top 15 token probabilities (top-k sampling @xcite ). evaluation of symbolic music is still an open issue @xcite . in the absence of automatic metrics measuring the distances between subsets of data, most works evaluate generated results with human surveys along with feature similarity metrics. the latter however cannot capture the quality of music, and is subject to irregularities in case of model over or underfitting. we decide here to replace them with an metric measuring the errors of prediction of the models. ## classification for our classification task, we experiment with the mmd dataset @xcite . it is, to our knowledge, the biggest midi dataset publicly available. it features more than 430k midi files of all genres of music with multiple tracks. each piece is matched to spotify and musicbrainz ids, allowing to link them with a wide variety of information such as artist or music genre. in order to get a more quality training corpus, we perform a preprocessing step which deduplicates the files of the same music and keeps only the best. this is explained in appendix b. we also merged the tracks of the instruments of the same class in order to reduce the overall complexity (appendix c). to handle multiple tracks, we placed program tokens before each pitch token of each note to specify its instrument. this strategy is similar to @xcite . we perform genre and artist classification, from the 40 and 100 most present genres and artist in the mmd dataset. the results, reported in it concurs with our results, and is explored in the next section by analyzing the geometry of the learned embeddings. ## learned embedding spaces we have shown so far that bpe improves the results of music modeling on the generation and classification tasks. our assumption is that, non-only the reduced sequence length allows to pack more information (longer music piece) within the same number of tokens, but mostly the vocabulary can be scaled while making the model efficiently learn and use the embedding representations of the newly created tokens with bpe. the embedding space, i.e. the way lms learn to represent tokens into a continuous space r d of d dimensions, has recently been studied @xcite @xcite . more specifically, it has been shown that most lms learn anisotropic embedding distributions @xcite , despite that their isotropy have been linked to improved performances on downstream tasks @xcite @xcite @xcite . isotropy is a measure of the uniformity of the space occupied by a manifold across all dimensions. a high isotropy is associated with a uniform variance of the distances between the points of the manifold across all dimensions. in our case the manifold is the collection of contextually learned embeddings @xmath1. an isotropic embedding space can be viewed as a space where the embeddings are uniformly spaced with uniform densities. the intrinsic dimension is an estimation of the minimal number of dimensions n required to represent a manifold in r d , @xmath2. it links with isotropy in the sense that an isotropic manifold occupies all the dimensions, hence its intrinsic dimension is close to d. we chose to estimate it with the principle component analysis (pca) @xcite and fishers @xcite methods as they are insensitive to redundancy, focus on common similarities and can scale to large number of points and dimensions. as the embedding matrix is often initialized with a stochastic method around the origin, a simple method can estimate high intrinsic dimensions even though the points coordinates have been very little or not even optimized. this can be the case when a large number of tokens has low frequencies or are absent from the training data, such as with pvdm. the intrinsic dimensions results are reported in section 8, along with the isoscores. in all cases, as the vocabulary grows with bpe, the intrinsic dimension increases, the embeddings occupy more space. isoscore is an estimation of isotropy based on the distance of the covariance matrix of a principle component analysis (pca) and the identity matrix, and is normalized between 0 and 1. as for the intrinsic dimension, the isoscore grows with the vocabulary size, indicating that the embeddings are more uniformly distributed. we also note that similarly to models trained on natural language @xcite , our bidirectional models learn more isotropic embeddings than causal (generative) ones. appendix g depicts umap representations of the embedding, showing the narrow cones and clusters they form. ## conclusion we showed that bpe can increase the quality of results of transformer models for symbolic music generation, and classification tasks, while significantly improving their efficiency and inference speed and making better use of their embedding spaces. bpe can be applied on top of any tokenization. the tokenization and decoding times are almost not affected by this extra step, when performed by a well-optimized rust code. considering the considerable benefits and low requirements of this technique, we advise anyone using transformer models with symbolic music to use bpe. there are still questions that remain uncovered. we showed that 40m parameters models can handle well vocabularies up to 20k tokens with mediumsize datasets. we however do not know what are the limits in vocabulary and dataset sizes over which the results might not improve. moreover, we experimented with bpe, but other common vocabulary building techniques exist, such as unigram @xcite and wordpiece @xcite . recent work on natural language showed that unigram yielded higher model performances than bpe @xcite , it might also be the case for symbolic music. future research will study these questions and hopefully find optimal tokenization guidelines to improve model performances under more various settings. bpe allows to build vocabulary based on data. hence, in case the data has specific token distributions, a model trained with this vocabulary might not generalize and perform well on data with opposite token distributions. bpe implies an additional step during data tokenization. in finally, although we experimented with two public datasets, two tokenizations and two tasks, we did not find a "limit" vocabulary size above which the results might not increase with. more research must be performed in order to find such limit.