P yramid C odec: Hierarchical Codec for Long-form Music Generation in Audio Domain (2024)

Abstract: Generating well-structured long music compositions, spanning several minutes, remains a challenge due to inefficient representation and the lack of structured representation. In this paper, we propose PyramidCodec, a hierarchical discrete representation of audio, for long audio-domain music generation. Specifically, we employ residual vector quantization on different levels of features to obtain the hierarchical discrete representation. The highest level of features has the largest hop size, resulting in the most compact token sequence. The quantized higher-level representation is up-sampled and combined with lower-level features to apply residual vector quantization and obtain lower-level discrete representations. Furthermore, we design a hierarchical training strategy to ensure that the details are gradually added with more levels of tokens. By performing hierarchical tokenization, the overall token sequence represents information at various scales, facilitating long-context modeling in music and enabling the generation of well-structured compositions. The experimental results demonstrate that our proposed PyramidCodec achieves competitive performance in terms of reconstruction quality and token per second (TPS). By enabling ultra-long music modeling at the lowest level, the proposed approach facilitates training a language model that can generate well-structured long-form music for up to 3 minutes, whose quality is further demonstrated by subjective and objective evaluations. The samples can be found at https://pyramidcodec.github.io/ .

## introduction with the advancement of language models, audio generation has achieved significant success, encompassing various domains such as common sound @xcite @xcite , speech (wang et al., 2023; copet * corresponding authors: weixue@ust.hk @xcite @xcite , and music @xcite @xcite . in these frameworks, audio tokenizers @xcite @xcite @xcite are employed to obtain discrete representations of audio, which are then used by transformer-based language modeling. in this paper, we focus on music generation in the audio domain, which is challenging due to the complexity and long-form nature of music. the most dominant idea for audio tokenization is applying vector quantization (vq). by perfoming vq in a residual manner, residual vector quantization (rvq) @xcite based high-fidelity universal neural audio compression methods, such as encodec @xcite , dac @xcite , are proposed, which exploit multiple codebooks to gradually capture the acoustic-level audio information from the most important to the least important. in @xcite , the hifi-codec is further proposed to balance the information of different levels of codec by using a group-residual vector quantization (grvq) method, which divides features into multiple groups and applies rvq to each group of features. however, to ensure high fidelity, these methods require a large number of codebooks, resulting in excessively long sequences for long-form music. in addition to applying rvq compression to the audio waveform, clam-tts @xcite obtains the discrete representation of mel spectrogram, achieving a high compression rate at approximately 10 hz. however, to ensure audio fidelity, they still employ 32 codebooks, which leads to a high token per second (tps) rate. although the rvq enables audio modeling using codebooks at different levels, in fact, they are only focusing on the "acoustic-level" modeling within a fixed time span, rather than "temporal-level" mod-eling at different spans. therefore, regardless of the strategy to improve rvq, the long-term modeling capacity of conventional methods is limited by the time span for modeling. for instance, some methods @xcite @xcite generate coarse acoustic tokens (first several rvq tokens), and the fine acoustic tokens are generated conditioning on coarse acoustic tokens in an auto-regressive manner. we also notice that @xcite seeks to integrate semantic information into the first codebook, hoping to generate conceptual outputs. however, for all these methods, tokenizers at all levels are still generated with the same compression rate and token length, hindering the long-term structural modeling of music. it is well-known that music has a hierarchical structure, in which high-level abstractions are the skeleton of music, containing the structure, theme, and rhythm of music, and the low-level information is the detailed high-frequency fluctuations of music. therefore, to generate structural long music, a hierarchical framework is needed to capture both high-level abstractions and low-level details. such hierarchical framework has also been applied to image generation such as vq-vae-2 @xcite and var @xcite . in this paper, we propose pyramidcodec, a hierarchical codec specifically designed generating of well-structured long-form music in the audio domain. pyramidcodec introduces an encoding scheme that operates across multiple scales. by incorporating a hierarchical framework, our codec effectively captures the essence of music at different levels of complexity. at the highest level, pyramid-codec captures the overall structure and essence of the music, producing a temporally compact representation enriched with abstract or skeletal information. as the scale level decreases within pyra-midcodec, the temporal resolution of the features increases, enabling the encoding of more intricate and nuanced aspects of the music. rvq is adopted to quantize the features at all levels, resulting in a multi-scale hierarchical codec. following the hierarchical codec design, we further develop a hierarchical training strategy to progressively enhance the codec's ability to reconstruct music at varying scales. given the groundtruth audio, multi-scale ground-truth targets and loss functions are proposed, following the same idea of hierarchical modeling. the model is trained to perform audio-form reconstruction at different scales. by employing the hierarchical training strat-egy, we can achieve comparable reconstruction quality at similar compression rates. based on the resulting codec, a language modelbased abstract-to-detail framework is finally proposed for long-form music generation. the main contributions are summarized as: ## proposed method in this section, we propose pyramidcodec, a hierarchical codec designed for efficient long-form music generation. the key problem is how to develop a framework that can jointly learn both high-level abstractions and low-level details, which differs from conventional audio codecs that generally adopt a fixed time span for tokenization at all levels. the proposed pyramidcodec involves pyramid encoding which gradually constructs the tokens at different time spans along with a series of upsampling, pyramid decoding which reconstructs the audio waveform from the tokens, and a hierarchical training strategy that enforces the details to be added gradually with more levels of tokens. the hierarchical codecs can be used to train an abstract-to-detail language model for music generation. the proposed method is illustrated in figure2. it is named pyramidcodec since the stack of tokenizers at different levels forms a pyramid structure. details of different parts are explained as follows. ## conclusion in this paper, we introduce pyramidcodec, a hierarchical codec designed for long-form music generation in the audio domain. we propose a hierarchical training strategy for pyramidcodec, where details are progressively incorporated using multiple levels of tokens. through pyramidcodec, we investigate the abstract-to-detail generation framework for music, and our experimental results highlight its effectiveness. in future work, we plan to explore the performance of pyramidcodec and the abstract-to-detail generation framework on real audio data, including speech, singing, music, and audio effects. additionally, we aim to explore conditional generation tasks, such as text to music.