ESC : Efficient Speech Coding with Cross-Scale Residual Vector Quantized Transformers (2024)

Abstract: Neural speech codecs aim to compress input signals into minimal bits while maintaining content quality in a low-latency manner. However, existing neural codecs often trade model complexity for reconstruction performance. These codecs primarily use convolutional blocks for feature transformation, which are not inherently suited for capturing the local redundancies in speech signals. To compensate, they require either adversarial discriminators or a large number of model parameters to enhance audio quality. In response to these challenges, we introduce the Efficient Speech Codec (ESC), a lightweight, parameter-efficient speech codec based on a cross-scale residual vector quantization scheme and transformers. Our model employs mirrored hierarchical window transformer blocks and performs step-wise decoding from coarse-to-fine feature representations. To enhance bitrate efficiency, we propose a novel combination of vector quantization techniques along with a pre-training paradigm. Extensive experiments demonstrate that ESC can achieve high-fidelity speech reconstruction with significantly lower model complexity, making it a promising alternative to existing convolutional audio codecs.

## introduction recent advancements in deep learning have demonstrated the superiority of neural speech codecs over traditional ones, which rely on complex expert design and psycho-acoustic knowledge @xcite . early efforts integrating deep generative models, such as wavenet @xcite and samplernn @xcite , into audio codecs have delivered promising results. these models, acting as powerful decoders, synthesize high-quality speech from intermediate representations produced by traditional codecs (kleijn 1 code and pretrained models available at https:// github.com/yzguu830/efficient-speech-codec @xcite . however, their auto-regressive nature of the decoding process often introduces significant inference latency, limiting their practical application. alternatively, some end-to-end neural audio codecs leverage the vector quantization (vq) network first introduced by @xcite . vq networks use a learnable collection of codevectors, known as a codebook, to quantize continuous vectors by assigning them to the nearest codeword. this discretization positions vqns well-suited for both generation and compression tasks. following this approach, existing vq codecs @xcite @xcite typically employ a three-stage architecture: a convolutional encoder and decoder, and a residual vector quantization (rvq) module @xcite applied in the latent space. the encoder and decoder downsample and upsample audio waveform features, creating hierarchical representations. rvq further refines vanilla vector quantization by minimizing quantization error through a series of codebooks that recursively quantize the residuals from previous stages. additionally, these codecs employ adversarial discriminators to remove artifacts and produce high-fidelity audio reconstructions. substantial effort has been dedicated to designing effective audio discriminators, including an improved feature matching loss @xcite , as well as various multi-resolution waveform and spectrogram discriminators @xcite @xcite . vq-based audio codecs have demonstrated remarkable performance in audio reconstruction, even at ultra-low bitrates. despite these advantages, we find that convolutional vq codecs heavily depend on powerful discriminators to produce high-quality audio, posing additional optimization challenges due to adversarial training. moreover, these codecs tend to confront computational constraints, as they require a large number of parameters to balance high compression rates and reconstruction performance. to address these issues, our work develops a more parameter-efficient speech codec by reducing model complexity and implementing the following architectural improvements: 1) replacing convolutional layers with efficient swin-transformer blocks (stbs) @xcite , which can better model acoustic features; 2) utilizing the cross-scale residual vector quantization (cs-rvq) scheme @xcite instead of rvq, extending quantization from a fixed level to multiple levels. in addition, training vq codecs frequently leads to a significant challenge: codebook collapse, where a fraction of the codebook remains underutilized in representing input vectors. this issue is frequently observed when training visual tokenizers for generative vision tasks @xcite @xcite . to address this problem in speech compression, we propose combining product vector quantization (pvq) @xcite , code factorization @xcite , and euclidean normalization @xcite to enhance codebook utilization. furthermore, we introduce a learning paradigm to facilitate optimization, which includes a pre-training stage where the codebooks are deactivated and trained subsequently. in summary, the key contributions of our work are as follows: ## conclusions in this paper, we introduce esc, the first fully transformer-based neural speech foundation model designed for multilingual speech coding. esc surpasses existing state-of-the-art time-domain vqbased codecs in terms of complexity and achieves comparable compression performance without the need for a powerful adversarial discriminator. our extensive evaluations demonstrate that the crossscale residual vector quantization scheme and the swin transformer backbones are better suited for neural speech coding than the convolutional blocks and residual vector quantization utilized in mainstream codecs. overall, our study suggests a promising direction for speech foundation models. future research could focus on expanding multiscale vector quantization techniques and investigating additional transformer variants optimized for speech signal modeling.