Codec- SUPERB : An In-Depth Analysis of Sound Codec Models (2024)

Abstract: The sound codec’s dual roles in minimizing data transmission latency and serving as tokenizers underscore its critical importance.Recent years have witnessed significant developments in codec models.The ideal sound codec should preserve content, paralinguistics, speakers, and audio information.However, the question of which codec achieves optimal sound information preservation remains unanswered, as in different papers, models are evaluated on their selected experimental settings.This study introduces Codec-SUPERB, an acronym for Codec sound processing Universal PERformance Benchmark.It is an ecosystem designed to assess codec models across representative sound applications and signal-level metrics rooted in sound domain knowledge.Codec-SUPERB simplifies result sharing through an online leaderboard, promoting collaboration within a community-driven benchmark database, thereby stimulating new development cycles for codecs.Furthermore, we undertake an in-depth analysis to offer insights into codec models from both application and signal perspectives, diverging from previous codec papers mainly concentrating on signal-level comparisons.Finally, we will release codes, the leaderboard, and data to accelerate progress within the community.

## introduction neural sound codec models were initially introduced to compress sound for efficient data transmission. the encoder of the codec model encodes the sound into codec codes, which are then transmitted. subsequently, the codec decoder then resynthesizes the sound using the received codes. neural codec codes can be utilized as tokens in sound language modeling (lm). lm has proven * equal first contribution, † equal second contribution, order is sorted randomly. the dual roles of minimizing data transmission latency and serving as sound lm tokenizers require an ideal codec to preserve content, paralinguistic, speaker, and audio information under low bitrate measured by thousand bits per second (kbps). however, the question of which codec achieves the most optimal information preservation across various aspects remains unanswered, as codec models are evaluated with various experimental settings in different papers. furthermore, prior codec papers mainly compare performance based on signal-level metrics, neglecting downstream application angles. to address the aforementioned limitations, we introduce codec-superb, shorted for codec sound processing universal performance benchmark, which firstly provides a holistic comparison of the current state-of-the-art codecs under the same, fair and comprehensive experimental setting. we highlight the following features of codec-superb: ## codec-superb platform design as shown in figure 1 , codec-superb is designed to foster sound codec development by providing a platform for connecting codec developers and codec users. codec-superb is user-friendly for reproducing the model evaluation, assessing the custom codec models, contributing datasets and metrics, and conducting comparative analyses for model characteristics. this is facilitated by three core components: an easy-to-follow codebase, a community-driven leaderboard website, and wellselected datasets. ## holistic evaluation in codec-superb codec models are assessed across diverse experimental settings in various papers. moreover, previous codec studies primarily focus on comparisons using signal-level metrics on their selected datasets, overlooking evaluations from downstream application angles. thus, we make efforts to address the above limitations by including diverse datasets, comprehensive signal-level metrics, and mainstream sound applications. the detailed evaluation process is illustrated in figure 2 . ## limitations our evaluation data spans 20 datasets, each containing numerous testing samples, resulting in a substantial amount of data that requires significant computation time and resources. as a solution, we aim to devise criteria for identifying representative data points within each dataset to expedite the evaluation process. ## 6 conclusion this study presents codec-superb, a public framework tailored to fairly and comprehensively evaluate codec models. comprised of a user-friendly codebase, a collaborative benchmark driven by the community, and meticulously crafted metrics paired with curated datasets, codec-superb streamlines result comparisons through an interactive online leaderboard. additionally, our comprehensive analysis provides valuable insights into codec models, examining both application and signal perspectives. this departure from prior codec research, primarily focusing on signallevel comparisons, allows for a richer understanding of codec performance. our innovative introduction of a unified overall score sets us apart from previous works, seamlessly integrating all signallevel metrics to enhance visualization. remarkably, this overall score exhibits robust correlations with each individual metric, underscoring its reliability. lastly, we commit to releasing codes, the leaderboard, and data resources to expedite progress and foster growth within the community.