Bridging the Granularity Gap for Acoustic Modeling (2023)

Abstract: While Transformer has become the de-facto standard for speech, modeling upon the fine-grained frame-level features remains an open challenge of capturing long-distance dependencies and distributing the attention weights. We propose Progressive Down-Sampling (PDS) which gradually compresses the acoustic features into coarser-grained units containing more complete semantic information, like text-level representation. In addition, we develop a representation fusion method to alleviate information loss that occurs inevitably during high compression. In this way, we compress the acoustic features into 1/32 of the initial length while achieving better or comparable performances on the speech recognition task. And as a bonus, it yields inference speedups ranging from 1.20x to 1.47x.By reducing the modeling burden, we also achieve competitive results when training on the more challenging speech translation task.

## introduction despite the success in speech processing tasks like automatic speech recognition (asr) @xcite and speech translation (st) @xcite , how to encode the speech features effectively is an open problem. different from modeling based on discrete tokens in natural language processing, a standard paradigm for acoustic encoding is taking the continuous features extracted by signal processing as input. in this process, the framing operation generates a lengthy sequence consisting of fine-grained features. for example, frame-level feature sequences are dozens of times longer than the corresponding subword-level transcriptions (see figure 1 ). such an input leads to the difficulties of capturing longdistance dependencies and distributing the attention weights among semantically incomplete modeling units @xcite . @xcite also demonstrates that the fine-grained character-level modeling yield significantly inferior performance compared with coarse-grained word-level counterparts. in addition, the long sequence also results in prohibitive computation costs due to the quadratic complexity of self-attention. a popular method is to compress fine-grained features to form coarser-grained modeling units by stacking multiple down-sampling modules before encoding @xcite . unfortunately, it does not work well when we increase the down-sampling (ds) ratio like 16 and 32 to attain semantically more complete units. this is in line with the intuition that it is difficult to compress dozens of frames directly into a single unit @xcite . it is something like that a few principal components can not preserve all the information in the principal component analysis method @xcite . we analyze the compression process in the stack method to shed light on the reason for the failure. we find that the similarity of representations in adjacent positions degrades after down-sampling, which leads to the non-trivial issue of information loss and increases the difficulty of subsequent compression. to address this issue, we propose progressive down-sampling (pds), which gradually aggregates the input frame-level representations into semantically more complete text-level representations, like character-level or subword-level units. in this way, the model behaves as a natural language processing (nlp) system, which distributes the attention only among a short sequence and enables reinvesting saved computational resources into a deeper model or a bigger batch. nevertheless, we find that information loss still occurs as the compression ratio increases. as a remedy, we combine the coarse-grained high-level representation at top layers and fine-grained low-level representations at bottom layers that may preserve information lost during compression. our method outperforms the standard transformer and conformer on the asr and st tasks, and has the potential for application on a wide variety of acoustic modeling tasks. it also makes a trade-off between performance and computational speedup. with an extremely high compression ratio of 32, our method achieves better or comparable performances on the asr task while yielding inference speedups ranging from 1.20× to 1.47×. the lower ratio of 8 or 16 brings remarkable improvement and relatively modest speedup. on the more challenging st task, pds also helps convergence and shows competitive results. ## related work unlike text that has explicit boundaries, audio is in general represented as continuous signals. although researchers have explored straightforward modeling based on the raw audio signal @xcite , the popular method for segmentation is framing with a frame size of 25ms and a frame shift of 10ms @xcite . this short frame shift allows the continuity of the speech signal, and the overlapping segments help to avoid information loss between consecutive frames. however, the fine-grained frame-level features may not be suitable for the state-of-the-art transformer architectures @xcite . the lengthy sequences composed of semantically incomplete units lead to the difficulties of capturing long-distance dependencies and distributing the attention weights to the most related positions. researchers @xcite ) investigate phoneme-level methods. for example, one can average frame-level features within phoneme-like units. but this needs a non-trivial recognizer for phoneme alignment. motivated by the work in the efficient models @xcite , researchers alleviate the modeling difficulty by the improved self-attention mechanisms @xcite @xcite . however, they ignore the inherent problem of long sequence modeling and the cross-attention module still suffers from the same issue. another line of research is to down-sample the fine-grained features for shorter sequences @xcite . a popular approach is to pass the features through a stack of strided convolutional layers before encoding @xcite . but the stack method does not work well in practice due to the loss of information due to consecutive convolutional operations. as a way to address this, several research groups use the progressive method to down-sample the acoustic sequence @xcite @xcite . the key difference is that the previous studies motivate efficient computation and only explore a modest compression ratio of 8. recently, @xcite reduce the lengths by 16 times at the intermediate layers. our work aims to develop a model that takes audio as input but behaves as an nlp style to ease the modeling burden, which requires extremely high compression. although there is no special design for efficiency, our method still yields a remarkable speedup. another related open problem for acoustic encoding is the variable information caused by silence or noise. researchers develop adaptive selection @xcite or dynamic down-sampling methods @xcite for avoiding useless features. however, the granularity of the filtered representation is still far from ideal. these two methods are complementary and we will explore their combination in future work. ## experiments we evaluate our method on the librispeech and aishell-1 asr datasets, and the must-c en-de st dataset. details about the data and model settings are described in appendix a. ## analysis next, we study a number of interesting problems on librispeech. we present the comparisons with previous work and more analyses in appendix b. ## conclusion in this paper, we explore how to compress the fine-grained frame-level acoustic features to the semantically complete text-level units, like characterlevel and subword-level representations. to alleviate this granularity gap, we first investigate the down-sampling process and reveal the risk of information loss in the popular stack method. this inspires us to propose progressive down-sampling, which gradually attains the coarser-grained representation during encoding. furthermore, we develop a representation fusion method to combine the high-level and low-level information, which is important for high down-sampling ratios. by this, we are the first to compress the acoustic features with a ratio of 32 on the asr task while achieving comparable or even better performances. the more challenging st task demonstrates that the alleviated granularity gap facilitates convergence effectively.