HENT - SRT : Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translation (2025)

Abstract: Neural transducers (NT) provide an effective framework for speech streaming, demonstrating strong performance in automatic speech recognition (ASR). However, the application of NT to speech translation (ST) remains challenging, as existing approaches struggle with word reordering and performance degradation when jointly modeling ASR and ST, resulting in a gap with attention-based encoder-decoder (AED) models. Existing NT-based ST approaches also suffer from high computational training costs. To address these issues, we propose HENT-SRT (Hierarchical Efficient Neural Transducer for Speech Recognition and Translation), a novel framework that factorizes ASR and translation tasks to better handle reordering. To ensure robust ST while preserving ASR performance, we use self-distillation with CTC consistency regularization. Moreover, we improve computational efficiency by incorporating best practices from ASR transducers, including a down-sampled hierarchical encoder, a stateless predictor, and a pruned transducer loss to reduce training complexity. Finally, we introduce a blank penalty during decoding, reducing deletions and improving translation quality. Our approach is evaluated on three conversational datasets Arabic, Spanish, and Mandarin achieving new state-of-the-art performance among NT models and substantially narrowing the gap with AED-based systems.

## introduction translation of spoken conversations across languages plays a crucial role in cross-cultural communication, healthcare, and education @xcite @xcite . traditionally, speech translation (st) systems have been built using a cascaded approach, where automatic speech recognition (asr) first transcribes speech into text, which is then passed to a machine translation (mt) system @xcite @xcite @xcite . this modular approach facilitates the use of large text corpora for mt training, at the cost of ( 1 ) complex beam search algorithms in streaming applications @xcite , (2) error propagation from the asr to mt model, (3) an inability to leverage paralinguistic information such as prosody, and (4) additional latency, as mt processing must wait for asr to complete. to overcome the limitations of cascaded systems, end-to-end speech translation (e2e-st) has emerged as a promising approach that directly maps source speech to target text, providing a more streamlined architecture, reduced latency, and competitive performance @xcite @xcite @xcite . however, most end-to-end speech translation (e2e-st) research has focused on offline attention-based encoder-decoder (aed) architectures. as label-synchronous systems, aeds require multiple input frames before emitting each output token, which limits their suitability for streaming applications and increases sensitivity to utterance segmentation @xcite . to enable streaming in aed models, researchers have proposed wait-k policies, which introduce a controlled buffering mechanism to balance latency and translation quality @xcite @xcite . however, finding the optimal wait-k policy is challenging, as it must balance latency and quality, and the necessary buffering cause delays, making aed-based models less suitable for streaming applications. in contrast, frame-synchronous architectures like connectionist temporal classification (ctc) @xcite and the neural transducer (nt) @xcite more naturally handle streaming data and demonstrate greater robustness to utterance segmentation, mitigating the over-and undergeneration issues in aed models @xcite . while ctc enforces strictly monotonic alignments, which limits its ability to handle word reordering @xcite , the neural transducer (nt) relaxes these constraints, allowing the generation of longer output sequences and the modeling of autoregressive token dependencies. this makes nt more suitable for translation, as illustrated in figure 2 . to improve nt's reordering capabilities, researchers have proposed augmenting the joiner with cross-attention @xcite and the predictor with an aed-based decoder @xcite , though these additions increase computational complexity and latency. to further enhance translation quality, @xcite introduced attention pooling for better encoder-predictor fusion at the frame level, while @xcite explored similar mechanisms. additionally, @xcite proposed a transducer-based model for unified asr and st, but its shared encoder struggles with reordering, making it less effective than aed in offline settings. in this work, we propose a hierarchical neural transducer architecture, hent-srt, foot_0 which decomposes speech translation (st) into an automatic speech recognition (asr) task followed by a translation task, effectively handling word reordering. the system employs multi-task training to optimize asr and st objectives simultaneously with a separate predictor and joiner for each task. our objectives are three-fold: (1) to close the performance gap with state-of-the-art aed models in offline st settings, (2) to improve computational efficiency during training and inference, and (3) to jointly model asr and st while maintaining asr performance. to this end, we introduce hent-srt, a neural transducer-based st framework that addresses (1) through a hierarchical encoder architecture, enabling more effective handling of reordering in translation. to improve st efficiency in (2), we integrate a stateless 1d-cnn predictor @xcite and adopt the zipformer architecture @xcite , which achieves superior asr performance compared to state-of-the-art aed models such as e-branchformer @xcite , while offering greater computational efficiency. additionally, to reduce training complexity, we employ the pruned transducer loss @xcite . to mitigate asr degradation in joint modeling for (3), we employ self-distillation with ctc consistency regularization @xcite . we evaluate the effectiveness of our approach on conversational speech translation across three language pairs: tunisian arabic-english, spanish-english, and chinese-english described in @xcite . furthermore, we conduct a comprehensive ablation study to analyze the impact of each st design choice in both offline and streaming setups. ## proposed approach let @xmath0, . . . , x t ) ∈ r t ×f denote the source speech, a sequence of t acoustic feature vectors of dimension f . the goal is to generate a target word sequence y u , a translation of length u . we use superscripts (s) and (t) to refer to the source and target languages, respectively. speech translation task is trained using discriminative learning by minimizing the negative log-likelihood @xmath1. transducers compute this probability by marginalizing over the set of all possible alignments @xmath2: u is the deterministic mapping from an alignment a to the sequence y (t) of its non-blank symbols. transducers parameterize p (a|x) using an encoder, a prediction network, and a joiner, as illustrated in figure 1a . the encoder maps x to a representation sequence f t 1:k , @xmath3, . . . , t }, the predictor transforms y (t) sequentially into g t 1:u , @xmath4, . . . , u }, and the joiner combines f t 1:k and g t 1:u to generate logits z t k,u whose softmax is the posterior distribution of a k (over v(t) ), i.e. hello, how are you? ¿hola, cómo estás? where u(k) ∈ {1, • • • , u } denotes the index in the label sequence at time k. the negative log of the quantity in (2) is known as the transducer loss. ## conclusion in this paper, we proposed hent-srt, a novel hierarchical transducer architecture for joint speech recognition and translation (st). the model factorizes the st task into two stages, asr followed by a translation, enabling more effective handling of word reordering. to improve computational efficiency, hent-srt design incorporates key transducer-based asr practices, including a downsampled encoder, stateless predictor, and pruned transducer loss. to maintain robust st performance without sacrificing asr accuracy, we apply self-distillation with ctc consistency regularization. additionally, we introduce a blank penalty mechanism during decoding, which effectively reduces deletion errors and enhances translation quality. experimental results show that hent-srt significantly outperforms previous state-ofthe-art transducer-based st models and closes the gap with attention-based encoder-decoder architectures, while achieving superior asr performance. moreover, our approach offers substantial gains in streaming scenarios without introducing additional delays.