CUNI - KIT System for Simultaneous Speech Translation Task at IWSLT 2022 (2022)

Abstract: In this paper, we describe our submission to the Simultaneous Speech Translation at IWSLT 2022. We explore strategies to utilize an offline model in a simultaneous setting without the need to modify the original model. In our experiments, we show that our onlinization algorithm is almost on par with the offline setting while being 3x faster than offline in terms of latency on the test set. We also show that the onlinized offline model outperforms the best IWSLT2021 simultaneous system in medium and high latency regimes and is almost on par in the low latency regime. We make our system publicly available.

## introduction this paper describes the cuni-kit submission to the simultaneous speech translation task at iwslt 2022 @xcite by charles university (cuni) and karlsruhe institute of technology (kit). recent work on end-to-end (e2e) simultaneous speech-to-text translation (st) is focused on training specialized models specifically for this task. the disadvantage is the need of storing an extra model, usually a more difficult training and inference setup, increased computational complexity @xcite and risk of performance degradation if used in offline setting @xcite . in this work, we base our system on a robust multilingual offline st model that leverages pretrained wav2vec 2.0 @xcite and mbart @xcite . we revise the onlinization approach by @xcite and propose an improved technique with a fully controllable qualitylatency trade-off. we demonstrate that without any change to the offline model, our simultaneous system in the mid-and high-latency regimes is on par with the offline performance. at the same time, the model outperforms previous iwslt systems in medium and high latency regimes and is almost on par in the low latency regime. finally, we observe a problematic behavior of the average lagging metric for speech translation @xcite when dealing with long hypotheses, resulting in negative values. we propose a minor change to the metric formula to prevent this behavior. our contribution is as follows: ## related work simultaneous speech translation can be implemented either as a (hybrid) cascaded system @xcite @xcite @xcite or an end-to-end model @xcite . unlike for the offline speech translation where cascade seems to have the best quality, the end-to-end speech translation offers a better qualitylatency trade-off @xcite @xcite . end-to-end systems use different techniques to perform simultaneous speech translation. han et al. (2020) uses wait-k @xcite model and metalearning @xcite to alleviate the data scarcity. @xcite uses a unidirectional encoder with monotonic cross-attention to limit the dependence on future context. other work @xcite proposes cross attention augmented transducer (caat) as an extension of rnn-t @xcite . @xcite proposed a hypothesis stability detection for automatic speech recognition (asr). the shared prefix strategy finds the longest common prefix in all beams. @xcite explore such strategies in the context of speech recognition and translation. the most promising is the longest common prefix of two consecutive chunks. the downside of this approach is the inability to parametrize the quality-latency trade-off. we directly address this in our work. ## onlinization in this section, we describe the onlinization of the offline model and propose two ways to control the quality-latency trade-off. ## experiments setup in this section, we describe the onlinization experiments. ## experiments and results in this section, we describe the experiments and discuss the results. ## submitted system in this section, we describe the submitted system. we follow the allowed training data and pretrained models and therefore our submission is constrained (see section 4.2.1 for model description). for stable hypothesis detection, we decided to use the local agreement strategy with @xmath0. as shown in section 5.2, the la-2 has the best latencyquality trade-off along with other la-n strategies. to achieve the different latency regimes, we use various chunk sizes, depending on the language pair. we decided not to use larger @xmath1, as it increases the computation complexity while having the same effect as using a different chunk size. the results on must-c tst-common are in from ## conclusion in this paper, we reviewed onlinization strategies for end-to-end speech translation models. we identified the optimal stable hypothesis detection strategy and proposed two separate ways of the qualitylatency trade-off parametrization. we showed that the onlinization of the offline models is easy and performs almost on par with the offline run. we demonstrated that an improvement in the offline model leads to improved online performance. we also showed that our method outperforms a dedicated simultaneous system. finally, we proposed an improvement in the average latency metric.