How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System? (2025)

Abstract: Simultaneous speech-to-text translation (SimulST) translates source-language speech into target-language text concurrently with the speaker’s speech, ensuring low latency for better user comprehension. Despite its intended application to unbounded speech, most research has focused on human pre-segmented speech, simplifying the task and overlooking significant challenges. This narrow focus, coupled with widespread terminological inconsistencies, is limiting the applicability of research outcomes to real-world applications, ultimately hindering progress in the field. Our extensive literature review of 110 papers not only reveals these critical issues in current research but also serves as the foundation for our key contributions. We: 1) define the steps and core components of a SimulST system, proposing a standardized terminology and taxonomy; 2) conduct a thorough analysis of community trends; and 3) offer concrete recommendations and future directions to bridge the gaps in existing literature, from evaluation frameworks to system architectures, for advancing the field towards more realistic and effective SimulST solutions.

## introduction the term ''simultaneous'' was first coined in the field of language interpretation, which is the practice of conveying a speaker's message orally in another language to listeners who would not otherwise understand it. 1 unlike consecutive interpreting @xcite , where interpretation occurs after the selecting audio-textual context to retain from the past @xcite , it offers an incomplete and overly simplistic view of the broader challenges inherent in translating continuous audio streams. this narrow focus has been reinforced over the years, and recent surveys have continued to emphasize this view, assuming human-segmented audio as the standard setting for the task @xcite , as well as reinforcing a glaring terminological inconsistency affecting the simulst literature. terms such as ''streaming'', ''online'', and ''real-time'' are often used interchangeably with ''simultaneous,'' and many terms are used without explicit definitions, leading to significant ambiguity and confusion in understanding and comparing research work, their results, and subsequent findings, ultimately hindering the progress in the field. in this paper, we aim to address this terminological chaos and provide a clearer understanding of simulst and all its challenges, with a particular focus on processing continuous audio streams and the difficulties therein. after a brief overview of the speech translation landscape ( §2), our contributions are structured as follows: simulst system, from audio acquisition to translation presentation, and propose a unified terminology to standardize the task. we also introduce a taxonomy based on the dichotomies identified in our analysis of fundamental system components ( §3). ## recommendations and future directions in this section, we outline best practices derived from the analysis in §4 and the recent advances in the field ( ), and we highlight key areas where future research is needed to develop more robust, accurate, and efficient simulst systems capable of meeting real-world demands ( ). use (at least) automatic pre-segmentation. as discussed in §4, the simulst community has predominantly relied on using gold segmentation for training and evaluating their systems. since this represents unrealistic conditions for real-world simulst applications, we encourage future research in the bounded speech scenario to use automatic segmentation instead as input for their models. offline automatic audio segmentation can be achieved using vad or neural-based tools such as shas ( §2.2). although all audio files are segmented before starting the simultaneous process, they provide a more realistic input, closer to real-world scenarios where audio segmentation (if any) is performed automatically and on the fly. this shift will better prepare models for practical deployment, ensuring that they can handle the challenges of processing speech that is not always segmented into well-formed sentences. ## conclusions in this paper, we examined the state of simultaneous speech translation research under several aspects, identifying significant gaps in the existing literature. our analysis of 110 papers revealed a predominant focus in simulst on human-segmented speech, which oversimplifies the task and neglects the complexities of real-world applications. we also uncovered substantial terminological inconsistencies, revealing real terminological chaos. to address these issues, we formalized the simulst task as a 6-step process and introduced a unified terminology to standardize research outcomes. we identified the core components of simulst systems (input, architecture, and output strategy), discussed current research trends, and provided key recommendations, including transitioning from human to automatic segmentation and adopting consistent terminology. we also emphasized the need for improvement in current evaluation frameworks, highlighting the importance of creating an easyto-use tool that can handle unbounded speech, incorporating contextual information during translation, and investigating more user-centric assessments to ensure that improvements measured by automatic metrics align with those in the user experience. claudio fantinuoli and bianca prandi. 2021. towards the evaluation of automatic simultaneous speech translation from a communicative perspective. notice that querying for ''speech'' already includes the results for ''speech-to-text'' and similar combinations. moreover, since we are interested in trends in simulst systems, we include only papers proposing models (i.e., excluding corpora, surveys, and metrics) and providing results for the speech-to-text task (i.e., speech-to-speech and/or text-to-text are not considered). only papers written in english and with an open-access version have been considered. the analysis resulted in 110 papers, categorized following our taxonomy (figure 2 ) and reported in the following in chronological order. notice that, in some cases, the number of papers on the various dichotomies does not sum to 110 since some work proposes, for instance, both cascade and direct models and appear in both categories.