J apanese-to- E nglish Simultaneous Dubbing Prototype (2023)

Abstract: Live video streaming has become an important form of communication such as virtual conferences. However, for cross-language communication in live video streaming, reading subtitles degrades the viewing experience. To address this problem, our simultaneous dubbing prototype translates and replaces the original speech of a live video stream in a simultaneous manner. Tests on a collection of 90 public videos show that our system achieves a low average latency of 11.90 seconds for smooth playback. Our method is general and can be extended to other language pairs.

## introduction live video streaming over the internet has become a very important form of communication in human society. it has many advantages such as fast, not constrained by distance, economical and safe. if the language barrier (ahmad abuarqoub, 2019) can be broken down in live video streaming, it will greatly promote global communication. however, the current common solution to crosslanguage live video streaming is to use automatic simultaneous interpretation @xcite @xcite to display translated subtitles. reading subtitles at the bottom of the screen is uncomfortable and degrades the viewing experience @xcite . our simultaneous dubbing prototype aims to help live video streaming break down language barriers. our prototype translates and replaces the original speech of a live video stream, creating a seamless viewing experience in the target language. the rest of this paper is organized as follows. first, section 2 reviews related works. then, section 3 describes our method for implementing simultaneous dubbing. after that, section 4 tests our system on a collection of 90 public videos in live video streaming asr ss live video streaming in target language image speech … video clips containing single sentences mt tts video clips containing translated sentences ## related works automatic simultaneous interpretation and automatic language dubbing are the two topics most closely related to our work. ## methods our prototype accomplishes simultaneous dubbing through three main steps as (figure 1 ), 1. segmenting the source video stream into video clips that contain one single sentence using automatic speech recognition @xcite and sentence segmentation @xcite . for automatic speech recognition, we use the transformerbased @xcite acoustic model and the seq2seq criterion @xcite implemented in flashlight (pratap et al., 2019) foot_4 . for sentence segmentation, we replace the backbone network of cytonnss @xcite foot_5 with transformer to improve accuracy. the main challenge of simultaneous dubbing is that the output of sentence segmentation (step 1) and machine translation (step 2) is irregular in time, but video streaming is constantly consuming data. for example, in the source stream, someone speaks a sentence for about 15 seconds. the system then spends another 5 seconds generating the translated speech waveform. this results in a 20-second data gap in the output stream. the adaptive playback method addresses this challenge while maintaining low latency (figure 3 ). ## evaluation our system is tested on a collection of 90 public videos of japanese interviews, speeches, presentations and lectures. the total running time of the collection is approximately 21 hours 45 minutes. the tests are run on a desktop computer equipped with one intel xeon e5-2630 v3 cpu and two nvidia quadro rtx 4000 gpus. the test results are shown in our system presets three modes, fast, balance and quality, for different trade-offs of speed and quality. users can select the mode according to the application. ## conclusion this paper presents our japanese-to-english simultaneous dubbing prototype. the system enables low-latency and smooth live video streaming in the target language. we believe this technology will find widespread use in global communications. in the future, we plan to add optical character recognition to our system. video streaming often displays some text, such as the slides that appear in a lecture. text in video streaming is an important source of information for viewers. therefore, we hope that by recognizing and translating the text in video streaming, our system can provide users with a complete viewing experience in the target language.