Speech Translation with Style: A pp T ek’s Submissions to the IWSLT Subtitling and Formality Tracks in 2023 (2023)

Abstract: AppTek participated in the subtitling and formality tracks of the IWSLT 2023 evaluation. This paper describes the details of our subtitling pipeline - speech segmentation, speech recognition, punctuation prediction and inverse text normalization, text machine translation and direct speech-to-text translation, intelligent line segmentation - and how we make use of the provided subtitling-specific data in training and fine-tuning. The evaluation results show that our final submissions are competitive, in particular outperforming the submissions by other participants by 5% absolute as measured by the SubER subtitle quality metric. For the formality track, we participate with our En-Ru and En-Pt production models, which support formality control via prefix tokens. Except for informal Portuguese, we achieve near perfect formality level accuracy while at the same time offering high general translation quality.

## introduction this paper presents apptek's submissions to the subtitling and formality tracks of the iwslt 2023 evaluation campaign. in the subtitling track, we participate in constrained and unconstrained conditions and in both language pairs english-to-german (en-de) and english-to-spanish (en-es). in the formality track, we participate in the zero-shot unconstrained condition for english-to-portuguese (en-pt) and english-to-russian (en-ru). this paper is organized as follows: section 2 briefly describes our data preparation. section 3 presents apptek's pipeline for subtitle translation. its different components, namely audio segmentation, speech translation (st), automatic speech recognition (asr), machine translation (mt) models, and our subtitle segmentation algorithm are described in sections 3.1-3.5. section 3.6 contains experiments and an analysis of our subtitling systems. section 4 presents apptek's approach to * equal contribution formality-controlled machine translation. finally, section 4.1 shows the results of our formality track submission. ## formality control apptek's production systems support formality or, as we call it, style control for selected language pairs @xcite . this year, we decided to test these systems in the unconstrained condition of the iwslt formality track for en-pt and ento-ru. each of these two systems is trained in a transformer big setup @xcite . the formality level is encoded with a pseudo-token in the beginning of each training source sentence with one of 3 values: formal, informal, no style. the system is trained on large public data from the opus collection @xcite that has been partitioned into the 3 style classes as follows. first, we write a sequence of regular expressions for the target language (in this case, european pt and ru) which try to match sentences containing formal or informal features. thus, for russian, we try to match either the formal or informal secondperson pronoun that corresponds to english "you", including their possessive forms. for portuguese, we additionally match the forms of most common verbs which agree with the corresponding pronoun. the regex list for russian is given in each list of regular expressions uses standard regex syntax and makes either case-sensitive or insensitive matches. for each sentence pair from the parallel data, the regex list is processed from top to bottom. as soon as a match in the target sentence is found, the formal or informal label is assigned to the sentence pair. the sentence pair is labeled with no_style if there is no match. if document information is available and at least 5% of the document sentence pairs are labeled as formal/informal according to the regex rules (with no sentences labeled with the opposite class), then all of the sentence pairs in the document are assigned the corresponding label. such data is useful to model stylistic traits which are not limited to the choice of second-person pronouns. note that document annotations are available for some of the iwslt data, including ted talks, opensubtitles (each subtitle file corresponds to a document), individual sessions of european parliament, etc. we further smooth the three style classes to ensure that e.g., sentences containing second-person pronouns can be translated well even when no style is specified at inference time. to this end, 5 to 8% of sentence pairs which had been assigned to one of the 3 style classes as described above are randomly re-assigned to one of the other two classes. for en-ru, the training data that had been partitioned into style classes in this way included about informal ignorecase \b(ты|теб[яе]|тобой|тво[йеёяю]|твоей|твоего|твоему|твоим|тво[ёе]м)\b formal ignorecase \b(вы|вами?|ваш[ае]?|вашей|вашего|вашему?|вашу|вас|вашим)\b when training the transformer big model, we balanced the contribution of formal, informal, and "no style" data by adding them in equal proportions (number of lines) to each sub-epoch. ## conclusion we described apptek's submissions to the subtitling and formality tracks of the iwslt 2023. for the subtitling track, we obtained good results, outperforming the other two evaluation participants either with our constrained or unconstrained cascaded approach on all 4 domains. part of this success is due to our subtitle creation process, in which we employ apptek's intelligent line segmentation models. however, the results varied by domain, with the domain of movie subtitles posing the most challenges for asr, and the domain of fitness-related videos (peloton) being hardest for mt. yet our biggest overall challenge, especially for the direct (end-to-end) submission was speech segmentation and creating sentence-like units, on real itv movies in particular, in which there is music, background noise, and multiple speakers. in the future, we plan to improve this component of our speech translation technology. we also plan to include length control in our direct models which showed to be an important factor for those applications with time constraints. our formality track participation was a one-shot attempt at a zero-shot task that showed the competitiveness of the formality control that we have implemented in apptek's production systems. however, our approach currently requires the creation of manual regular expression rules for partitioning the parallel training data into formality classes, and the participation in the iwslt evaluation revealed some weaknesses of this approach for one of the involved target languages. in the future, we plan to further improve our approach, reducing or eliminating the need for writing rules.