Evaluating Multilingual Speech Translation under Realistic Conditions with Resegmentation and Terminology (2023)

Abstract: We present the ACL 60/60 evaluation sets for multilingual translation of ACL 2022 technical presentations into 10 target languages. This dataset enables further research into multilingual speech translation under realistic recording conditions with unsegmented audio and domain-specific terminology, applying NLP tools to text and speech in the technical domain, and evaluating and improving model robustness to diverse speaker demographics.

## introduction the nlp and speech communities are rapidly expanding, which has motivated increased interest in multilingual scientific communication and accessibility. from the automatic captioning at naacl 2019 provided by microsoft to the current acl 60-60 initiative foot_0 for the 60th anniversary of acl at 2022, it is clear that transcription and translation in the technical domain is needed, desired, and still a disproportionate challenge for current models compared to standard datasets in these spaces. translating technical presentations presents challenging conditions, from domain-specific terminology and adaptation, to recordings often captured with a laptop microphone and light background noise, diverse speaker demographics as well as unsegmented speech typically 10-60 minutes in duration. we have curated evaluation sets from presentations at acl 2022 which have been professionally transcribed and translated with the support of acl and the 60-60 initiative. in this paper we describe the methodology to create this dataset, considerations and methods to evaluate speech translation models with it, and open challenges we believe this dataset may support research towards. we release all data and intermediate steps to support further research in this space. we present the acl 60/60 evaluation sets to enable greater development of tools by the field for the field. specifically, we hope that this data enables further research into speech translation and other nlp applications in the technical domain with resegmentation and terminology, given a diverse speaker set and realistic recording conditions, with the goal of increased accessibility and multilinguality. our dataset is publicly available through the acl anthology. 2 ## evaluation under realistic conditions to evaluate transcription and translation under realistic conditions may require different metrics than with e.g. provided segmentation. here we present the necessary metrics in order to discuss the dataset creation process. ## related work previous work has studied data from the acl anthology for term mining and identification @xcite and concept relation @xcite in the scientific domain. few speech translation datasets in the technical domain exist but those that do such as the qcri educational corpus @xcite have primarily targeted educational lectures and videos. additional datasets specifically for speech translation evaluation @xcite are primarily 'general domain.' significant previous work has studied various aspects of translation post-editing, including postediting effort @xcite , evaluating postediting quality and reference bias @xcite , bias from the initial mt quality and output patterns @xcite , and the the efficacy of post-editing in highly technical domains @xcite and resulting translation biases @xcite . the impact of automatic segmentation quality on various st metrics has been evaluated in recent iwslt shared tasks @xcite @xcite and research @xcite @xcite using other datasets (ted) with longer reference segmentations than ours. with longer sequences there is greater potential for variation, and past campaigns have observed larger differences between segmentations than seen here and even improvements over the provided segmentation. significant additional work has been done in the simultaneous translation space, which we do not address here. ## conclusions we introduced a new dataset to evaluate multilingual speech translation from english into ten target languages specifically in the technical nlp domain. we have discussed in detail the steps to create the corpus and the tools and considerations required. we have also provided a further view into evaluation methodology mimicking realistic conditions where segmentation is not provided. we hope that this dataset may be useful for the field to study the effectiveness of the tools we develop both for translation and additional applications in the technical domain in an increasingly multilingual space.