JWS ign: A Highly Multilingual Corpus of B ible Translations for more Diversity in Sign Language Processing (2023)

Abstract: Advancements in sign language processing have been hindered by a lack of sufficient data, impeding progress in recognition, translation, and production tasks. The absence of comprehensive sign language datasets across the world’s sign languages has widened the gap in this field, resulting in a few sign languages being studied more than others, making this research area extremely skewed mostly towards sign languages from high-income countries. In this work we introduce a new large and highly multilingual dataset for sign language translation: JWSign. The dataset consists of 2,530 hours of Bible translations in 98 sign languages, featuring more than 1,500 individual signers. On this dataset, we report neural machine translation experiments. Apart from bilingual baseline systems, we also train multilingual systems, including some that take into account the typological relatedness of signed or spoken languages. Our experiments highlight that multilingual systems are superior to bilingual baselines, and that in higher-resource scenarios, clustering language pairs that are related improves translation quality.

## introduction there are around 300 sign languages recorded up to date (united nations, 2021). however, sign language translation research is extremely skewed towards a limited number of sign languages, primarily those from high-income countries @xcite , while ignoring the vast majority of sign languages used in low and middle-income countries @xcite . a similar phenomenon was observed in the spoken foot_0 languages machine translation community and was shown by @xcite to be harmful, calling on the nlp community to do more research on low resource spoken languages @xcite . this issue is exacerbated by the fact that approximately 80% of people with disabling hearing loss in the world reside in middle and low-income countries @xcite . our first contribution towards addressing these challenges is to present jwsign, a highly multilingual corpus of bible translations in 98 sign languages, made accessible through an automated loader. to the best of our knowledge, jwsign is one of the largest and most diverse datasets to date in sign language processing ( §3). there is precedent in natural language processing (nlp) for using bible translations as a starting point for many under-resourced languages that may not have any parallel resources in other domains. bible corpora have played a major role in research in speech and text areas of nlp ( §2). we complement the jwsign dataset with baseline experiments on machine translation, training a transformer-based system for 36 bilingual pairs of languages in the dataset. such bilingual systems, trained individually for each language pair (one sign language and one spoken language), are the default procedure in recent literature. however, sign language translation (slt) has proven to be a challenging task, due to obstacles such as very limited amounts of data, variation among individual signers and sub-optimal tokenization methods for sign language videos. a potential way to improve over bilingual systems (the predominant kind at the time of writing) is to build multilingual systems. linguistic studies have suggested a good level of similarity and mutual intelligibility among some sign languages @xcite even from different continents (e.g. ghanaian sign language in africa and american sign language in north america). in this work, we therefore explore different multilingual settings for sign language translation ( §4). ## related work the following section explores the motivation behind the research by examining works that have utilized the bible in various modalities ( §2.1). it delves into the limited coverage of many sign languages within popular existing datasets for sign language translation ( §2.2), and provides a comprehensive overview of the state-of-the-art methods employed for automatic translation of sign language videos into text ( §2.3). ## jwsign in this section, we give an overview of jwsign ( §3.1) and list key statistics, comparing jwsign to other recent datasets ( §3.2). finally, we explain our process of creating fixed data splits for (multilingual) machine translation experiments ( §3.3). ## experimental setup we perform preliminary machine translation experiments on the jwsign dataset. in this section we explain our preprocessing steps ( §4.1), how different models are trained ( §4.2) and our method of automatic evaluation ( §4.3). ## results and discussion although all language pairs in this work are considered very low-resourced when compared to spoken languages @xcite , going forward we use the term high to refer to language pairs with more than 10,000 training samples, medium for language pairs with training samples between 1,000 and 9,999 inclusive, low for language pairs with training samples between 500 and 999 inclusive, very low for language pairs with training samples less than 500 and zero for language pairs with no training samples. our main findings are summarized in ## conclusion and future work in this study, we introduced jwsign as a unique resource aimed at promoting diversity in sign lan-guage processing research which has been so far dominated by few sign languages. we conducted a series of baseline experiments using jwsign to attempt improving the scores of automatic sign translation systems in different scenarios. we demonstrate that multilingual training leads to better translation quality compared to bilingual baselines. on the other hand, our experiments did not show a clear benefit for a fine-tuning approach in lowerresource scenarios. similarly, we found that clustering data by language family, even though intuitively promising, is only beneficial in higher-resource settings. more generally, the overall translation quality is still very low. this is in line with other recent studies such as @xcite who report bleu scores in a similar range. regardless, we firmly believe that as we strive to improve translation systems, it is crucial to ensure early diversification of the sign languages used to train these systems. by incorporating a wide range of sign languages during the training phase, we can enhance the inclusivity and effectiveness of the resulting translation systems. as part of our future research, we aim to develop enhanced models utilizing jwsign that can effectively handle multiple sign languages. furthermore, jwsign presents a distinctive opportunity to address the existing gaps in sign language processing, such as the development of a sign language identification tool. jwsign can also serve as a valuable tool for linguists to explore and compare various sign languages in an attempt to gain more insights, such as further inquiries into the typological relatedness of sign languages.