Semi-Supervised Spoken Language Glossification (2024)

Abstract: Spoken language glossification (SLG) aims to translate the spoken language text into the sign language gloss, i.e., a written record of sign language. In this work, we present a framework named S emi- S upervised S poken L anguage G lossification ( S^3 LG) for SLG. To tackle the bottleneck of limited parallel data in SLG, our S^3 LG incorporates large-scale monolingual spoken language text into SLG training. The proposed framework follows the self-training structure that iteratively annotates and learns from pseudo labels. Considering the lexical similarity and syntactic difference between sign language and spoken language, our S^3 LG adopts both the rule-based heuristic and model-based approach for auto-annotation. During training, we randomly mix these complementary synthetic datasets and mark their differences with a special token. As the synthetic data may be less quality, the S^3 LG further leverages consistency regularization to reduce the negative impact of noise in the synthetic data. Extensive experiments are conducted on public benchmarks to demonstrate the effectiveness of the S^3 LG. Our code is available at https://github.com/yaohj11/S3LG .

## introduction sign language is the most primary means of communication for the deaf. translating between sign and spoken language is an important research topic, which facilitates the communication between the deaf and the hearing @xcite . to support the development of applications, sign language gloss has been widely used as an intermediate step for generating sign language video from spoken language text @xcite or the inverse direction @xcite . the sign language gloss is the written representation of the signs. as a generally adopted way for sign language transcription, gloss is sufficient to convey most of the key information in sign * corresponding authors: wengang zhou and houqiang li language. in this work, we focus on the first step of the former task named spoken language glossification (slg), which aims to translate the spoken language text into the sign language gloss. slg is typically viewed as a low-resource sequence-to-sequence mapping problem. the previous methods @xcite @xcite rely on the encoder-decoder architectures @xcite to jointly align the embedding space of both languages in a data-driven manner. since the data collection and annotation of sign language requires specialized knowledge, obtaining a large-scale text-gloss dataset is timeconsuming and expensive @xcite . as a result, the performance of slg models is limited by the quantity of parallel data @xcite . witnessing the success of introducing monolingual data to enhance lowresource translation quality @xcite @xcite in neural machine translation (nmt), we are motivated to explore the accessible unlabeled spoken language texts to improve slg. in this work, we present a framework named semi-supervised spoken language glossification (s 3 lg) to boost slg, which iteratively annotates and learns from pseudo labels. to implement the above idea, the proposed s 3 lg adopts both the rule-based heuristic and model-based approach to generate pseudo glosses for unlabeled texts. the rule-based synthetic data has high semantic accuracy, however, the fixed rules make it difficult to cover complex expression scenarios. the model-based approach on the other hand is more flexible for learning the correspondence between sign language and spoken language and generates pseudo gloss with higher synthetic diversity. these complementary synthetic datasets are randomly mixed as a strong supplement for the training of the slg model. besides, the model-based synthetic data is generated by the slg model, which sets a good stage for iteratively re-training the slg model. in addition, s 3 lg introduces a simple yet efficient design from three aspects. firstly, in each iteration, the training process is separated into two stages, i.e., pre-training and fine-tuning for domain adaptation. secondly, to encourage the model to learn from the noisy pseudo labels, we apply the consistency regularization term to the training optimization and gradually increase the weight of the consistency regularization in the training curriculum. it enforces the consistency of the predictions with network perturbations @xcite based on the manifold assumption @xcite . thirdly, to encourage the slg model to learn complementary knowledge from different types of synthetic data, a special token is added at the beginning of input sentences to inform the slg model which data is generated by the rule-based or model-based approach @xcite . through end-toend optimization, our s 3 lg achieves significant performance improvement over the baseline. surprisingly, the experiments show that the translation accuracy on low-frequency glosses is promisingly improved. we conjecture that the slg model acts differently in annotating the high-frequency and low-frequency glosses, and such bias is mitigated by the rule-based synthetic data. in summary, our contributions are three-fold: ## related work in this section, we briefly review the related works on spoken language glossification and semi-supervised learning. spoken language glossification. camgoz et al. considering gloss as a text simplification, li et al. (2022) propose a novel editing agent. instead of directly generating the sign language gloss, the agent predicts and executes the editing program for the input sentence to obtain the output gloss. by leveraging the linguistic feature embedding, egea gómez et al. (2021) achieve remarkable performance improvement. egea gómez et al. (2022) further apply the transfer learning strategy result in continues performance increasing. recently, zhu et al. (2023) first introduce effective neural machine translation techniques to slg with outstanding performance improvements, which lays a good foundation for further research. semi-supervised learning. generating pseudo labels for the unlabeled data is a widely adopted semi-supervised learning algorithm in low-resource nmt, known as back-translation @xcite and self-training @xcite , respectively. with the target-side monolingual data, back-translation obtains pseudo parallel data by translating the target-side sentences into the sourceside sequences. as an effective data augmentation method, it is widely adopted in the inverse task of slg, i.e., gloss-to-text translation @xcite @xcite . due to the lack of the sign language gloss corpus, it is hard to incorporate large-scale monolingual data in the training process of slg with back-translation (rasooli and tetreault, 2015). in contrast, the self-training requires source-side monolingual data to generate pseudo parallel data based on a functional sourceto-target translation system. since it is hard to optimize a neural translation system with an extremely limited amount of parallel data @xcite @xcite , this motivates us to go along this direction and design more effective algorithms. two-stage training 𝒇𝒇(ɵ 𝒌𝒌 ) synthetic data 𝒟𝒟 𝑈𝑈 𝑘𝑘-1 golden data 𝒟𝒟 𝐿𝐿 monolingual data 𝒳𝒳 𝑈𝑈 rule-based annotating k iterations model-based annotating 𝒇𝒇( � ɵ 𝒌𝒌-𝟏𝟏 ) final model 𝒇𝒇( � ɵ 𝑲𝑲 ) figure 1: overview of the proposed s 3 lg. it iteratively annotates and learns from pseudo labels to obtain the final slg model f ( θk ) after total k iterations. for the k-th iteration, the synthetic data d k-1 u is conducted by randomly mixing the two complementary pseudo glosses generated by the fixed rules and the previously obtained slg model f ( θk-1 ), respectively. note that at the first iteration, only the rule-based synthetic data is available. ## methodology in this section, we first introduce the overview of our s 3 lg in sec. 3.1; then we elaborate on the annotating methods for monolingual data in sec. 3.2; and finally, we detail the training strategy in sec. 3.3. ## conclusion in this work, we present a semi-supervised framework named s 3 lg for translating the spoken language text to sign language gloss. with the goal of incorporating large-scale monolingual spoken language texts into slg training, we propose the s 3 lg approach to iteratively annotate and learn from pseudo glosses. through a total of k iterations, the final slg model achieves significant performance improvement against the baseline. during each iteration, the two complementary synthetic data generated from the rule-based and modelbased approaches are randomly mixed and marked with a special token. we introduce a consistency regularization to enforce the consistency of the predictions with network perturbations. extensive ablation studies demonstrate the effectiveness of the above designs. besides, the translation accuracy on low-frequency glosses is improved.