Genre Transfer in NMT :Creating Synthetic Spoken Parallel Sentences using Written Parallel Data (2022)

Abstract: Text style transfer (TST) aims to control attributes in a given text without changing the content. The matter gets complicated when the boundary separating two styles gets blurred. We can notice similar difficulties in the case of parallel datasets in spoken and written genres. Genuine spoken features like filler words and repetitions in the existing spoken genre parallel datasets are often cleaned during transcription and translation, making the texts closer to written datasets. This poses several problems for spoken genre-specific tasks like simultaneous speech translation. This paper seeks to address the challenge of improving spoken language translations. We start by creating a genre classifier for individual sentences and then try two approaches for data augmentation using written examples:(1) a novel method that involves assembling and disassembling spoken and written neural machine translation (NMT) models, and (2) a rule-based method to inject spoken features. Though the observed results for (1) are not promising, we get some interesting insights into the solution. The model proposed in (1) fine-tuned on the synthesized data from (2) produces naturally looking spoken translations for written-to-spoken genre transfer in En-Hi translation systems. We use this system to produce a second-stage En-Hi synthetic corpus, which however lacks appropriate alignments of explicit spoken features across the languages. For the final evaluation, we fine-tune Hi-En spoken translation systems on the synthesized parallel corpora. We observe that the parallel corpus synthesized using our rule-based method produces the best results.

## introduction style transfer has been one of the well-studied tasks in the field of artificial intelligence (ai). most of the earlier works using deep learning were in the domain of computer vision (cv; @xcite . the task of text style transfer (tst) saw a surge in research @xmath0. because of its easy maintenance and availability, written data have been extensively experimented with. this works for most tasks; however, with the increasing popularity of processing speech, such as simultaneous translation of spoken language, the need for speech-specific data grows. however, transcribing large volumes of audio datasets is a tedious and costly process. in addition, creating a parallel dataset for such tasks requires the further step of translation. applying methods of tst can offer a solution to this problem by utilizing the available large amount of written parallel text. the utilization of written parallel data for the creation of spoken ones is not straightforward. a major issue with this task is having a clear distinction between spoken and written genres. spoken genre spans a broad spectrum with spontaneous conversations or speeches at one extreme and prepared speeches at the other one. the lack of spontaneity in the latter case can make sentences indistinguishable from the written genre. the existing parallel datasets rarely contain genuine examples of spontaneous speech, as most of the filler words and pauses are cleaned while transcribing audio samples. most works in the field do not address the challenge of transfer between two similar linguistic styles. since most of them are based on data-driven approaches, they are prone to failing in this setting due to insufficiently distinct features of the two genres. the common alternative, simple rule-based approaches model only content-independent features. we thus see a need for a combination of both approaches. the main objective of this paper is to improve spoken translation systems. we address the lack of parallel spontaneous speech corpora using tst from written genre to spoken genre in the context of nmt. in this work, we try different data augmentation methods to construct synthetic spoken-style parallel dataset from existing written genre parallel datasets. we inject disfluencies in both languages on the phrase level. even though not all disfluencies are consistent across languages in real life, we try to preserve them in the constructed parallel data, at least on the phrase level. we propose a data-driven approach involving nmt models from both genres. we combine the encoders and decoders extracted from translation systems trained on each of the genres separately. we also propose a rule-based method for constructing spoken-style parallel data by injecting spoken features at the phrase-level to existing written parallel datasets. we check the applicability of the created synthetic parallel datasets for hindi-english spoken language translation systems. to summarize, the main contributions of this paper are: we discuss work related to our approach in section 2. we provide an overview of the used datasets in section 3. we describe our proposed genre classifier in section 4 and continue with our data augmentation methods in sections 5 and 6. we provide analysis of the results in section 7. we discuss some alternatives and future directions in section 8 and finally conclude our paper in section 9. we publish our source code, and pre-trained models on github. foot_0 2 related works our work touches upon three topics: neural machine translation (section 2.1), text style transfer (section 2.2), and evaluation of stylised texts (section 2.3). ## data we use the samanantar corpus for our experiments. we split the corpus into written and spoken parts using the data sources. we consider sources from lectures (coursera, nptel, khanacademy, and kurzgesagt), and opensubtitles belonging to the spoken genre, and wikipedia to the written genre. there are abundant examples where sentences have lexical filler words; however, very few contain nonlexical fillers or repetitions. we have a total of 171, 416 parallel sentences for the spoken genre and 216, 183 for the written one. we separate 5, 000 sentences as a test set, 10, 000 as validation set, and the remaining sentences as a training set. we ensure no training sentence appears in the test or validation data. ## genre classifier classifying genre plays a vital role in the evaluation of genre transfer tasks. majority of the papers in tst use deep-learning-based classifiers to classify the specific style. along the lines of existing approaches, we train a bert-based genre classifier. we train two models: with and without voice 2.0 dataset for en classifier. since we have parallel we use distilbert @xcite as our pretrained model for en language. it is a fast, cheap and light transformer model trained by distilling bert base. it has 40% fewer parameters and is 60% faster than bert-base-uncased. we train the model on en sentences from samanatar corpus (en base ) and samanantar + voice 2.0 corpus (en voice ). we use huggingface trainer for our experiments. we use the same tokenizer as distil-bert and set the sequence max length to 256. we train the models for 2 epochs with a batch size of 8 while keeping the best checkpoints at each 500 steps. we do not train the model further as its performance stopped improving after 2 epochs. we use 10000 sentences as test (comprised of examples from samanantar and voice 2.0), 20000 sentences as validation, and the remaining sentences in the dataset as training data. we use indicbert @xcite ) as our pretrained model for hi genre classification. indicbert is a multilingual albert @xcite trained on 12 indic languages. the model has sota performance when compared to other multilingual models. we train the model on hi sentences from opensubtitles, online lectures, and wikipedia in samanantar corpus (hi base ). we keep the hyperparameters similar to the en classifiers. ## rule-based injection of spoken language features the ultimate objective of our work is to improve translation quality of spoken language translation systems. however, spoken parallel corpora are rare, and the existing ones clearly lack spontaneity as evident from our previous discussion. in this work, we try to create spontaneous synthetic parallel corpus using the written corpora. the written genre parallel sentences to normalize them @xcite . we then create word alignments between en and hi sentences using multilingual bert @xcite . we start with bigrams in the source language. since we are interested in phrase alignments, we keep those bigrams with their indices of corresponding alignments belonging to a continuous span of numbers. we combine two bigrams and their corresponding alignments if they have overlapping indices. this ensures we get valid phrases and their mapping from en to hi. we perform this combination process until convergence. after identifying phrases from the given word alignments, we add spoken features like non-lexical filler words (erm, emm, umm, eh, etc.), lexical filler words (so, like, etc.), and repetitions on both source and target sides with some probability p . we also add fillers at the beginning of sentences with probability p i . figure 4 illustrates that back-translated outputs don't make enough sense. this can be attributed to the low quality of our @xmath1. thus, we resort to using examples without back-translation. we denote this model by m rule . ## nmt with genre transfer the previous method fails to transfer spoken features like informal words, and introducing more grammatical words. to counter this, we try data driven approaches. we stitch models trained on written and spoken genres together. we use this model to get translations across genres. we also try experiments with fine-tuning the stitched model on the augmented data created in section 5. we use pretrained nmt models by tiedemann and thottingal (2020) to fine-tune on our task. they train marianmt (junczys-dowmunt et al., 2018) model on opus corpus @xcite , and made the models publicly available. we fine-tune the models on the spoken and written genre datasets and label them as m sp and m wr , respectively, for both translation directions. we use huggingface trainer for our training purposes. we train the spoken and written models for 10 epochs with a batch size of 8. we evaluate at every 500 steps while using early stopping callback. during inference, we stitch m sp and m wr to get m mix . we use the models to translate from en (and hi) in written to hi (and en) in spoken genre. we evaluate the generated parallel data on the written genre test set. ## discussion the @xmath2, but also introduces the filler words at correct places. on the other hand, the baseline model failed to recognize and add fillers and repetitions. since we didn't have any spontaneous examples in hi, we could not empirically evaluate @xmath3. however, we expect similar performance from that as well. we note that our output style evaluation relies on our genre classifier. for en, it has a varied training set but still it fails to work with unseen non-lexical fillers. for hi, the classifier is even less reliable and should be used with caution. clearly, there is a need for better genre classifiers. it would be interesting to see the results for m mix fine-tuned on back-translated data with added spoken features. due to lower quality back-translation outputs, we could not perform the experiments with it. one approach can be finetuning the models on the samanantar dataset for the hi-en language pair and then using the model for back-translation. another interesting experiment can also be checking the dependency of the model on controlled spoken features in training data and the extent to which they can be added without disturbing the content. ## conclusion this paper proposes two main methods to synthesize parallel spoken data using existing writtengenre parallel texts. given the written input, the data-driven method produces a naturally-looking spoken output; however, it fails to ensure appro-priate parallelism of explicit spoken features. the alternative rule-based approach has precise alignments of such features. furthermore, we check the usability of the created parallel texts by fine-tuning hi-en nmt models on merely 50k sentence pairs. the model fine-tuned on the synthetic corpus created using our rule-based method gives the best results. for @xmath4, it produces relatively decent results, and unlike the baseline model, it introduces correct non-lexical fillers at the proper places.